what's important to realise is that your "competitors" and the other websites in your google search results are also submitting their sites to directories, article sites etc. You will need to have more quality backlinks than them overall to rank higher than them.
so its just now what *you* do, but also what you're competition is doing. You will need to do more than them to rank higher in the SERPs.
Just submitting and creating backlinks on its own is only half the story.
Also yes, site size makes a difference - the more quality content you have, the better it is for your SERP rankings.