Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
If you only have a couple of links in and you have a site with 10000 pages spread throughout 5 levels then expect it to take a long time for google to crawl.
However if you have 1000's of inbound links then expect a to get all pages in the serps in a small period of time.
joined:Jan 31, 2004
Is it true that the G and Y's spiders only crawl the top two levels of pages on a site?
What do you mean by "top two levels"?
Directory structure need play no part in spider algos - spiders simply follow links.
Perhaps the question should be rephrased thus
"Is it true that G and Y's spiders only follow links two clicks deep?"
In this case, the answer would still be NO.
Many have speculated that the PR algo used by Google somehow uses directory structure (if only for making initial guesses). A flat directory structure might be useful for PR, but I've never seen any real evidence for this.
GoogleGuy has often recommended site maps. I think that's good advice.
If you have a lot of pages have the front page point to a hundred or so 2nd level pages then those pages each point to a hundred or so pages. This looks like a site map. Put the links on the bottom of the page on the 2nd level. Make the 2nd level pages content pages with your most important words. A site map does not have to be a page of links take advantage of the outgoing/internal links and word density and make a content page out of each of your sitemap pages. Just make sure that your pages are real content and that you did not buy your site for PR. Google may frown upon that.
It is easy to prove. I had a site ...
That's like saying "the falling piano missed me therefore there must be a God".
What you said really did not prove anything as such. Your changes could have coincided with a big algorithm change, backlinks update, PageRank update (depending on when your talking about), 'sandbox' period expiry etc. etc. etc.
PLUS - just because something is true in the past does not make it sound advice for the future. Many, many, many sites do incredibly well using as many 'levels' as they want. Internal linking structure is a major part of good indexing and ranking and (in my opinion) always will be, but let's not make up "levels" ghost stories to scare the newbies with.
Meanwhile it will check on index.html more frequently.
So less deep directory structure might give you quicker re-indexing of a site.
Either way, it will crawl beyond 2 deep.