Forum Moderators: phranque
I have an established website which originally only contained 5 pages. I have completely changed the functionality of this site and it now contains 100s of pages, going down to 3 levels. Google, however, is only spidering down to the 2nd level (1 level past the index page). The index page is PR6 btw but the second level pages are new so PR0.
Yesterday I made two changes to the site:
1) I accidentally created a blank robots.txt!
2) I added a link from the index page to about 150 3rd level pages (which were not being spidered) to see if G may take a look at them!
Result: today G did a get on the robots.txt, then the index page and did not return!
Could an empty robots.txt have caused this, did it not like so many links or am I just being paranoid?!
Best regards
Shady
An empty robots.txt, a nonexistent robots.txt, and a robots.txt that contains
User-agent: *
Disallow:Generally, it is better to have the file even if it is blank, because that will save you thousands of 404-Not Found errors per month as spiders look for the missing robots.txt file. This makes it easier to use your error log to find real errors.
Check your pages for other errors, incorrect on-page robots meta-tags, etc. On the other hand, sometimes Googlebot justs checks-in on frequently-updated pages and then leaves, even on well-indexed sites.
Jim
The index page was spidered down a level everyday since I created it, until I added these additional links. Nothing else changed.
I have reduced the links to less than 100 for the whole page and we'll see what happens tomorrow :-)
