Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
18.104.22.168 - - [08/Feb/2004:08:22:08 +0100] "GET /robots.txt HTTP/1.0" 200 98 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
22.214.171.124 - - [08/Feb/2004:08:22:09 +0100] "GET / HTTP/1.0" 200 12306 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
Everyday googlebot requests the robots.txt file, and after that it gets the index file and thats it.
Other bots index my entire site, froogle preditcor looks ok, search engine spider simulators look fine...
Anybody knows what the problem is?
[edited by: Marcia at 4:15 am (utc) on Feb. 9, 2004]
[edit reason] URL not necessary. [/edit]
I think if you get more links pointing at your site (and at deep pages, rather than just the index page) your site should get more pages crawled more frequently.
This site got only the index.html spidered. Another site that was launched just about a month before, was expanded with about 50 pages, and got completely indexed over the weekend.
I think Google has sharpened the rules of getting in the index - maybe storage/analysis capacity starting to play a role? Maybe it won't even fully index sites if they don't have more than, say, about 10 backlinks from different sites?
126.96.36.199 - - [09/Feb/2004:01:45:49 -0500] "GET / HTTP/1.0" 200 26432 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
I did notice that msnbot, openbot and Turnitinbot were all on the site last night and all indexed some pages. Could they have had anything to do with Googlebot leaving?
Can anyone interpret the above code? It appeared twice during the night.
Same thing happens here on a new domain, but it *should* request more files (links from withing the document). I checked the error log, but Gbot did request the file correctly so the server/site shouldn't be the problem.
Gbot comes to my index page once a day and deep crawls once a month, usually a week before any major update - just changed recently. This has been happening since October. However I've also noticed the index page is returning a 304 status (not modified since)and assumed this was the reason for the lack of gbot activity.
Anyway just changed my index and added 100 or so pages so I'm hoping freshbot will be along in the next few days.
Can only wait and see. ;)
I'd think the call for your site map's a good sign.. :-)
Mebbe Gbot will be back now that it shows a site map in it's doc. batrrels for the site, and one that hasn't been followed at that...
Adding a PR7 link to the sitemap led to it being indexed, but it doesn't show up in the SERPs yet. Still, it should have spidered more links from the sitemap by now.
Older sites are getting deepcrawled allright, so it's either a problem with new sites, or some adjustment in the bot NOT to deepspider new sites unless they meet certain terms (e.g. mucho links :) ).