Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
126.96.36.199 - - [08/Feb/2004:08:22:08 +0100] "GET /robots.txt HTTP/1.0" 200 98 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
188.8.131.52 - - [08/Feb/2004:08:22:09 +0100] "GET / HTTP/1.0" 200 12306 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
Everyday googlebot requests the robots.txt file, and after that it gets the index file and thats it.
Other bots index my entire site, froogle preditcor looks ok, search engine spider simulators look fine...
Anybody knows what the problem is?
[edited by: Marcia at 4:15 am (utc) on Feb. 9, 2004]
[edit reason] URL not necessary. [/edit]
No errors detected! This Robots.txt validates to the robots exclusion standard!
my robots.txt file is just this:
What my concern is, that only googlebot seems to be having problems with indexing my site.
I think if you get more links pointing at your site (and at deep pages, rather than just the index page) your site should get more pages crawled more frequently.
This site got only the index.html spidered. Another site that was launched just about a month before, was expanded with about 50 pages, and got completely indexed over the weekend.
I think Google has sharpened the rules of getting in the index - maybe storage/analysis capacity starting to play a role? Maybe it won't even fully index sites if they don't have more than, say, about 10 backlinks from different sites?
Some of my sites do not have a robots.txt file but some do and they've all been validated ok.
This has been going on for aprox. the past 4 - 5 weeks.
184.108.40.206 - - [09/Feb/2004:01:45:49 -0500] "GET / HTTP/1.0" 200 26432 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
I did notice that msnbot, openbot and Turnitinbot were all on the site last night and all indexed some pages. Could they have had anything to do with Googlebot leaving?
Can anyone interpret the above code? It appeared twice during the night.
Same thing happens here on a new domain, but it *should* request more files (links from withing the document). I checked the error log, but Gbot did request the file correctly so the server/site shouldn't be the problem.
Just 2 request robots.txt and index..
inktomisearch.com does the same aswell.. :(
It grabbed my index page and Sitemap only.
Here's hoping for a deep crawl sooner than later
Gbot comes to my index page once a day and deep crawls once a month, usually a week before any major update - just changed recently. This has been happening since October. However I've also noticed the index page is returning a 304 status (not modified since)and assumed this was the reason for the lack of gbot activity.
Anyway just changed my index and added 100 or so pages so I'm hoping freshbot will be along in the next few days.
Can only wait and see. ;)
I'd think the call for your site map's a good sign.. :-)
Mebbe Gbot will be back now that it shows a site map in it's doc. batrrels for the site, and one that hasn't been followed at that...
Adding a PR7 link to the sitemap led to it being indexed, but it doesn't show up in the SERPs yet. Still, it should have spidered more links from the sitemap by now.
Older sites are getting deepcrawled allright, so it's either a problem with new sites, or some adjustment in the bot NOT to deepspider new sites unless they meet certain terms (e.g. mucho links :) ).
I am getting the same thing on my site. I submitted to Google about 6-8 weeks ago. It does the same thing everytime it comes to my site; goes to robots.txt then to the home pages and then leaves. It does this process, sometimes multiple times per day.