Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
I don't want to hurry up spidering!
It' just that simple:
Googlebot does in fact NOT request robot.txt every visit - i really know how to analyze my logs ;-) and it's also mentioned here somewhere at the WebmasterWorld faq that google only checks for the file every couple of month or less!
(edited by: Yidaki at 6:43 pm (utc) on April 18, 2002)
I realize what it says at the end of that link, and you almost had me convinced.
But, I went into our logs over the past 6 months and checked periodically. I checked a few from within one week in a few spots within the 6 month period.
Every time that Googlebot came to the site he grabbed the robots.txt and then proceeded on his way. How else would it know where and where not to go?
If they aren't requesting your file properly, you can find the contact info to report bad Googlebot behavior at
i'll timbuktu my server and check the logs for this last update to have the last proof - i'll post the results.
>How else would it know
>where and where not to go?
Good question - don't know the answer. From the cached robots file? Maybe he rebuilds the structure of each site's index (as "defined" in the robots.txt) only every couple of months? Curious ...
In order to save bandwidth Googlebot only downloads the robots.txt file once a day or whenever we have fetched many pages from the server. So, it may take a while for Googlebot to learn of any changes that might have been made to your robots.txt file. Also, Googlebot is distributed on several machines. Each of these keeps its own record of your robots.txt file. Finally, you may want to check that your syntax is correct against the standard at: [robotstxt.org...] If there still seems to be a problem, please let us know, and we will correct it.
available at: [google.com...]
the reports of my logs are a bit to big to download from my home acount know - i'll check them tomorrow morning and tell you the answer in my case.
Anyways - thanks for the snippet you posted: it says all and turns my original question into a "good" question again, i guess :-)
So, will google check the robots.txt, if it's submitted or not?
So ciml, you're right - i'll be patient and wait for the next hello of googlebot. No need to submit anything to google.