Submitting robots.txt to google

Would google look sooner for it?

[Yidaki]

I would like to let google know about a updated robots.txt file. Stupid - in the old file i disallowed one (PR)-important directory to be crawled.

Should i submit the robots.txt-URL to google? Would google refresh it and "rebuild" the index of our page?

[agerhart]

I have never heard of anyone submitting the robots.txt to Google.

Just put the updated version online and wait for Googlebot to come around. We just had an update, so in the coming weeks you will see the little guy coming around again.

[WebGuerrilla]

I don't think it would help. I've never seen anything that suggests that Google pays any attention to page submissions. The next time they are out crawling, they'll check the robots file and note the changes.

[Yidaki]

agerhart, google was allready at my page _after_ the last update and did a deep crawl with the _old_ robots.txt!

I'm afraid, google will not look for the new robots.txt for a long future time ...

[Yidaki]

WebGuerrilla, you're wrong - googlebot only looks for the robots.txt every couple of month - not with every crawl!

I wouldn't ask ;-)

[ciml]

I'm afraid you just need to patient, Yidaki.

You can urgently remove a page (described here [google.com]) but you can't hurry up spidering.

[agerhart]

Googlebot does in fact request the robots.txt every time it comes to your site...it is required to.

[Yidaki]

all, thanks for your replies but:

I don't want to hurry up spidering!

It' just that simple:
Googlebot does in fact NOT request robot.txt every visit - i really know how to analyze my logs ;-) and it's also mentioned here somewhere at the WebmasterWorld faq that google only checks for the file every couple of month or less!

(edited by: Yidaki at 6:43 pm (utc) on April 18, 2002)

[Yidaki]

regarding the frequency of robots.txt request:

[webmasterworld.com...]

But the proof are my logs and the visits of the googlebots since i know of its existance!

[ciml]

I don't think that's the case any more, Yidaki. I've not seen Googlebot request robots.txt less than once per crawl in the next six months.

Anyone else had this problem?

[agerhart]

Yidaki,

I realize what it says at the end of that link, and you almost had me convinced.

But, I went into our logs over the past 6 months and checked periodically. I checked a few from within one week in a few spots within the 6 month period.

Every time that Googlebot came to the site he grabbed the robots.txt and then proceeded on his way. How else would it know where and where not to go?

[agerhart]

And this is next to the quote that you linked to -- "Last Updated:02/19/2001"

Sorry Brett!

[WebGuerrilla]

It's been over a year since that was updated. In Google's early days, it didn't check robots.txt as often as it should, but I haven't seen that kind of behavior in a very long time. All the log files I look at show the same thing agerhart and ciml mentioned.

If they aren't requesting your file properly, you can find the contact info to report bad Googlebot behavior at

[google.com...]

[Yidaki]

agerhart,

i'll timbuktu my server and check the logs for this last update to have the last proof - i'll post the results.

>How else would it know
>where and where not to go?

Good question - don't know the answer. From the cached robots file? Maybe he rebuilds the structure of each site's index (as "defined" in the robots.txt) only every couple of months? Curious ...

[agerhart]

In order to save bandwidth Googlebot only downloads the robots.txt file once a day or whenever we have fetched many pages from the server. So, it may take a while for Googlebot to learn of any changes that might have been made to your robots.txt file. Also, Googlebot is distributed on several machines. Each of these keeps its own record of your robots.txt file. Finally, you may want to check that your syntax is correct against the standard at: [robotstxt.org...] If there still seems to be a problem, please let us know, and we will correct it.

available at: [google.com...]

[Yidaki]

agerhart,

the reports of my logs are a bit to big to download from my home acount know - i'll check them tomorrow morning and tell you the answer in my case.

Anyways - thanks for the snippet you posted: it says all and turns my original question into a "good" question again, i guess :-)

So, will google check the robots.txt, if it's submitted or not?

[Yidaki]

Folks, i'm sorry ... please accept my apologize - the googlebot in fact did request the robots.txt every time it hat visited our server (in the last months). I didn't see him in my log reports because i write my own logfile and forgot to also record and write the robots.txt requests ... call me a fool! :-]

So ciml, you're right - i'll be patient and wait for the next hello of googlebot. No need to submit anything to google.