homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Submitting robots.txt to google
Would google look sooner for it?

 6:05 pm on Apr 18, 2002 (gmt 0)

I would like to let google know about a updated robots.txt file. Stupid - in the old file i disallowed one (PR)-important directory to be crawled.

Should i submit the robots.txt-URL to google? Would google refresh it and "rebuild" the index of our page?



 6:11 pm on Apr 18, 2002 (gmt 0)

I have never heard of anyone submitting the robots.txt to Google.

Just put the updated version online and wait for Googlebot to come around. We just had an update, so in the coming weeks you will see the little guy coming around again.


 6:13 pm on Apr 18, 2002 (gmt 0)

I don't think it would help. I've never seen anything that suggests that Google pays any attention to page submissions. The next time they are out crawling, they'll check the robots file and note the changes.


 6:17 pm on Apr 18, 2002 (gmt 0)

agerhart, google was allready at my page _after_ the last update and did a deep crawl with the _old_ robots.txt!

I'm afraid, google will not look for the new robots.txt for a long future time ...


 6:19 pm on Apr 18, 2002 (gmt 0)

WebGuerrilla, you're wrong - googlebot only looks for the robots.txt every couple of month - not with every crawl!

I wouldn't ask ;-)


 6:24 pm on Apr 18, 2002 (gmt 0)

I'm afraid you just need to patient, Yidaki.

You can urgently remove a page (described here [google.com]) but you can't hurry up spidering.


 6:26 pm on Apr 18, 2002 (gmt 0)

Googlebot does in fact request the robots.txt every time it comes to your site...it is required to.


 6:35 pm on Apr 18, 2002 (gmt 0)

all, thanks for your replies but:

I don't want to hurry up spidering!

It' just that simple:
Googlebot does in fact NOT request robot.txt every visit - i really know how to analyze my logs ;-) and it's also mentioned here somewhere at the WebmasterWorld faq that google only checks for the file every couple of month or less!

(edited by: Yidaki at 6:43 pm (utc) on April 18, 2002)


 6:38 pm on Apr 18, 2002 (gmt 0)

regarding the frequency of robots.txt request:


But the proof are my logs and the visits of the googlebots since i know of its existance!


 6:46 pm on Apr 18, 2002 (gmt 0)

I don't think that's the case any more, Yidaki. I've not seen Googlebot request robots.txt less than once per crawl in the next six months.

Anyone else had this problem?


 6:47 pm on Apr 18, 2002 (gmt 0)


I realize what it says at the end of that link, and you almost had me convinced.

But, I went into our logs over the past 6 months and checked periodically. I checked a few from within one week in a few spots within the 6 month period.

Every time that Googlebot came to the site he grabbed the robots.txt and then proceeded on his way. How else would it know where and where not to go?


 6:49 pm on Apr 18, 2002 (gmt 0)

And this is next to the quote that you linked to -- "Last Updated:02/19/2001"

Sorry Brett!


 6:53 pm on Apr 18, 2002 (gmt 0)

It's been over a year since that was updated. In Google's early days, it didn't check robots.txt as often as it should, but I haven't seen that kind of behavior in a very long time. All the log files I look at show the same thing agerhart and ciml mentioned.

If they aren't requesting your file properly, you can find the contact info to report bad Googlebot behavior at



 6:53 pm on Apr 18, 2002 (gmt 0)


i'll timbuktu my server and check the logs for this last update to have the last proof - i'll post the results.

>How else would it know
>where and where not to go?

Good question - don't know the answer. From the cached robots file? Maybe he rebuilds the structure of each site's index (as "defined" in the robots.txt) only every couple of months? Curious ...


 6:58 pm on Apr 18, 2002 (gmt 0)

In order to save bandwidth Googlebot only downloads the robots.txt file once a day or whenever we have fetched many pages from the server. So, it may take a while for Googlebot to learn of any changes that might have been made to your robots.txt file. Also, Googlebot is distributed on several machines. Each of these keeps its own record of your robots.txt file. Finally, you may want to check that your syntax is correct against the standard at: [robotstxt.org...] If there still seems to be a problem, please let us know, and we will correct it.

available at: [google.com...]


 7:11 pm on Apr 18, 2002 (gmt 0)


the reports of my logs are a bit to big to download from my home acount know - i'll check them tomorrow morning and tell you the answer in my case.

Anyways - thanks for the snippet you posted: it says all and turns my original question into a "good" question again, i guess :-)

So, will google check the robots.txt, if it's submitted or not?


 8:19 am on Apr 19, 2002 (gmt 0)

Folks, i'm sorry ... please accept my apologize - the googlebot in fact did request the robots.txt every time it hat visited our server (in the last months). I didn't see him in my log reports because i write my own logfile and forgot to also record and write the robots.txt requests ... call me a fool! :-]

So ciml, you're right - i'll be patient and wait for the next hello of googlebot. No need to submit anything to google.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved