Welcome to WebmasterWorld Guest from 23.20.241.155

Forum Moderators: goodroi

Message Too Old, No Replies

Submitting robots.txt to google

Would google look sooner for it?

     
6:05 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I would like to let google know about a updated robots.txt file. Stupid - in the old file i disallowed one (PR)-important directory to be crawled.

Should i submit the robots.txt-URL to google? Would google refresh it and "rebuild" the index of our page?

6:11 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I have never heard of anyone submitting the robots.txt to Google.

Just put the updated version online and wait for Googlebot to come around. We just had an update, so in the coming weeks you will see the little guy coming around again.

6:13 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member




I don't think it would help. I've never seen anything that suggests that Google pays any attention to page submissions. The next time they are out crawling, they'll check the robots file and note the changes.
6:17 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



agerhart, google was allready at my page _after_ the last update and did a deep crawl with the _old_ robots.txt!

I'm afraid, google will not look for the new robots.txt for a long future time ...

6:19 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



WebGuerrilla, you're wrong - googlebot only looks for the robots.txt every couple of month - not with every crawl!

I wouldn't ask ;-)

6:24 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I'm afraid you just need to patient, Yidaki.

You can urgently remove a page (described here [google.com]) but you can't hurry up spidering.

6:26 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Googlebot does in fact request the robots.txt every time it comes to your site...it is required to.
6:35 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



all, thanks for your replies but:

I don't want to hurry up spidering!

It' just that simple:
Googlebot does in fact NOT request robot.txt every visit - i really know how to analyze my logs ;-) and it's also mentioned here somewhere at the WebmasterWorld faq that google only checks for the file every couple of month or less!

(edited by: Yidaki at 6:43 pm (utc) on April 18, 2002)

6:38 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



regarding the frequency of robots.txt request:

[webmasterworld.com...]

But the proof are my logs and the visits of the googlebots since i know of its existance!

6:46 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I don't think that's the case any more, Yidaki. I've not seen Googlebot request robots.txt less than once per crawl in the next six months.

Anyone else had this problem?

6:47 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Yidaki,

I realize what it says at the end of that link, and you almost had me convinced.

But, I went into our logs over the past 6 months and checked periodically. I checked a few from within one week in a few spots within the 6 month period.

Every time that Googlebot came to the site he grabbed the robots.txt and then proceeded on his way. How else would it know where and where not to go?

6:49 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



And this is next to the quote that you linked to -- "Last Updated:02/19/2001"

Sorry Brett!

6:53 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member




It's been over a year since that was updated. In Google's early days, it didn't check robots.txt as often as it should, but I haven't seen that kind of behavior in a very long time. All the log files I look at show the same thing agerhart and ciml mentioned.

If they aren't requesting your file properly, you can find the contact info to report bad Googlebot behavior at

[google.com...]

6:53 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



agerhart,

i'll timbuktu my server and check the logs for this last update to have the last proof - i'll post the results.

>How else would it know
>where and where not to go?

Good question - don't know the answer. From the cached robots file? Maybe he rebuilds the structure of each site's index (as "defined" in the robots.txt) only every couple of months? Curious ...

6:58 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



In order to save bandwidth Googlebot only downloads the robots.txt file once a day or whenever we have fetched many pages from the server. So, it may take a while for Googlebot to learn of any changes that might have been made to your robots.txt file. Also, Googlebot is distributed on several machines. Each of these keeps its own record of your robots.txt file. Finally, you may want to check that your syntax is correct against the standard at: [robotstxt.org...] If there still seems to be a problem, please let us know, and we will correct it.

available at: [google.com...]

7:11 pm on Apr 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



agerhart,

the reports of my logs are a bit to big to download from my home acount know - i'll check them tomorrow morning and tell you the answer in my case.

Anyways - thanks for the snippet you posted: it says all and turns my original question into a "good" question again, i guess :-)

So, will google check the robots.txt, if it's submitted or not?

8:19 am on Apr 19, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Folks, i'm sorry ... please accept my apologize - the googlebot in fact did request the robots.txt every time it hat visited our server (in the last months). I didn't see him in my log reports because i write my own logfile and forgot to also record and write the robots.txt requests ... call me a fool! :-]

So ciml, you're right - i'll be patient and wait for the next hello of googlebot. No need to submit anything to google.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month