Google looks for robots.txt like mad today - (deprecated) Google News Archive forum at WebmasterWorld - WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

Google looks for robots.txt like mad today

SEOPTI

3:31 pm on Oct 6, 2003 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

I get at least 10 requests for robots.txt with each domain today. What's going on there?

dmorison

7:45 pm on Oct 7, 2003 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Yeah I noticed that; think the 'bot went on a mass robots.txt binge.

Only happened on one of my domains though; but from several Google IP's.