Forum Moderators: open

Message Too Old, No Replies

Google Visited

But what did it do?

         

giggle

8:28 am on Jan 14, 2003 (gmt 0)

10+ Year Member



I recently noticed this entry in my stats:

* User Session 216.239.46.77 - Google Inc. (CA,US)
2003-01-10 09:39:29 - user created
2003-01-10 09:39:29 /robots.txt
2003-01-10 09:39:29 /

What happened here? Did Google visit, look for a robots.txt file, not find it and then go away again?

Brett_Tabke

11:40 am on Jan 14, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Ya, common. They grab the robots.txt by itself and add it to their robots.txt database. Then they'll come back some other time to crawl the site.