Forum Moderators: goodroi

Message Too Old, No Replies

crawl confusion

Understanding my hits log

         

cardpro

2:20 pm on Mar 13, 2005 (gmt 0)

10+ Year Member



I'm trying to understand my hits log. What would this mean: crawl-66-249-64-6.googlebot.com - - [10/Mar/2005:23:02:17 -0500] "GET /robots.txt HTTP/1.0" 404 148 "" "Googlebot/2.1 (+http://www.google.com/bot.html)"
crawl-66-249-64-6.googlebot.com - - [10/Mar/2005:23:02:18 -0500] "GET / HTTP/1.0" 200 19863 "" "Googlebot/2.1 (+http://www.google.com/bot.html)"

Thanks for any help

wruppert

2:35 pm on Mar 13, 2005 (gmt 0)

10+ Year Member



The first record means Google tried to get your robots.txt file, and did not find it - that is what the "404" means.

The second record is Google successfully getting whatever page is returned at "www.yourdomain.com" - your home page. That is what the "200" means.

The gory details of Apache logs can be found at [httpd.apache.org...]

cardpro

10:13 pm on Mar 13, 2005 (gmt 0)

10+ Year Member



Thanks for the reply. By google not finding my robot.txt file, does that penalize my ranking?

ThomasB

12:20 am on Mar 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



cardpro, you have to name the file robots.txt - and no, you won't have a disadvantage by not having a robots.txt file

sem4u

8:59 am on Mar 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you save a blank notepad file, call it robots.txt and upload it to the root directory it will get rid of any further robots.txt 404 errors.