homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

google and robots.txt - do I need one?
cheater copperpot

 3:42 am on Aug 7, 2002 (gmt 0)

Simple question:

Will google find me if i dont have a robots.txt file? or do i need one? I dont have any file named robots.txt on my server now.




 3:43 am on Aug 7, 2002 (gmt 0)

No, the Robots Exclusion Standard is to (you guessed it) exclude robots - not include. Welcome to the board.


 4:14 am on Aug 7, 2002 (gmt 0)


However, if you monitor your site logs, you can save yourself from having to wade through a lot of
404-Not Found errors by uploading at least a blank robots.txt file, as recommended by GoogleGuy.

This prevents your server from sending a 404 response code back to robots which request robots.txt,
and from logging that error response. But since the file is empty, it doesn't exclude any robots.

More info here [robotstxt.org].



 4:19 am on Aug 7, 2002 (gmt 0)

No, you don't need to have a robots.txt file, and including one does not guarantee any bots you want to exclude will request and/or recognize it. However, you can use it as a tracking tool if you habitually review your access logs. Also, if robots.txt is requested by a bot and not found, it will generate 404 errors in your access log. If you don't mind the 404's in your access logs, aren't tracking robots.txt accesses, and don't wish to exclude any robots.txt-minding bots, it really isn't necessary at all. If you choose to include one, on the other hand, there is a ton of posts on WebmasterWorld (and throughout the web) for more information :)

<added>jdMorgan pretty much said already this while I was typing this!</added>

cheater copperpot

 5:45 am on Aug 7, 2002 (gmt 0)

ok cool guys thanks alot. I guess I will just go ahead and throw a blank one in for now as recommended by GG.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved