Welcome to WebmasterWorld Guest from 54.196.232.162

Forum Moderators: goodroi

Message Too Old, No Replies

google and robots.txt - do I need one?

     
3:42 am on Aug 7, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:June 21, 2002
posts:116
votes: 0


Simple question:

Will google find me if i dont have a robots.txt file? or do i need one? I dont have any file named robots.txt on my server now.

Thanks

3:43 am on Aug 7, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


No, the Robots Exclusion Standard is to (you guessed it) exclude robots - not include. Welcome to the board.
4:14 am on Aug 7, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


CC,

However, if you monitor your site logs, you can save yourself from having to wade through a lot of
404-Not Found errors by uploading at least a blank robots.txt file, as recommended by GoogleGuy.

This prevents your server from sending a 404 response code back to robots which request robots.txt,
and from logging that error response. But since the file is empty, it doesn't exclude any robots.

More info here [robotstxt.org].

Jim

4:19 am on Aug 7, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:July 6, 2001
posts:410
votes: 0


No, you don't need to have a robots.txt file, and including one does not guarantee any bots you want to exclude will request and/or recognize it. However, you can use it as a tracking tool if you habitually review your access logs. Also, if robots.txt is requested by a bot and not found, it will generate 404 errors in your access log. If you don't mind the 404's in your access logs, aren't tracking robots.txt accesses, and don't wish to exclude any robots.txt-minding bots, it really isn't necessary at all. If you choose to include one, on the other hand, there is a ton of posts on WebmasterWorld (and throughout the web) for more information :)

<added>jdMorgan pretty much said already this while I was typing this!</added>

5:45 am on Aug 7, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:June 21, 2002
posts:116
votes: 0


ok cool guys thanks alot. I guess I will just go ahead and throw a blank one in for now as recommended by GG.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members