Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Quick tip

Check out your robots.txt

5:39 pm on Jun 11, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member googleguy is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Oct 8, 2001
votes: 0

I've seen this hit a couple webmasters lately and wanted to mention it. If someone requests a robots.txt file, don't return some custom 404 page or strange page with HTML, graphics, etc. This can happen if your webserver is configured to return a pretty page for requests when the page doesn't exist.

Just a straight text robots file or no robots.txt at all is preferred. Just lookin' out for things the sometimes catch people..

12:34 pm on June 18, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 27, 2002
votes: 0

Afterburner - yes - and check out Brett's robots.txt tutorial at [searchengineworld.com...]

You might also want to use the robots.txt validator located here:


This 31 message thread spans 2 pages: 31