homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

This 31 message thread spans 2 pages: < < 31 ( 1 [2]     
Quick tip
Check out your robots.txt
GoogleGuy




msg:1527214
 5:39 pm on Jun 11, 2002 (gmt 0)

I've seen this hit a couple webmasters lately and wanted to mention it. If someone requests a robots.txt file, don't return some custom 404 page or strange page with HTML, graphics, etc. This can happen if your webserver is configured to return a pretty page for requests when the page doesn't exist.

Just a straight text robots file or no robots.txt at all is preferred. Just lookin' out for things the sometimes catch people..

 

Mardi_Gras




msg:1527244
 12:34 pm on Jun 18, 2002 (gmt 0)

Afterburner - yes - and check out Brett's robots.txt tutorial at [searchengineworld.com...]

You might also want to use the robots.txt validator located here:

[searchengineworld.com...]

This 31 message thread spans 2 pages: < < 31 ( 1 [2]
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved