| This 31 message thread spans 2 pages: < < 31 ( 1 [2] ) | |
| Quick tip
Check out your robots.txt |
GoogleGuy
msg:1527214 | 5:39 pm on Jun 11, 2002 (gmt 0) |
I've seen this hit a couple webmasters lately and wanted to mention it. If someone requests a robots.txt file, don't return some custom 404 page or strange page with HTML, graphics, etc. This can happen if your webserver is configured to return a pretty page for requests when the page doesn't exist. Just a straight text robots file or no robots.txt at all is preferred. Just lookin' out for things the sometimes catch people..
|
Mardi_Gras
msg:1527244 | 12:34 pm on Jun 18, 2002 (gmt 0) |
Afterburner - yes - and check out Brett's robots.txt tutorial at [searchengineworld.com...] You might also want to use the robots.txt validator located here: [searchengineworld.com...]
|
| This 31 message thread spans 2 pages: < < 31 ( 1 [2] ) |
|
|
|
