Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: goodroi

Message Too Old, No Replies

Dangers of the robots.txt file

A warning to the amateurs like myself

     
2:07 am on Apr 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 14, 2002
posts:41
votes: 0


Hi everyone,

Just wanted to share something I learned concerning the robots.txt file. I always took this file for granted and most of my sites don't even have one. Unfortunately, recently I built a website with several subdomains all running under one program. Due to a bug in the program, the website was returning an error page instead of a 404 File Not Found when queried for a missing file. Google always queries the robots.txt file before going any further in the website and it was getting the text of the error page instead of a File Not Found. Apparently, Google didn't like that and did not bother to crawl any further.

So be careful and make sure your robots.txt file is functioning properly.

Joe

3:53 am on Apr 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 25, 2003
posts:43
votes: 0


what code did you use for the robots.txt file?
6:37 am on Apr 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 14, 2002
posts:41
votes: 0


I used .NET to get the queried file from the correct location based on the subdomain.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members