Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Dangers of the robots.txt file

A warning to the amateurs like myself



2:07 am on Apr 23, 2003 (gmt 0)

10+ Year Member

Hi everyone,

Just wanted to share something I learned concerning the robots.txt file. I always took this file for granted and most of my sites don't even have one. Unfortunately, recently I built a website with several subdomains all running under one program. Due to a bug in the program, the website was returning an error page instead of a 404 File Not Found when queried for a missing file. Google always queries the robots.txt file before going any further in the website and it was getting the text of the error page instead of a File Not Found. Apparently, Google didn't like that and did not bother to crawl any further.

So be careful and make sure your robots.txt file is functioning properly.



3:53 am on Apr 23, 2003 (gmt 0)

10+ Year Member

what code did you use for the robots.txt file?


6:37 am on Apr 23, 2003 (gmt 0)

10+ Year Member

I used .NET to get the queried file from the correct location based on the subdomain.

Featured Threads

Hot Threads This Week

Hot Threads This Month