homepage Welcome to WebmasterWorld Guest from 54.211.219.178
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Dangers of the robots.txt file
A warning to the amateurs like myself
joeuz




msg:1529588
 2:07 am on Apr 23, 2003 (gmt 0)

Hi everyone,

Just wanted to share something I learned concerning the robots.txt file. I always took this file for granted and most of my sites don't even have one. Unfortunately, recently I built a website with several subdomains all running under one program. Due to a bug in the program, the website was returning an error page instead of a 404 File Not Found when queried for a missing file. Google always queries the robots.txt file before going any further in the website and it was getting the text of the error page instead of a File Not Found. Apparently, Google didn't like that and did not bother to crawl any further.

So be careful and make sure your robots.txt file is functioning properly.

Joe

 

tejas_shah




msg:1529589
 3:53 am on Apr 23, 2003 (gmt 0)

what code did you use for the robots.txt file?

joeuz




msg:1529590
 6:37 am on Apr 23, 2003 (gmt 0)

I used .NET to get the queried file from the correct location based on the subdomain.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved