homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Does content type matter?
is text/html okay

 6:15 pm on Nov 7, 2003 (gmt 0)

It's a long story but basically my robots.txt gets sent with a "text/html" content-type header instead of the normal "text/plain".

will this affect SE's ability to read it? I doubt it but just want to make sure. thanks



 8:27 am on Nov 8, 2003 (gmt 0)

fairly a good question. i can only assume that it does not make a difference that much because it starts with text/, but i can't tell you, because this is very specific about which robot is looking for it etc. any can do such a check for mimetype or not. maybe someone here made a testscenario about that already, but i don't know.

if you want to have this corrected (on my server it's text/plain and i have no probs), contact your server admin.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved