Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- "User-Agent"?


pendanticist - 2:03 am on Jan 22, 2003 (gmt 0)


Ok, with some assistance, I got it figured out.

The proverbial white space.

This is what it should be:

User-Agent: #########
Disallow:/path/

as opposed to:

User-Agent:#########
Disallow:/path/

I don't understand why the results noted the problem being the lack of double quotation marks surrounding "User-Agent" if a white space is the culprit.

Pendanticist.


Thread source:: http://www.webmasterworld.com/robots_txt/152.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com