Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- "User-Agent"?


pendanticist - 1:35 am on Jan 22, 2003 (gmt 0)


I understand what you're saying, Jim. However, I only have one Disallow in there and, at the moment it is not "User-Agent" but just User-Agent.

That is where SEW's robots.txt tester finds an error by illustrating the need for "" marks - one on either side of User-Agent.

Thusly:

# Robots.txt file from http*//yadayada.com
#
# All robots will spider the domain

User-Agent:#########
Disallow:/path/

<shrug>

Pendanticist.


Thread source:: http://www.webmasterworld.com/robots_txt/152.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com