Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt


Alternative_Future - 3:57 pm on Apr 2, 2003 (gmt 0)


Thanks for the quick response Dreamquick,

I did have the name of the robot in there no harm is saying its name (LinkChecker) but it just doesnt obbey my rules! Last month it hogged nearly 1Gb of bandwidth! Any ideas on how to only allow it access to my links dir?

[added]Also the validator said all was fine, hence the reason i wanted to validate it from another human[/added]

Thx again,

-gs


Thread source:: http://www.webmasterworld.com/robots_txt/26.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com