homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

disallow is allowing
Is this done correctly?

10+ Year Member

Msg#: 284 posted 6:25 pm on Feb 11, 2004 (gmt 0)

I have created a very simple robots.txt file to try and tell me if anything is crawling my web that is ignoring my robots.txt file. However , when testing it using the spider sim on searchengineworld it says that is has spidered it. Can anyone tell me if I have this correct? It checks out on the robots.txt validator. The file looks like this....

# robots.txt

User-agent: *
Disallow: /BotID/Disallow/

I have a disallow.htm file (blank page) in the disallow folder and have a transparent gif on the home page that is linked to [mysite.com...]

Any suggestions? thanks.......


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved