Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

disallow is allowing

Is this done correctly?



6:25 pm on Feb 11, 2004 (gmt 0)

10+ Year Member

I have created a very simple robots.txt file to try and tell me if anything is crawling my web that is ignoring my robots.txt file. However , when testing it using the spider sim on searchengineworld it says that is has spidered it. Can anyone tell me if I have this correct? It checks out on the robots.txt validator. The file looks like this....

# robots.txt

User-agent: *
Disallow: /BotID/Disallow/

I have a disallow.htm file (blank page) in the disallow folder and have a transparent gif on the home page that is linked to [mysite.com...]

Any suggestions? thanks.......


Featured Threads

Hot Threads This Week

Hot Threads This Month