Welcome to WebmasterWorld Guest from 54.145.176.120

Forum Moderators: goodroi

Message Too Old, No Replies

disallow is allowing

Is this done correctly?

   
6:25 pm on Feb 11, 2004 (gmt 0)

10+ Year Member



Hello,
I have created a very simple robots.txt file to try and tell me if anything is crawling my web that is ignoring my robots.txt file. However , when testing it using the spider sim on searchengineworld it says that is has spidered it. Can anyone tell me if I have this correct? It checks out on the robots.txt validator. The file looks like this....

#
# robots.txt
#

User-agent: *
Disallow: /BotID/Disallow/

I have a disallow.htm file (blank page) in the disallow folder and have a transparent gif on the home page that is linked to [mysite.com...]

Any suggestions? thanks.......

 

Featured Threads

Hot Threads This Week

Hot Threads This Month