Hello, I have created a very simple robots.txt file to try and tell me if anything is crawling my web that is ignoring my robots.txt file. However , when testing it using the spider sim on searchengineworld it says that is has spidered it. Can anyone tell me if I have this correct? It checks out on the robots.txt validator. The file looks like this....
# # robots.txt #
User-agent: * Disallow: /BotID/Disallow/
I have a disallow.htm file (blank page) in the disallow folder and have a transparent gif on the home page that is linked to [mysite.com...]