joined:Jan 28, 2004
I have created a very simple robots.txt file to try and tell me if anything is crawling my web that is ignoring my robots.txt file. However , when testing it using the spider sim on searchengineworld it says that is has spidered it. Can anyone tell me if I have this correct? It checks out on the robots.txt validator. The file looks like this....
I have a disallow.htm file (blank page) in the disallow folder and have a transparent gif on the home page that is linked to [mysite.com...]
Any suggestions? thanks.......