Forum Moderators: goodroi
For some time now I have been trying to exclude spyders from the php script 057hov.php as you can see below.
I've run the robots.txt through the validator and it check out ok but the following line (as recorded within the script) shows that a google bot is clearly spidering it..
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: admin.php
Disallow: /cach/
Disallow: 057hov.php
I'm wondering if anyone can let me know what mistake I am making?
Wed Oct 12 9:58 CDT *66.249.65.225*crawl-66-249-65-225.googlebot.com*Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)**
Thanks, Steve.