Forum Moderators: goodroi

Message Too Old, No Replies

Disallow: line in robots.txt

         

stefan009

5:13 pm on Oct 14, 2005 (gmt 0)

10+ Year Member



Hello,

For some time now I have been trying to exclude spyders from the php script 057hov.php as you can see below.

I've run the robots.txt through the validator and it check out ok but the following line (as recorded within the script) shows that a google bot is clearly spidering it..

User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: admin.php
Disallow: /cach/
Disallow: 057hov.php

I'm wondering if anyone can let me know what mistake I am making?

Wed Oct 12 9:58 CDT *66.249.65.225*crawl-66-249-65-225.googlebot.com*Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)**

Thanks, Steve.

jdMorgan

5:33 pm on Oct 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Robots use prefix-matching; Anything that *starts with* the string specified in your Disallow directive should not be fetched.

Therefore, you need to start each local URL-path with "/".

Jim