How to block 1 file - Crawler, Spider, and User Agent ID forum at WebmasterWorld - WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

How to block 1 file

stratocaster

7:40 pm on Oct 27, 2002 (gmt 0)

10+ Year Member

Question,

I have setup the robots.txt to block a specific file, but on my pages i have index all and follow. Is this a problem?

andreasfriedrich

9:46 pm on Oct 29, 2002 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

No, since a spider that adheres to the Robots Exclusion standard won´t even see the meta tags, since it won´t request that page.

Andreas