Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Blocking particular URL form robots.txt

6:35 am on Apr 22, 2010 (gmt 0)

New User

5+ Year Member

joined:Apr 22, 2010
votes: 0


I have heard a lot about the blocking of folders and sub-folders from robots.txt file and i have used them a lot, but I still can't find the way to block a particular URL there and also if i have a particular folder to block which exists in a tree, means main-folder/fold1/fold2/fold3/fold4 and i am trying to block just fold4 folder. What to do? and if i have dynamic URLs and to block them what i have to do?

Can anybody suggest it?
6:46 am on Apr 22, 2010 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
votes: 443

If that link can be found off site (a link to your page via another site) you will be indexed. Nature of the beast.

Meanwhile, visit the Apache forums or use the new advanced search at WW for robots.txt for other pointers.

Do realize that your robots.txt is also a blueprint for robots (or scrapers) to find content on your site... the baddies will ignore instructions and go straight for the goodies.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members