Welcome to WebmasterWorld Guest from 54.197.116.116

Forum Moderators: goodroi

Message Too Old, No Replies

Blocking particular URL form robots.txt

   
6:35 am on Apr 22, 2010 (gmt 0)

5+ Year Member



Hello,

I have heard a lot about the blocking of folders and sub-folders from robots.txt file and i have used them a lot, but I still can't find the way to block a particular URL there and also if i have a particular folder to block which exists in a tree, means main-folder/fold1/fold2/fold3/fold4 and i am trying to block just fold4 folder. What to do? and if i have dynamic URLs and to block them what i have to do?

Can anybody suggest it?
6:46 am on Apr 22, 2010 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



If that link can be found off site (a link to your page via another site) you will be indexed. Nature of the beast.

Meanwhile, visit the Apache forums or use the new advanced search at WW for robots.txt for other pointers.

Do realize that your robots.txt is also a blueprint for robots (or scrapers) to find content on your site... the baddies will ignore instructions and go straight for the goodies.