| Blocking particular URL form robots.txt
|
love2respectseo
msg:4119857 | 6:35 am on Apr 22, 2010 (gmt 0) |
Hello,
I have heard a lot about the blocking of folders and sub-folders from robots.txt file and i have used them a lot, but I still can't find the way to block a particular URL there and also if i have a particular folder to block which exists in a tree, means main-folder/fold1/fold2/fold3/fold4 and i am trying to block just fold4 folder. What to do? and if i have dynamic URLs and to block them what i have to do?
Can anybody suggest it?
|
tangor
msg:4119859 | 6:46 am on Apr 22, 2010 (gmt 0) |
If that link can be found off site (a link to your page via another site) you will be indexed. Nature of the beast.
Meanwhile, visit the Apache forums or use the new advanced search at WW for robots.txt for other pointers.
Do realize that your robots.txt is also a blueprint for robots (or scrapers) to find content on your site... the baddies will ignore instructions and go straight for the goodies.
|
|
|
