homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Blocking particular URL form robots.txt

Msg#: 4119855 posted 6:35 am on Apr 22, 2010 (gmt 0)


I have heard a lot about the blocking of folders and sub-folders from robots.txt file and i have used them a lot, but I still can't find the way to block a particular URL there and also if i have a particular folder to block which exists in a tree, means main-folder/fold1/fold2/fold3/fold4 and i am trying to block just fold4 folder. What to do? and if i have dynamic URLs and to block them what i have to do?

Can anybody suggest it?



WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

Msg#: 4119855 posted 6:46 am on Apr 22, 2010 (gmt 0)

If that link can be found off site (a link to your page via another site) you will be indexed. Nature of the beast.

Meanwhile, visit the Apache forums or use the new advanced search at WW for robots.txt for other pointers.

Do realize that your robots.txt is also a blueprint for robots (or scrapers) to find content on your site... the baddies will ignore instructions and go straight for the goodies.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved