homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Blocking particular URL form robots.txt

 6:35 am on Apr 22, 2010 (gmt 0)


I have heard a lot about the blocking of folders and sub-folders from robots.txt file and i have used them a lot, but I still can't find the way to block a particular URL there and also if i have a particular folder to block which exists in a tree, means main-folder/fold1/fold2/fold3/fold4 and i am trying to block just fold4 folder. What to do? and if i have dynamic URLs and to block them what i have to do?

Can anybody suggest it?



 6:46 am on Apr 22, 2010 (gmt 0)

If that link can be found off site (a link to your page via another site) you will be indexed. Nature of the beast.

Meanwhile, visit the Apache forums or use the new advanced search at WW for robots.txt for other pointers.

Do realize that your robots.txt is also a blueprint for robots (or scrapers) to find content on your site... the baddies will ignore instructions and go straight for the goodies.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved