homepage Welcome to WebmasterWorld Guest from 54.224.179.98
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to disallow by specific keyword?
virtualreality




msg:4657865
 9:45 am on Mar 27, 2014 (gmt 0)

I want to block all pages on my site that include the word /tabid/

So if I add Disallow: /tabid/ to my robots.txt will this disable all URLs levels for example:

www.site.com/folder/tabit/folded.aspx
www.site.com/folder/folder/tabid/file.aspx
www.site.com/tabid/folder/folder/file.aspx
etc..

Or what is the correct way to Disallow by keyword - in this case "tabid", located in different places in my URLs?

Thanks!

 

phranque




msg:4657888
 11:17 am on Mar 27, 2014 (gmt 0)

the Disallow directive matches the specified pattern from left-to-right.

for those bots which support wildcard extensions you can use:
Disallow: *tabid
tangor




msg:4657914
 12:20 pm on Mar 27, 2014 (gmt 0)

As pranque said, correct, just bear in mind that not all spiders are that smart (ie, wildcards). If concern is Google, Bing or Yahoo, then you are good.

virtualreality




msg:4657919
 12:29 pm on Mar 27, 2014 (gmt 0)

Many thanks!

Hemanth Malli




msg:4679007
 7:04 am on Jun 11, 2014 (gmt 0)

Hi,
There is no another way to block the URl'S. You can just remove the tabid then the URL are dissapled.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved