Welcome to WebmasterWorld Guest from 54.196.244.45

Forum Moderators: goodroi

Message Too Old, No Replies

How to disallow by specific keyword?

     
9:45 am on Mar 27, 2014 (gmt 0)

Full Member

5+ Year Member

joined:June 26, 2008
posts:229
votes: 2


I want to block all pages on my site that include the word /tabid/

So if I add Disallow: /tabid/ to my robots.txt will this disable all URLs levels for example:

www.site.com/folder/tabit/folded.aspx
www.site.com/folder/folder/tabid/file.aspx
www.site.com/tabid/folder/folder/file.aspx
etc..

Or what is the correct way to Disallow by keyword - in this case "tabid", located in different places in my URLs?

Thanks!
11:17 am on Mar 27, 2014 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


the Disallow directive matches the specified pattern from left-to-right.

for those bots which support wildcard extensions you can use:
Disallow: *tabid
12:20 pm on Mar 27, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:6148
votes: 281


As pranque said, correct, just bear in mind that not all spiders are that smart (ie, wildcards). If concern is Google, Bing or Yahoo, then you are good.
12:29 pm on Mar 27, 2014 (gmt 0)

Full Member

5+ Year Member

joined:June 26, 2008
posts:229
votes: 2


Many thanks!
7:04 am on June 11, 2014 (gmt 0)

New User

joined:Feb 16, 2013
posts: 7
votes: 0


Hi,
There is no another way to block the URl'S. You can just remove the tabid then the URL are dissapled.