Welcome to WebmasterWorld Guest from 54.157.200.1

Forum Moderators: goodroi

Message Too Old, No Replies

How to disallow by specific keyword?

     
9:45 am on Mar 27, 2014 (gmt 0)

Full Member

5+ Year Member

joined:June 26, 2008
posts:248
votes: 5


I want to block all pages on my site that include the word /tabid/

So if I add Disallow: /tabid/ to my robots.txt will this disable all URLs levels for example:

www.site.com/folder/tabit/folded.aspx
www.site.com/folder/folder/tabid/file.aspx
www.site.com/tabid/folder/folder/file.aspx
etc..

Or what is the correct way to Disallow by keyword - in this case "tabid", located in different places in my URLs?

Thanks!
11:17 am on Mar 27, 2014 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10749
votes: 43


the Disallow directive matches the specified pattern from left-to-right.

for those bots which support wildcard extensions you can use:
Disallow: *tabid
12:20 pm on Mar 27, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:7207
votes: 453


As pranque said, correct, just bear in mind that not all spiders are that smart (ie, wildcards). If concern is Google, Bing or Yahoo, then you are good.
12:29 pm on Mar 27, 2014 (gmt 0)

Full Member

5+ Year Member

joined:June 26, 2008
posts:248
votes: 5


Many thanks!
7:04 am on June 11, 2014 (gmt 0)

New User

joined:Feb 16, 2013
posts: 7
votes: 0


Hi,
There is no another way to block the URl'S. You can just remove the tabid then the URL are dissapled.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members