homepage Welcome to WebmasterWorld Guest from 23.22.173.58
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to disallow by specific keyword?
virtualreality

5+ Year Member



 
Msg#: 4657863 posted 9:45 am on Mar 27, 2014 (gmt 0)

I want to block all pages on my site that include the word /tabid/

So if I add Disallow: /tabid/ to my robots.txt will this disable all URLs levels for example:

www.site.com/folder/tabit/folded.aspx
www.site.com/folder/folder/tabid/file.aspx
www.site.com/tabid/folder/folder/file.aspx
etc..

Or what is the correct way to Disallow by keyword - in this case "tabid", located in different places in my URLs?

Thanks!

 

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4657863 posted 11:17 am on Mar 27, 2014 (gmt 0)

the Disallow directive matches the specified pattern from left-to-right.

for those bots which support wildcard extensions you can use:
Disallow: *tabid
tangor

WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4657863 posted 12:20 pm on Mar 27, 2014 (gmt 0)

As pranque said, correct, just bear in mind that not all spiders are that smart (ie, wildcards). If concern is Google, Bing or Yahoo, then you are good.

virtualreality

5+ Year Member



 
Msg#: 4657863 posted 12:29 pm on Mar 27, 2014 (gmt 0)

Many thanks!

Hemanth Malli



 
Msg#: 4657863 posted 7:04 am on Jun 11, 2014 (gmt 0)

Hi,
There is no another way to block the URl'S. You can just remove the tabid then the URL are dissapled.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved