homepage Welcome to WebmasterWorld Guest from 54.242.241.20
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Order of lines in robots.txt file
Can you control a bot with the order of the lines?
bouncybunny

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3090841 posted 9:59 pm on Sep 20, 2006 (gmt 0)

This is a sample from my robots.txt file.

User-agent: *
Disallow: /folder/stuff/
Disallow: /folder/stuff/
Disallow: /folder/stuff/

User-agent: Googlebot
Allow: /folder/*action=this_link
Disallow: /folder/*action=
Disallow: /folder/*other_stuff

What I am trying to achieve is this;

I want Google to follow links with "action=this_link" in the URL. But I want to block all other links with the term "action=" in the URL.

Am I right in thinking that, if I put the 'Allow' line first, that I will achive my aim?

Much appreciated.

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3090841 posted 3:05 pm on Sep 22, 2006 (gmt 0)

Using this robots.txt:
User-agent: *
Disallow: /folder/stuff/
Disallow: /folder/

User-agent: Googlebot
Allow: /folder/*action=this_link
Disallow: /folder/*action=
Disallow: /folder/*other_stuff

_______________________________

Will give you this result with Google:
site.com/folder/action=this_link Allowed
site.com/folder/action=that_link Blocked
site.com/folder/action= Blocked
site.com/folder/other_stuff Blocked

bouncybunny

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3090841 posted 3:37 am on Sep 23, 2006 (gmt 0)

Thanks.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved