Welcome to WebmasterWorld Guest from 23.22.220.37

Forum Moderators: goodroi

Message Too Old, No Replies

Order of lines in robots.txt file

Can you control a bot with the order of the lines?

     
9:59 pm on Sep 20, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Mar 10, 2006
posts:661
votes: 0


This is a sample from my robots.txt file.

User-agent: *
Disallow: /folder/stuff/
Disallow: /folder/stuff/
Disallow: /folder/stuff/

User-agent: Googlebot
Allow: /folder/*action=this_link
Disallow: /folder/*action=
Disallow: /folder/*other_stuff

What I am trying to achieve is this;

I want Google to follow links with "action=this_link" in the URL. But I want to block all other links with the term "action=" in the URL.

Am I right in thinking that, if I put the 'Allow' line first, that I will achive my aim?

Much appreciated.

3:05 pm on Sept 22, 2006 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


Using this robots.txt:
User-agent: *
Disallow: /folder/stuff/
Disallow: /folder/

User-agent: Googlebot
Allow: /folder/*action=this_link
Disallow: /folder/*action=
Disallow: /folder/*other_stuff

_______________________________

Will give you this result with Google:
site.com/folder/action=this_link Allowed
site.com/folder/action=that_link Blocked
site.com/folder/action= Blocked
site.com/folder/other_stuff Blocked

3:37 am on Sept 23, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Mar 10, 2006
posts:661
votes: 0


Thanks.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members