Welcome to WebmasterWorld Guest from 54.227.83.19

Forum Moderators: goodroi

Message Too Old, No Replies

Order of lines in robots.txt file

Can you control a bot with the order of the lines?

     

bouncybunny

9:59 pm on Sep 20, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



This is a sample from my robots.txt file.

User-agent: *
Disallow: /folder/stuff/
Disallow: /folder/stuff/
Disallow: /folder/stuff/

User-agent: Googlebot
Allow: /folder/*action=this_link
Disallow: /folder/*action=
Disallow: /folder/*other_stuff

What I am trying to achieve is this;

I want Google to follow links with "action=this_link" in the URL. But I want to block all other links with the term "action=" in the URL.

Am I right in thinking that, if I put the 'Allow' line first, that I will achive my aim?

Much appreciated.

goodroi

3:05 pm on Sep 22, 2006 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Using this robots.txt:
User-agent: *
Disallow: /folder/stuff/
Disallow: /folder/

User-agent: Googlebot
Allow: /folder/*action=this_link
Disallow: /folder/*action=
Disallow: /folder/*other_stuff

_______________________________

Will give you this result with Google:
site.com/folder/action=this_link Allowed
site.com/folder/action=that_link Blocked
site.com/folder/action= Blocked
site.com/folder/other_stuff Blocked

bouncybunny

3:37 am on Sep 23, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Thanks.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month