| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Froogle wants my images!|
| 12:47 pm on Apr 6, 2004 (gmt 0)|
I've tried to add a store to froogle, and have recieved an email saying that my robots.txt doesn't allow access to my images folder. Fair enough.
My current rule is:
What I want to know is, if I add
Will that work, or will googlebot 'inherit' the Disallow: /images rule from 'User-agent: *'? Alternatively, is their a better way of doing it?
| 5:41 am on Apr 14, 2004 (gmt 0)|
Why don't you allow the froogle agent access to the images folder and block all else.
| 5:59 am on Apr 14, 2004 (gmt 0)|
Put the more specific record first. Googlebot, and then "*".
Most if not all robots accept the first record that matches their user-agent string.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved