Forum Moderators: open
:D
Merchants who meet Froogle's criteria and want to submit a data feed must allow Google to crawl their websites by removing any robots.txt files that disallow our Googlebot spider.
Also talked about here [webmasterworld.com].
Agent: googlebot
Disallow:
There's a *huge* difference between allowing googlebot into a specific directory where the images are, and giving it free run over the enitre web site.
I have plenty of content that googlebot doesn't need to bother itself with, but have removed the restriction on the images folder to appease them. If that's not good enough, I guess I'll rethink whether I really need a froogle feed or not.
Why would they possibly need *complete* unrestricted access to the site anyway?