RandomDot - 5:25 pm on Sep 30, 2007 (gmt 0)

The basic model with robots.txt and .htaccess to control access and use of the different content in various ways on a website has a flaw - it's either/or - there is not an if/and - especially not against robots.

With regards to the reviews of restaurants - google has a conflict of interests - since they're building the same business - just on a global scale - what the other website is using on a local basis - they're building on a global basis - google is nice to provide links back - because .. it's their pipeline so their system keeps updated based on the other persons content... again, who is going to earn most on it -(either in service, revenue or anything else) - guess a few times.

Keep in mind that google is smart enough to give that link back and give them something back .... don't know anybody else who is doing it, so they're the best option currently available - wouldn't want to cut the pipeline before something better shows up - but then again - google might just be a little roman who knows that infrastructure and resources is the key of any empire - and that you in generel should make sure that your population is happy, entertained and well fed.