Forum Moderators: open
Do you follow robots.txt protocol? What if I want my business to be excluded from your directory?
The robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable.
We do respect and follow Robots Exclusion Protocol.
If you do not want to be listed in our directory, e-mail a request to our custumer support or use a robots.txt to disallow our robot to crawl our website content by adding following text to your robots.txt file:
User-agent: SnookBot
Disallow: /
Surprised nobody else has noticed this thing crawling as they've been very busy in the last couple of years.
