Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
I am going to write a robots.txt exclusion file to address this issue. I believe the right format should be as follows (the URLs listed are not the real ones, just examples):
Would this be the correct syntax in order to have the crawlers not crawl the site under the example URLs listed?
Thanks for your help.
That is not valid robots.txt syntax.
What you need to do is to split those domains out, and give each one its own robots.txt. This can be done, even though they all go to the same hosting account. Then simply disallow all robots from those domains you don't want indexed.
The method used to do this will depend on your server, e.g. Apache or IIS. It's fairly easy with Apache if you can use mod_rewrite.