Spiders excluding from a second class domain

how to avoid indexing & spidering of specific urls

[billy_t9]

Actually my problem is this..how can I avoid to be indexed and spidered all the sites that are from a url of this type
http:// [big]something. [/big] domain.com/ [big]more_urls [/big]

Also due to my AKAMAI server based content every time someone does not end my domain url with the slash "/" automatically goes through the AKAMAI servers to this url http:// [big]something.[/big]domain.com/[big]more_urls[/big]
ANY clue how to solve this problem?

[Brett_Tabke]

I assume the problem is that all your thrid level domains feed into the same directory as your main domain? hence, a robots.txt would only work if you wanted all of them blocked?

How are you determining that it is a third level domain? Isn't there some way to control which robots.txt is returned to the spider?

[billy_t9]

well according to AKAMAI if someone does not enter the exact url (i.e. [mydomain.com...] it turns him to a url of this type [[b]origin.mydomain.com[...]
where this type of domain is the exact as the first one but without the www so if someone links my site with no slash at the end any of my urls then it goes to the origin(type) url.
This domain actually is the origin where AKAMAI servers use to get the content and to serve it to the rest of the servers that they have around the world so to serve it with no geographical delay

[billy_t9]

Actually you can browse all over my site without using the "www" but the "origin" and while using the origin you have and the most update content of my site as at this url [origin......] are parameters where I have put to get at specific intervals the content for a refresh of the AKAMAI servers and to reflect at the GOOD url

[Brett_Tabke]

Sounds like wildcard dns (always bad in relation to search engines). That looks like one you'll have to take up with the host. I don't see anyway on the surface of dealing with it.