Hosting experts please read & advise

hosting problem

[borkboing]

Hi, thank you for taking the time!

I have a problem with a hosting situation.

I have found out that search engine spiders are following my main domain in search of pages within the subdomains.

For example:
The main url would be: [maindomain.com...]
I want to spider: [subdomain.com...]
MSN is spidering [maindomain...]

The latter example does not exist, and to make it exist by dropping pages and graphics in the maindomain.com would be of no benefit to the subdomains (the spiders need to go to [subdomain.com...]

I have a 4 year contract for renting a server, and am locked into a hosting company that does not know what to do, and I need to turn to someone outside for help.

Is there a robot script I can use? Is there a setting I can tell my hosting company to use? Can anyone assist?

[ogletree]

Spiders just follow links. If you want something spidered put a link to it. If you have a domain that you do not want spidered put it in your robots.txt. If you want a page not spidered put a noindex nofollow tag on it. I'm a little confused by your examples. Are you talking about 2 domains or a domain and a subdomain.

www .domain.com
sub .domain.com

or

www .subdomain.com
www .maindomain.com

[borkboing]

That is a good question, it is
www.subdomain.com
www.maindomain.com

It is confusing to me as there are no links between the two that I have put in. The main site is www.example.com, and an example of a subdomain is www.example.co.nz.

I found MSN spidering
[example.com...]

It should be spidering [example.co.nz...]

[edited by: txbakers at 10:37 pm (utc) on Oct. 24, 2004]
[edit reason] no specific URLS please [/edit]

[ogletree]

There are lots of ways for SE's to find URL's. Just dissallow it and wait for it to be found.

[borkboing]

That is normally sound advice, but the subdomain sites would have all long ago been listed by the search engines. Instead, not a one of them is on Google or Yahoo.