Forum Moderators: open

Message Too Old, No Replies

Mixed-up Spidering

Yahoo spiding main site using subdomain url

         

anchordesk

3:36 pm on Apr 27, 2004 (gmt 0)

10+ Year Member



Curious if anyone else is seeing this. I have one web site with about 40 pages. All internal linking is absolute (e.g. http:*/www.mydomain.com/folder/something.html)

I also host a small non-profit subdomain not related to the main site located on the same domain which gets spidered independently of the main site. (e.g. http:*/uniquename.mydomain.com).

Both subdomain and main site have been in existence peacefully for about 8 years.

Since yesterday, Yahoo has been trying to crawl my main site files using the subdomain instead of www. Everything 404's. Quite odd how Yahoo crossed the subdomain with the main site when everything is hard linked absolute. Someone else linking to my site incorrectly using the subdomain does not appear to be an issue.

Any ideas what's going on?

soapystar

4:12 pm on Apr 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



yes, this has been posted before. Almost as though its checking up on you to see if your domains are setup identically. I.e. identical sites just the domain name being different but internal structure identical.

anchordesk

4:17 am on May 8, 2004 (gmt 0)

10+ Year Member



I think it's a Yahoo bug. Been 10 days now and Yahoo keeps requesting crossed, non-existant URLs more than valid URLs.

soapystar

8:54 am on May 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



why would you assume its a bug?...Yahoo are spam paranoid!

mars9820

9:46 am on May 8, 2004 (gmt 0)

10+ Year Member



I guess his log files also get paranoid...LOL.