Why does a site drop away and Slurp never returns?

[agonyx2z]

I've perused a lot of messages here about Yahoo and like many of you I manage some web sites and one and only one of these has totally disappeared from Yahoo. I'm comparing three different sites, all in the same exact business with similar position in the "real world", all Goliaths and no Davids. I have access to all the raw logs and have looked at each one line by line (after grepping obviously).

Why does a site drop away and never get Slurped? Is this Yahoo's way of saying "Pay us the $299 or go away?" When the owners complained about Yahoo to us in early August, we immediately submitted the site URL without error. I haven't seen a single Slurp there yet.

There is one single thing that makes this site different: it uses the <cough> .NET platform (pages are .aspx) with a lot of Flash. The metas (do they matter to Yahoo?) are right, there are pertinent paragraphs in the beginning of each page, there are about 15 pages total. The site is normally indexed everywhere.

This is not a shady area like pr0n or warez, this is a mainstream area of interest with no sales or catalogue, just brand awareness and info. Imagine a well-know maker of golf equipment, for example. Not that I could name one myself... :)

On the bright side, I am playing with the Yahoo API and I find it very useful, especially the News requests. I just can't see why hundreds of sites that point to the one I ask about are in Yahoo under their keyword and not the actual site with the name in the domain and on every page.

[Vimes]

HI,

What does the site:domainname.com return.

Does Slurp hit the homepage and then / or robots.txt file and leave.

If so you could be suffering from this:

ht*p://search.yahoo.com/search?p=yahoo+penalty+site%3Awww.webmasterworld.com&ei=UTF-8&n=100&x=wrt

and here:
[webmasterworld.com...]

Vimes.

[edited by: Vimes at 6:29 am (utc) on Oct. 19, 2006]

[agonyx2z]

Doing the site: search yields this:

URL not found
We were unable to find any results for the given URL in our index:

Plus, I added 10 sites to the site explorer, all with authorized y_key files. Only two came back 24 hours or so later with "failed". One was due to a redirect problem and the other was this Yahoo un-indexable site. Looking at the server log, I can see my own manual GET of the yahoo HTML file but there was NO GET from Slurp or anyone else other than me, yet it fails saying they can't access this file.

I wrote to the hosting company to ask them if they were by any chance blocking certain ip addresses. This is a dedicated <spit> Windows server by the way. (Oh ya, I already said it was .NET)