Slurping But Not Listed in New Yahoo

Why?

[Angonasec]

I'm concerned. We had a top listing (via Google) in the old Yahoo! web results, but we are not even listed in the new Yahoo! search results. Yes, I've read their note about 'don't worry etc.." but I AM worried.

You see Slurp visits our site about *30 times every day*, but only ever visits robots.txt!

There's nothing on our site to inhibit Slurp, though we used to block Inktomi's NiprMil bots on the 198.25 and 198.26 IPs. We allow those now in case that has any bearing on this.

Why does Slurp stop at our robots file?
Here's the footprint.
66.196.65.38 - - [25/Feb/2004:02:17:16 -0500] "GET /robots.txt HTTP/1.0" 200 3848 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

I'm wondering, does this Slurp IP pass on the data from robots text to another Yahoo bot, such as Yahoo Seeker, which then does crawl our site? We get Yahoo Seeker crawling, but I think that is associated with their shopping directory, not the new Yahoo Web Search.

Strangely, two of our old sites that have links to our main domain, *are* listed in the new Yahoo! Search. Concerned that the new Yahoo! search bot might be regarding these as doorway pages, or 'spam', I've removed all links on them to our main site, except one static 'Site moved ... click here ...'

Since our non-commercial site is highly ranked in Google, I'm sure Yahoo would want it in their new search results.

Is there anything I an do to get it listed again, quickly (apart from paying for a listing!)?

[BarkerJr]

Just a thought, that IP is an Inktomi IP.

[internetheaven]

Barker, are you implying that you think the Inktomi spider is about to be squashed?

It would make sense for anyone trying to focus themselves as an individual search engine would develop a new crawler too. Maybe the fact that Yahoo isn't pure Inktomi results will now be reflected by the creation (or the release of an already created) of the Yahoo spider.

[helenp]

welcome.........
You arenīt alone with that problem.
yahoo/slurp spider my robots.txt daily and only,
and there are many out there in the same situacion.

[Alternative Future]

Yeah same scenario here, been this way for roughly 7mths now.

Used to be listed in Ink till it found my dot info in place of the dot net, before I had a 301/302 in place. When I used the redirect, the dot net went from the SERP's permanently and the dot info followed slowly afterwards. Went through some months till it started to find the robots.txt with a 200 and now been waiting several months.

Hope this gets addressed shortly...

-George

[Angonasec]

I've had encouraging responses to my Emails to Y! about this, and I'm hoping Tim can fix it when he's recovered from the Pub Conference.

Colin

It's refreshing that Y! are taking notice of our accumulated feedback to get the Y! Bot working properly.

[walkman]

Join the gang :)
robots.txt is taken several times a day. I e-mailed and will see what they say.

I had the .net pointing at it and Inktomi indexed that a while back. 301s are not helping so I gave my feedback.

[Net_Wizard]

I got hundreds of...

Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; [alltheweb.com...]

activities today with IP 66.77.73.xx

I know it's a ATW crawler but I've never seen this fast crawling before, about 10-11 secs apart. As if it was unleashed. Could this be going to YS database?