Hourly requests by Yahoo Slurp

Grabs Robots.txt then Runs - Any Way to Make it Grab my Content?

[pmkpmk]

Hi,

I'm not ranking too good in Yahoo - a thing I want to address for some time now and never found the time to do so.

What's strange though is, that I get almost hourly hits by the Inktomi spider. But all it rquests is robots.txt. Once in an hour it returns, requests robots.txt, and leaves.

Only on very few occasion it requests different files, and never more than 4-5 in a batch.

But usually it simply requests robots.txt and leaves. Isn't that strange? Any explanation?

Or - better - anything I can put INTO robots.txt to make my site more attractive to the spider?

That's my current robots.txt:

User-agent: *
Disallow: /mail_database/

The mail_database line is to catch spiders which violate robots.txt. The original ida was to ban them, but it wasn't worth the effort.

Any ideas?

[martinibuster]

How are your inbound links? I get good results pointing a bunch of links from older established sites that slurp likes.

As an experiment, I just put up a couple sites, without benefit of those links, just a couple directory links and "other" links but slurp is nowhere to be seen.

[pmkpmk]

Thanks for adjusting the subtitle (if it was you).

Inbound links are an issue I neglected for a long time, but I'm working on it as often as my time allows. Most inbound links come from DMOZ and its mirrors and from microsites under my control (and on the same IP - no dup content though). Unfortunately only a few from 3rd party sites, however a few of them are authorities in my niche.