Yahoo Slurp Visits Many Pages, But Only Take 10K of Content

Google & MSN Take it All... Why Not Slurp?

[catchmeifucan]

one of my clients site used to have more than 2000 pages indexed by Yahoo. early april the site got hit by Yahoo's update and only less than 200 pages left in index. I am looking at the detailed log file today and found Yahoo Slurp is visiting the site, but never fully index each page.

here are some data from sawmill

Spider Hits Page views Visitors Size
1 Googlebot 44,659 38.5 % 44,659 74 1.25 G
2 MSN Robot 40,185 34.6 % 40,185 11 1.26 G
3 Yahoo Slurp 28,770 24.8 % 28,770 951 114.61 M

Notice Yahoo Slurp is visting a lot of pages but only 1/10 of the bandwidth used. I then took a close look at the log detail and found Slurp only eat less than 10k for each page it crawls, compared to 30-40k Googebot and MSN Robot.

What's wrong?

[BillyS]

Slurp takes advantage of compression... do you compress pages with gzip?

[catchmeifucan]

I don't know, how do I know?

[BillyS]

You've got mail