Why Yahoo Slurp is wasting bandwidth

[abhishekkaushik]

What is going on with search engine bots like Yahoo Slurp which are crawling my sites 20+ pages everyday again and again since three months and never include them into their index.

[jeep4x4]

Yahoo hate new domains. Just need wait 3-4 month for index done...

[SEOPTI]

I see the same crawling the same pages over and over like 100 times a day and never including them. I will change my robots.txt and ban Yahoo. They are even sending less traffic than MSN.

[abhishekkaushik]

jeep4x4....my domain is almost a six months old. Perhaps SEOPTI idea is good to ban it extra cost from bandwidth matters....

[vincevincevince]

abhishekkaushik... are you giving out correct Last Modified headers, and responding to If Modified Since?

[abhishekkaushik]

Vince..... My pages are completely static and i have no idea whether my host support "Last Modified configuration", how can i know about it? I heard that this configuration does not matter much and search engines can identify any changes themselves.

[SEOPTI]

I think they don't respect "if modified since". They are just shooting in the dark in order to find at least some useful content for their poor directory.

[edited by: SEOPTI at 1:02 pm (utc) on July 12, 2007]

[abhishekkaushik]

They are just shooting in the dark in order to find at least some useful content for their poor directory.

I don't know they are that conscious about listing creamy content in their directory instead of asking for money.