Blocking Ask spider

Pulls hundreds of thousands of pages, delivers zero traffic

[midlifecrisis]

The subject sums it up.

In March alone Ask spidered 400,000 pages from one of my sites alone, yet none of the pages show up in the index. Pages from other sites are in the index but are hopelessly outdated, between 6 and 12 months old. From my logs there is literally zero referrals from Ask at all.

Comments anyone?

[BillyS]

Ask seems to add to their index in chunks. My site is much smaller, but in one month's time it went from 85 page to over 400 (still only about half the site). It took them about three months of spidering the full site before they let half the pages in.

Not sure why you dont get any referral from ASK. They send me nearly as much traffic as MSN.

[martinibuster]

I may be mistaken, so someone correct me if I'm wrong, but I'm fairly certain that AJ incorporates new websites and (maybe new content from sites already in the index) at least three months later, as if there's a quarterly update. This may have changed, but I haven't heard about it.

Not sure why you dont get any referral from ASK.

Could be the topic matter. Some topics don't feature ten AdWords ads above the fold.

They've recently backtracked and promised to reduce that amount to seven ads above the fold, but that's still too many. They show more ads on top than Yahoo. Even MSN shows only three ads on top. MSN says too many ads degrades the user experience.

I have to add that showing that many ads on top makes AJ irrelevant to many webmasters because their advertising arrangement causes less organic referrals. A shame, really. They have a great technology and I would love to root for them. But they need to straighten out the excessive advertising units first. Otherwise they're going to stay irrelevant to webmasters.

With the cheap price of bandwidth, might as well let them take your content, they might see the light down the road and decide to do the right thing.

[midlifecrisis]

mb, I checked the ads for some of my relevant kw's (for which I rank in top 3 positions) and there are no more than three above the fold. All my sites are technically oriented stuff so maybe Ask is just not the right environment.

Regarding the time frame I'll wait another couple weeks. Only yesterday I noticed that the 400k site is spidered yet again heavily by Ask, this time pulling about 10K pages a day.

[newbieatseo]

Yeah I was hammered in March to the tune of 1 million hits and around 15 gig bandwidth usage but very few of my pages are currently listed. I thought I might have to ban that spider in April but it calmed down and didn't suck up my bandwidth. I think I used 18gig overall in March and I am on a 6gig a month plan :)

[midlifecrisis]

Luckily I moved to dedicated servers in April (2 TB/month limit). Highly recommended if you are approaching that kind of volume. Also dedicated servers give you much more flexibility with scripts, cgi's etc.