Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Blocking Ask spider

Pulls hundreds of thousands of pages, delivers zero traffic



4:55 pm on May 9, 2005 (gmt 0)

10+ Year Member

The subject sums it up.

In March alone Ask spidered 400,000 pages from one of my sites alone, yet none of the pages show up in the index. Pages from other sites are in the index but are hopelessly outdated, between 6 and 12 months old. From my logs there is literally zero referrals from Ask at all.

Comments anyone?


10:53 pm on May 10, 2005 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Ask seems to add to their index in chunks. My site is much smaller, but in one month's time it went from 85 page to over 400 (still only about half the site). It took them about three months of spidering the full site before they let half the pages in.

Not sure why you dont get any referral from ASK. They send me nearly as much traffic as MSN.


12:01 am on May 11, 2005 (gmt 0)

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

I may be mistaken, so someone correct me if I'm wrong, but I'm fairly certain that AJ incorporates new websites and (maybe new content from sites already in the index) at least three months later, as if there's a quarterly update. This may have changed, but I haven't heard about it.

Not sure why you dont get any referral from ASK.

Could be the topic matter. Some topics don't feature ten AdWords ads above the fold.

They've recently backtracked and promised to reduce that amount to seven ads above the fold, but that's still too many. They show more ads on top than Yahoo. Even MSN shows only three ads on top. MSN says too many ads degrades the user experience.

I have to add that showing that many ads on top makes AJ irrelevant to many webmasters because their advertising arrangement causes less organic referrals. A shame, really. They have a great technology and I would love to root for them. But they need to straighten out the excessive advertising units first. Otherwise they're going to stay irrelevant to webmasters.

With the cheap price of bandwidth, might as well let them take your content, they might see the light down the road and decide to do the right thing.


7:31 am on May 11, 2005 (gmt 0)

10+ Year Member

mb, I checked the ads for some of my relevant kw's (for which I rank in top 3 positions) and there are no more than three above the fold. All my sites are technically oriented stuff so maybe Ask is just not the right environment.

Regarding the time frame I'll wait another couple weeks. Only yesterday I noticed that the 400k site is spidered yet again heavily by Ask, this time pulling about 10K pages a day.


2:56 pm on May 13, 2005 (gmt 0)

10+ Year Member

Yeah I was hammered in March to the tune of 1 million hits and around 15 gig bandwidth usage but very few of my pages are currently listed. I thought I might have to ban that spider in April but it calmed down and didn't suck up my bandwidth. I think I used 18gig overall in March and I am on a 6gig a month plan :)


5:10 pm on May 13, 2005 (gmt 0)

10+ Year Member

Luckily I moved to dedicated servers in April (2 TB/month limit). Highly recommended if you are approaching that kind of volume. Also dedicated servers give you much more flexibility with scripts, cgi's etc.

Featured Threads

Hot Threads This Week

Hot Threads This Month