Why did AltaVista spider 160,000 pages

Recently joined the "light web"

[mtaco]

Why did AltaVista spider 160,000 pages if it wasn't going to do anything with them?

Just grumping mostly, but its been two months now since I changed the site to use spiderable links, and one week after I did that, AV pulled all 160,000 pages from our site ( I wrote a perl script to check, and the links were unique).

We still only have the old 30 pages in altavista though, with the old style links.

Grump. Grump. Grump. Yet the have an up to date copy of the home page.

[jeremy goodrich]

Perhaps one day, AV will return to former glory, however, every time somebody at WebmasterWorld asserts, "AV will make a come back" it seems that hope is ebbed further away...

At one point, I even took the time to ban their spiders from my sites (bandwidth drain) but these days, I hardly notice or care. ;)

[mtaco]

So basically, their spiders a running around the web slurping up sites, but not actually doing anything with the data?

Huh. So how come AV slurps 160,000 pages, but "All The Web" did less then Google?

Ah well. Can't wait until the grub.org guys get a Mac OS X version of their client running, so I can throw 160,000 pages at them. :-)

[bird]

Typically, it will take AV something between two and four months to actually index the pages they spidered. At least that was the rate last time I checked. So you might be just a little too impatient in this case.

[heini]

AV has in the top serps a very fresh index, likely the freshest of all engines. So it looks like their policy is to keep top listings very fresh, and negelecting freshness for secondary results a bit.