Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: open
Just grumping mostly, but its been two months now since I changed the site to use spiderable links, and one week after I did that, AV pulled all 160,000 pages from our site ( I wrote a perl script to check, and the links were unique).
We still only have the old 30 pages in altavista though, with the old style links.
Grump. Grump. Grump. Yet the have an up to date copy of the home page.
At one point, I even took the time to ban their spiders from my sites (bandwidth drain) but these days, I hardly notice or care. ;)
Huh. So how come AV slurps 160,000 pages, but "All The Web" did less then Google?
Ah well. Can't wait until the grub.org guys get a Mac OS X version of their client running, so I can throw 160,000 pages at them. :-)