What does Webcrawler now do?

[Marcia]

I've seen Webcrawler categorized as a meta-search. I'd like to verify this, and if so, from what sources are results being gathered?

Is Webcrawler actually being used?

[rcjordan]

>being used

For me, it's definitely a fifth-tier SE, coming in at 1% for an old site that's been out there for years.

[JamesR]

I am actually getting a trickle of traffic from users drilling down a few levels in the directory (LookSmart) and clicking on my site. It was quite a pleasant little surprise when I first saw the referers. I guess the directory is the freshest data on the portal but I can't figure out why people would bother. Maybe old habits die hard. (I take that back...they are just following the directory links showing up in the search results)

[djuice999]

webcrawler just sits around the house these days and eats bon bons. She isn't good for much but sweet memories.

[Marcia]

>sweet memories

Close. Comes around about one time a month, whispers a sweet nothing and leaves.

[grnidone]

I can't imagine that webcrawler can afford to hire people to work for them. I haven't heard of anyone that uses webcrawler, so I would think that they have a hard time getting advertising money.

Is webcrawler somebody's server in a garage that they have "on the side"?

-G

[djuice999]

grnidone,

Webcrawler is now owned by excite.com. It is the same thing as excite, except for the ranking algo, and the way results are displayed. But the database is the same. I think excite has only one person working on the old crawler these days just dusting it off in the mornings. They use the most out dated database too. They don't update any more.

[austtr]

I recall seeing a post somewhere a long time ago by an ex employee of Webcrawler who said that the first thing Excite did when they acquired Webcrawler was to terminate all the staff.... and that it was never intended to be kept as a functioning entity.

Nobody seems to know (or at least make public) why Webcrawler is still there. It seems to be pretty much irrelevant, both for listing of sites and delivering traffic. Maybe some Excite results come from the Webcrawler database?

Why it continues to offer a submission form, thus implying it will do something with the submitted data, is a mystery.

[CaveToad]

Perhaps without a staff, and with the help of a paid for by excite server, the little ol' crawler can support itself financially through the meager traffic it gets. Someone is probably still using it out there. If only a trickle. And with minimal outlay of cash, by using hand-me-down databases from Excite, they can just afford to let it run and take what little they may from it financially.

[tedres]

[...first thing Excite did when they acquired Webcrawler was to terminate all the staff...]

If it was me you're thinking of, I mentioned that when Excite acquired Magellan/McKinley, most of us were laid off within a few months. I was there when we acquired WebCrawler (still have the t-shirt too!) I think because WebCrawler was a spider, not a directory, they had a smaller and more tech oriented staff which continued to work on the WC project. It's probable that they merged their efforts into improving Excite and kept WC alive, but in limbo.

BTW, Brian Pinkerton, the original Webcrawler programmer, went back to school and last year completed his PhD documenting the Webcrawler project. Some folks might be interested in reading his dissertation. - See: [thinkpink.com...]

[backus]

I was never much of a webcrawler fan, but I always like Excite. Recently, I feel they have let me down somewhat.

[CaveToad]

Originally when it first appeared on the scene I really like it. It was quick and relavant to what I wanted usually. Then down came Excite and washed WebCrawler out.
maybe someday it'll be back but I doubt it.
How 'bout a garage sale for these has-been search engines, you could pick up a few unused engines with decent algos, infoseek, webcrawler, go, and have yourself a nice little engine with it all.

[CaveToad]

dupe. d'oh!