DogPile Directory

where's the data come from?

[bobriggs]

First, I put a post in alternative search engines about AV a few days ago, and it was moved to PFI or something like that. So, apologies first if this is in the wrong forum. I used to go to /home.htm and now I watch /active.cgi so I'm not as familiar with each of the forums. I'm going to start from /home.htm to figure out what goes where.

That being said, I found a semi-parked domain that I own in the dogpile directory. When I say semi-parked, it is parked, but I used one of the silly create the page scripts and put some stuff on it... I guess you know what I'm talking about. - Just the under construction stuff...

I found it in the dogpile directory. But nobody but me knows that the domain even exists. THE ONLY WAY DOGPILE COULD KNOW about the domain is from the registrar, or crawling the registrar's database.

I'm just curious. I'm pretty good at looking at the logs, id'ing spiders, and knowing what to expect from there.

But what does a DogPile spider look like? Really? Where does it get its data? Inktomi? I know that dogpile is a meta-search engine, however, this site is in the DOGPILE DIRECTORY, (ie not a result from another search engine) never submitted, really just parked with a 'this site coming soon' or something.

How in the world do you get into the dogpile directory?

[mbauser2]

Go to dogpile.com and click on one of the Directory topic links like, say, "Auto". When you get to the "Auto" page, look in your browser's address bar. It'll say "www.looksmart.com". That's probably where the directory results come from.

Are you the original registrant of your domain? It might have gotten indexed in a "previous life" when somebody else owned it.

[bobriggs]

Oops, I see now what you're saying.

However, what I'm talking about is Dogpile Web Catalog results. It shows up after ODP/dmoz and ah-ha, but before Kanoodle, sometimes after Kanoodle. After you click the button for 'try the next x engines'

Something like:
Search engine: Dogpile Web Catalog found x results...

[mbauser2]

Search engine: Dogpile Web Catalog found x results...

The words "Dogpile Web Catalog" are a link directly to the Dogpile Web Catalog [dpcatalog.dogpile.com], which has link entitled About this service [dpcatalog.dogpile.com], which explains it.

Oh, look: It's Thunderstone [search.thunderstone.com], the robodirectory. (The "texis" in the URLs is a dead giveaway.) Why does anybody waste money on that thing?

[TallTroll]

Hey, I got a Thunderstone referral just the other week (which considering the hammering the spider has been handing out, isn't a particularly good ROI)

I'm glad I DON'T pay for it (that would upset me)