AOL titles

not from the ODP

[JamesR]

A search for "cheap hotel" on search.aol.com will give you some funny looking titles. This may have been mentioned before, but it seems that AOL does not always use ODP titles for indexing but uses the title tag from the site itself. If AOL has no spiders, how are they doing this? Using their proxy cache? Are they using another source besides ODP for their data?

[seth_wilde]

AOL does use both ODP and Inktomi for there results.

Also AOL does spider and index all of their ODP listings. Although they don't use the title tag when they show the ODP results, your rank is determined by a combination of your ODP title/description/category and your source code weight/prominence.

[JamesR]

Thanks Seth, appreciate the insight. How long have they been doing this? I know Brett has solidly argued against the existence of an AOL spider and I had stopped checking for one. I guess this accounts for all the wacky listings we've seen.

[seth_wilde]

"How long have they been doing this"
I first learned they were using more than just the ODP title/description a few months after they first switched to ODP they've pretty much have always used Inktomi to back up the ODP listings, but lately inktomi listings seem to be showing up more often at the top of serps.

Here's an example-
Try doing a search for "Fischer-Tropsch" now after all the encyclopedia listings you'll see Remote Gas Strategies at #7. If you compare the source code to the dmoz.org listing for this site you'll confirm that this is an ODP listing using an ODP title. You'll also notice that the words Fischer-Tropsch aren't used at all in the ODP title/description/category, but they are used in the source code. So AOL is not only using the ODP info but also the source code info.

"has solidly argued against the existence of an AOL spider"
hmmmm.......I don't know about this, I haven't heard the arguments against an AOL spider.....

[tedster]

AOL pulls LOTS of data with their proxy servers. One single AOL search can result in 6 requests for each file related to a home page. And each request is from a different AOL IP.

Six!

They're doing something with all that info. Seems to me they would hardly need a separate spider with all that coming in.

[redzone]

Haven't seen any evidence of the AOL proxy servers cacheing URL for indexing purposes. We've been optimizing under the assumption that when the URL is reviewed by a DMOZ editor, at that time the page is cached, and placed into the AOL indexing algorithm.

AOL using DMOZ as their entry point into their index, puts a nice, clean SPAM filter into place.

[tedster]

I was just guessing, redzone -- trying out the idea for size. I should make it clear when I'm doing stuff like that.

You're probably right about AOL not using data from their proxy server for the index. It would open them up to too much spam.

However, not all AOL entries are given their ODP descriptions & titles. I'd love to figure out how to get a non ODP description in there!

[seth_wilde]

"not all AOL entries are given their ODP descriptions & titles"

Those are Inktomi listings

[tedster]

Do you know what determines which sites get their Inktomi descriptions? I see sites that are in the ODP, and yet Inktomi listings are used.

[bigjohnt]

...stay tuned for the addition of GoTo.....

[tedster]

Yes, those top three spots should be very interesting. I wonder if AOL will retain the descriptions from GoTo. Sure hope so, we pay for them!

[seth_wilde]

"I see sites that are in the ODP, and yet Inktomi listings are used."

I think your seeing sites with both ODP and Inktomi listings, and their inktomi listing is just being ranked higher (have any examples?)

"I wonder if AOL will retain the descriptions from GoTo."

I'm sure they will, netscape is already showing it's top two results from goto, and they retain the goto description.

[tedster]

>>I think your seeing sites with both ODP and Inktomi listings, and their inktomi listing is just being ranked higher (have any examples?)<<

Thanks for the idea, seth. Now that I've looked closer, what I'm seeing is sites that have several ODP listings. I wasn't on the lookout for that possibility.

You've raised another question for me, however. My understanding was that the AOL listings for "Web Sites" were from ODP and listings for "Web Pages" were from Inktomi. Are you saying that both can show up in one place?

[seth_wilde]

"My understanding was that the AOL listings for "Web Sites" were from ODP and listings for "Web Pages" were from Inktomi."

I had always been under the same understanding. It wasn't until james brought up The example for "cheap hotel" that I noticed Inktomi listings in the "web sites" section. This must be something new that they're doing.

[tedster]

Thanks, seth.

I remember reading james' post, but somehow it didn't sink in until now. I wonder what memory I just wrote over in order to get this info to stick!

[NFFC]

>This must be something new that they're doing

You may be right seth, it looks as if it's related to the number of results that would be returned at DMOZ for an "exact phrase" search, if low then Ink.

Certainly see this on a few domains when searching with regional criteria, sometime the number of results was as low as 4, not anymore. The same goes for Netscape searches, using Netscape Search not Lycos.

[JamesR]

Thanks for the follow up Seth. Definitely seeing many Inktomi pages showing up higher than ODP listing that has no competition in the directory. Expected to show up #1 on many phrases in AOL but are currently ranking behind Ink.

[boyleman]

Those additional listings in the "cheap hotel" example, and others, aren't necessarily coming from Ink. I've researched some of the searches that bring up this non-ODP data, and I've seen sites that are included there that are not in Ink. So I suspect that the data is coming from somewhere other than Ink. I sort of tend to agree with tedster's theory earlier, that they're using data from their cacheing servers. They have all the data, they just need to extract it so they can use it for indexing sites! My theory is that that is where the extra on-page data is coming from in cases where a site shows up for a keyword that is not in the ODP listing. And I also suspect that the extra listings in the search for "cheap hotel" come from there as well, although it seems that they have to filter it somehow, since there would seem to be more of those kind of listings for a given search if they didn't. I think NFFC is pretty close too. My experience has been that when the sites returned from dmoz have too low of relevancy, then these "extra" sites are added in.

[bigjohnt]

O.K. Anybody?

What is AOL using for a backup Database now? Searching the Web Pages, a particular site I am analyzing comes up number one, with a description obviously written by a third party
"With 20+ years experience, this group claims ..yada...yada...yada.."

My guess is the "old" ODP dbase doing the backup,(MAYBE mixed in with Inktomi???) and the description is from an editor who has since revised the description. Would not be total Inktomi - I've never in all my days heard of a site describing itself as this group, those guys, etc...
Any other ideas? More coffee less sleep, more coffee, less sleep..

[NFFC]

>with a description obviously written by a third party

Looksmart would be my guess, search for the description there. Still Ink results though.

[bigjohnt]

NFCC, Your idea that it may be Looksmart triggered a search. The same description is in Magellan and Exite, leading me to believe that it is indeed an older description from Looksmart. Kudos to you! Thanks!. Now, I thought AOL used ODP... they use ODP, AND looksmart, AND Ink, and soon GoTo? ... Way more coffee, WAY less sleep....

My new mantra - More coffee,less sleep, more coffee, les...ZZZZzzzZZZZZZzzz

[seth_wilde]

It looks like we caught the begining of an update. Aol just dropped all percentage rankings (eg..98%) from their listings. This would explain why we're noticing things that we've never noticed before.

[boyleman]

They've definitely made a change to how they display the listings. So far it's hard for me to tell if they've made any significant algo changes, or if it's just the layout and the elimination of the percentages that they've done. As for that stuff we were discussing earlier in this thread - that wasn't just the result of an update happening, because I had noticed it happening for a quite a while. It's just that with most searches you didn't see the wierd additional listings.

[boyleman]

I checked just now, and even though the display format is still different, for the search "cheap hotels", the same phenomenon of non-dmoz sites are showing up in the Web Sites section.