aol.com crawler = googlebot.com?

[damiengazier]

I have a web site visited every day 200 times by a crawler called aol.com , and i was wondering if it was the same crawler than the one for google (googlebot)?

If not , Can anyone tell me to which search engine this crawler correspond? and how to remove it?

thanks

[crawler216]

why would you want to remove it? its aol's crawler which does the same job google does, just not as good, but still nice to have. im not sure if i can post the url but im sure a mod will snip it if i cant, aol dot com.

[EliteWeb]

damiengazier, welcome to webmasterworld! :) are you sure that it is a crawler, a robot, a spider coming to your site? If possible post the tag from the bot. Tons of my traffic comes from the aol.com domain name but its not to be confused with any bots.

[wilderness]

Isn't AOL still using Inktomi as their spider?

[damiengazier]

I don't known if it is a crawler , but he visit my web site with a lot of IP address like : 205.188.208.74 , 205.188.209.144 ,it is always the same 3 first number . I just wonder what it is and how to remove it? because i think aol.com use google for directory .

Thanks again

[Giacomo]

205.188.208.74 = cache-dc06.proxy.aol.com 
205.188.209.144 = cache-dq10.proxy.aol.com

[webmasterworld.com...]

[EliteWeb]

Aol caches data to their servers to save bandwidth. So when you request a page it is going to be cached. You do not want to limit access to this or else you can render some aol people helpless. :)

[wilderness]

<snip>else you can render some aol people helpless</snip>

We webmasters get blamed for everything else!
Now we have to find a way to enter lined into robots.txt and htaccess to prevent the "hopeless" AOL subscriber from becoming helpless?
Isn't that sort of an oxymoron? They start with a free disk believing there is no consequence to 1000+ free hours for one month. 24 x 30 equals 720 ;)
Somebody else had a hand in their rendering long before AOL :)

BTW isn't it terrible that I should feel this was towards nearly one-third of my traffic!

[kevinpate]

> Aol caches data to their servers to save bandwidth

At times I sure wish AOL had some more storage room for the files it nabs to the cache for the sake of the aol subscribers.

It's a tad annoying to see the same files heading off via an aol proxy in a 2 day window, or on several occasions seeing the same files heading out more than once on a given day.

Not thing one I can do it about though. It's frustrating, but then again, some of them aol users are camping buddies of mine, and one just doesn't pull the plug on someone today if you might need to borrow some matches from them after a downpour on a future trip to the woods.