homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

t-online.de german isp that spiders? Marvin/1.0
jeremy goodrich

 7:07 pm on Apr 9, 2001 (gmt 0)

This one got me yesterday.

Not too many visits, and the log entries are weird, as though it was making 2 requests in the same second. I believe t-online.de is an access provider, and it looks like they use the german infoseek for their search. Anybody else got more info on this?



 10:27 pm on Apr 9, 2001 (gmt 0)

Dear collegue,
May I respectfully draw your attention to the lively discussion on German search engines in the European forum, which has been split 4 times already and where T-Online and its search engine Infoseek.de (of which they own 25%) is one of the hottest topics. The news that they are spidering will be welcome in my part of the world, believe me.
Jan - your friendly European moderator.

jeremy goodrich

 2:00 pm on Apr 10, 2001 (gmt 0)

Ahh, you spotted my American ignorance! How shameful of me to display it in the open, for everybody to see!

I wish to humbly apologize for ignoring my European brethren, I will be more consciencious in the future, and endevour to visit this forum.


 4:00 pm on Apr 10, 2001 (gmt 0)

Do you have listings in infoseek.de? They have in the past respidered already indexed pages. Activity has been very low though for the last few months. It has been one of the major SEs for Germany, but for the history and the problems of infoseek.de you might want to go to the european fo.....okay.
I could not find that "spider" (if it really is one) in my logs. You might want to ask some of the more experienced members in the eur....yes.


 9:44 am on Apr 23, 2001 (gmt 0)

Marvin/1.0 is the new spider from Infoseek.de because the old one (infoseek-sidewinder) is not fast enough.
The new one hits your site up to ten times a second, and it does a deep crawl (not that deep like FAST or Google, but it does).
Unfortunately IS has many problems with its index, it DOES need new hardware, the place within its db is absolutely full...I directly heard this from a employee of IS.
So Marvin spiders the net, but itīs not possible to get this new pages into the index of IS. Strange.

That could be a reason why T-Online donīt want to continue the partnership with IS: as an owner of IS T-Online has to pay a lot for the new hardware, and itīs well less expensive to pay Google for its results.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved