homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
T_Rex
littleman




msg:395636
 7:01 pm on Mar 29, 2001 (gmt 0)

Looks like Lycos is actually crawling. Lycos_Spider_(T-Rex) is requesting pages I haven't submitted in months.

 

PeteU




msg:395637
 12:02 am on Mar 30, 2001 (gmt 0)

yep, started on tuesday reqesting some of my oldest domains, now is pretty much full spidering run, looks very familiar "no deep spider but follow crossdomain links" thing like some others do

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved