homepage Welcome to WebmasterWorld Guest from 54.166.173.147
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
T_Rex
littleman




msg:395636
 7:01 pm on Mar 29, 2001 (gmt 0)

Looks like Lycos is actually crawling. Lycos_Spider_(T-Rex) is requesting pages I haven't submitted in months.

 

PeteU




msg:395637
 12:02 am on Mar 30, 2001 (gmt 0)

yep, started on tuesday reqesting some of my oldest domains, now is pretty much full spidering run, looks very familiar "no deep spider but follow crossdomain links" thing like some others do

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved