homepage Welcome to WebmasterWorld Guest from 54.197.183.230
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Mirar Search Indexing Agent
adware/spyware toolbar active now
Mokita

5+ Year Member



 
Msg#: 3211108 posted 9:56 am on Jan 7, 2007 (gmt 0)

Previous thread:
[webmasterworld.com...]

Full UA:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent see getmirar.com)

IP: 64.128.107.nnn belonging to Time-Warner Telecom and Net-Nucleus.

Requested our default home page, but no robots.txt.

 

bull

10+ Year Member



 
Msg#: 3211108 posted 5:15 pm on Jan 10, 2007 (gmt 0)

Tried to spider all DMOZ-listed directories only.

Mokita

5+ Year Member



 
Msg#: 3211108 posted 9:06 pm on Jan 10, 2007 (gmt 0)

This bot has returned several times since I made the original post - always only taking the default home page.

The sites I am seeing it in are listed in dmoz and I haven't seen it in sites not listed - so your comment rings true.

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3211108 posted 9:57 pm on Jan 11, 2007 (gmt 0)

I just saw it too and had the same experience as bull. It didn't request robots.txt for any of the sites. Its behavior makes it seem as if it thinks since the URL is in DMOZ it must not be disallowed in robots.txt so it just skips the checking part and takes what it wants. I wonder if that's a valid assumption? Still it would be courteous to at least read the robots.txt file so it doesn't wind up as the topic of discussion here in a somewhat negative way. :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved