Welcome to WebmasterWorld Guest from 54.196.212.62

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Mirar Search Indexing Agent

adware/spyware toolbar active now

     

Mokita

9:56 am on Jan 7, 2007 (gmt 0)

5+ Year Member



Previous thread:
[webmasterworld.com...]

Full UA:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent see getmirar.com)

IP: 64.128.107.nnn belonging to Time-Warner Telecom and Net-Nucleus.

Requested our default home page, but no robots.txt.

bull

5:15 pm on Jan 10, 2007 (gmt 0)

10+ Year Member



Tried to spider all DMOZ-listed directories only.

Mokita

9:06 pm on Jan 10, 2007 (gmt 0)

5+ Year Member



This bot has returned several times since I made the original post - always only taking the default home page.

The sites I am seeing it in are listed in dmoz and I haven't seen it in sites not listed - so your comment rings true.

GaryK

9:57 pm on Jan 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just saw it too and had the same experience as bull. It didn't request robots.txt for any of the sites. Its behavior makes it seem as if it thinks since the URL is in DMOZ it must not be disallowed in robots.txt so it just skips the checking part and takes what it wants. I wonder if that's a valid assumption? Still it would be courteous to at least read the robots.txt file so it doesn't wind up as the topic of discussion here in a somewhat negative way. :)
 

Featured Threads

Hot Threads This Week

Hot Threads This Month