Welcome to WebmasterWorld Guest from 54.146.201.80

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Mirar Search Indexing Agent

adware/spyware toolbar active now

     
9:56 am on Jan 7, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 21, 2005
posts:379
votes: 0


Previous thread:
[webmasterworld.com...]

Full UA:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent see getmirar.com)

IP: 64.128.107.nnn belonging to Time-Warner Telecom and Net-Nucleus.

Requested our default home page, but no robots.txt.

5:15 pm on Jan 10, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:June 3, 2002
posts:566
votes: 0


Tried to spider all DMOZ-listed directories only.
9:06 pm on Jan 10, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 21, 2005
posts: 379
votes: 0


This bot has returned several times since I made the original post - always only taking the default home page.

The sites I am seeing it in are listed in dmoz and I haven't seen it in sites not listed - so your comment rings true.

9:57 pm on Jan 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


I just saw it too and had the same experience as bull. It didn't request robots.txt for any of the sites. Its behavior makes it seem as if it thinks since the URL is in DMOZ it must not be disallowed in robots.txt so it just skips the checking part and takes what it wants. I wonder if that's a valid assumption? Still it would be courteous to at least read the robots.txt file so it doesn't wind up as the topic of discussion here in a somewhat negative way. :)