MyCrawler

What is it? ban it?

[guitaristinus]

Downloaded whole site yesterday. IP address from China.

There is a website at mycrawler.com. Looks like it is in Hebrew and is a parking spot for the domain.

Doing a Wayback search (archive.org) on domain shows that the site advertised a crawler program called MyCrawler on Jan 21, 2002. Had a 404 link going to tucows.com. Googled site:tucows.com MyCrawler. Nothing at Tucows. From Sept 28, 2002 the domain was either parked or showing ads for prescription drugs.

Searched Yahoo and found that it was being downloaded from softlookup.com. "MyCrawler Pro lets you search and retrieve information and files from Web sites." Last update of the program was in 1999.

Published by an Israeli company called Teleducation and Internet Ltd. Unable to locate their address, teleducation.co.il, at this time.

Search on MSN found that it was being (file is no longer there) downloaded from filelibrary.com. "MyCrawler Pro Pro 3.0 Collect and download information from Web sites. Shareware WIN95/98/NT/2000." Dated Aug. 10, 2000.

Will ban it unless someone can give reason not to.

[volatilegx]

Remind me not to get on your bad side. You'd have my social security number and mother's maiden name inside of 10 minutes :)

[blend27]

what user agent did it use to do so?

[guitaristinus]

blend27,

The user agent is MyCrawler. Here is a line from my log:

166.111.249.135 - - [23/May/2005:12:22:33 -0400] "GET /directory/ HTTP/1.1" 200 8192 "-" "MyCrawler"

It didn't check robots.txt. Banned with .htaccess.