Mojeek crawler is active?

[SumGuy]

I'm blocking 5.102.173.71 in my router (not sure what CIDR it's part of) and I'm seeing today that it was trying to hit my web server. This IP is host-name (crawl-5-102-173-71.mojeek.com).

So I throw mojeek.com into my address bar and it claims to be "a growing independent search engine which does not track you".

I've never heard of it, but I might try it.

I'll probably un-block at least the /24 and see what it does.

I take it that it was known here in the past, but doesn't have much of a profile as an active crawler?

[dstiles]

Mojeek is a valid UK SE with an address in Sussex. I allow:
MojeekBot
from 5.102.173.64/28

[Peter_S]

Hi,

[en.wikipedia.org...]

It's not a big player, but I think this is good to keep an eye on it.

I also noticed a surge of activity , since one month.

[brotherhood of LAN]

Employee of Mojeek here.

Yes, we are very small fry compared to G,B. Mojeek respects robots.txt and rate limits requests. We also provide results to other search engines.

[Peter_S]

Keep up the good job brotherhood of LAN (and your co workers) !

[brotherhood of LAN]

Cheers. The work is mostly of Marc Smith who started it in 2004. He can claim the world's first 'no tracking policy' well before other 'privacy' engines. Appreciate any support for alternative views into the web beyond big tech (like allowing our crawler). Makes for a healthier web.

[lucy24]

doesn't have much of a profile as an active crawler

It's harmless. Within the present calendar year I find a whopping total of 22 Mojeek visits across three sites, each of them robots.txt + one page. One of those was a 410 from ten years back, another was for an URL that has been redirected for pretty exactly two years--both suggesting that they're following some very, very old links.

[mack]

When it comes to blocking anything I would tend to see if you have a reason to do so first. If we were all to block things simply because we don't know of them it would be incredibly hard for any new search company to build an index.

As for Mojeek, I have been watching them for several years and they do exactly what it says on the tin.

Mack.

[Peter_S]

For me, if :

- ua includes RobotID + url to read about it (page must exist)
- access / respect robot.txt
- IP or IP range identifiable by reverse dns, or listed on the about page.

It's welcome :-)

[brotherhood of LAN]

reverse DNS is definitely the finer point, after all anyone can proclaim a UA.

[Peter_S]

reverse DNS is definitely the finer point, after all anyone can proclaim a UA.

Yes, this is a concern I mentioned here [webmasterworld.com...]

[tangor]

Not on my hit list ... has always been respectful!

[lucy24]

Anyone can put on a fake name. But not many UAs are actually spoofed; 99 times out of 100, if someone claims to be {name}, that’s who they are.

Remember when fake Googlebots were everywhere? Over the years they’ve plummeted--to the point where in ten years’ time it may not even be necessary to check for them. There are lots of fake Baidus, for all the good it does them. Other names can probably be counted on the fingers of one hand.

[Peter_S]

This is very true. I wonder where all these scrappers are now. May be some of Google's refines of its algorithms put out of business plenty of these sites which were rehashing scrapped content. However, at the time of the holy AI, there is no more need to scrap, may be.

[jmccormac]

Well behaved and active.

Regards...jmcc