lucy24 - 12:38 am on Sep 19, 2013 (gmt 0)
Anyone know anything about meanpath dot com and/or the meanpathbot? Forums search is entirely silent :(
Its most recent crawl was from an from OVH range, meaning it was a priori blocked except for robots.txt. It did ask -- but it went on to ask for the front page of my test site, which is 100% roboted-out. As in:
"Which part of
User-Agent: *did you not understand?"
I don't know if it would try to dig deeper, if permitted. Equally important, I can't tell if it's serving any good and worthy purpose. This is assuming for the sake of discussion that a legitimate new search engine is a "good and worthy" thing. ymmv on this point. But here I'm not sure of its legitimacy in the first place.