Forum Moderators: open

Message Too Old, No Replies

MLBot from Metadatalabs

         

incrediBILL

12:30 am on Apr 22, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Claims they are "building an index of media on the web" and use robots.txt but I didn't notice if they did or didn't.

71.41.200.* "MLBot (www.metadatalabs.com)"

I've seen it attempt to hit my site from a couple of IPs in that block almost daily since 3/1/08 and they're crawling from a RoadRunner business connection so they probably aren't crawling too far or too fast at that speed.

What I find most amusing is it already knew about pages on my site which they've never been allowed to crawl.

keyplyr

5:08 am on Apr 22, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



After they took a few pages at one of my sites a couple months ago, I checked them out. For some reason I decided they were not beneficial to me and I denied them via robots.txt.

Since then, they come, read robots.txt and leave.

wilderness

11:41 pm on Apr 22, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



IF you dns the site owmer and then search google for same name?
You'll see a blog which is on-topic.