WebIndexer/1-dev

Seems to be trying to grab DMOZ listed sites....

[bobothecat]

Have noticed this bot hitting pages/sites that have a DMOZ listing. Noticed it on several of mine, seems to use the same IP, which resolves to Netsonic.

66.180.173.43 - - [14/Jul/2005:14:19:12 -0500] "GET /robots.txt HTTP/1.0" 403 208 "-" "WebIndexer/1-dev (WebIndexer; [e-mail]; [e-mail])"

[Dijkgraaf]

Yes, the same bot has visited my site, requesting the home page twice, the second time after 15.5 hours of the first time.
It does request robots.txt and understands a 301 response.
Other than I can't say more about its behaviour until it starts a deeper crawl.
Same IP address as yours.

[Dave_A]

It could be someone trying to set up a search engine and trying to use DMOZ as a place to start finding websites from.
They all have to start somewhere I suppose? Allowing a spider to follow links found would seem a little wasted really and depending on what website you start from you could end up with different maps of the web.

[GaryK]

From their website: WebIndexer creates a searchable index of the World Wide Web starting at the URL that you specify.

Home Page:
[personalmicrocosms.com...]