Forum Moderators: open

Message Too Old, No Replies

envolk[ITS]spider

         

wilderness

4:09 pm on Sep 6, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



71.102.140.247 - - [06/Sep/2005:06:01:18 -0700] "GET /robots.txt HTTP/1.0"
200 3490 "-"
"envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)"

GaryK

3:47 am on Sep 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Apart from requesting robots.txt was it well behaved?

Dijkgraaf

6:39 am on Sep 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I've had a visit from them as well.
Seems well behaved.

It does request robots.txt again before fetching each page, but various spiders/bots have that bad habbit.

So far it only seems to be requesting pages that are externally linked to, it hasn't done a deep crawl of my site as yet.

wilderness

1:51 pm on Sep 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Apart from requesting robots.txt was it well behaved?

Only grabbed robots and main page of my smaller site.

Don