Forum Moderators: open

Message Too Old, No Replies

what spider is that

anybody can help me identify this spider?

         

MkSlim

2:36 pm on May 8, 2005 (gmt 0)

10+ Year Member



I had crawler.de visit my site yesterday. It hit my home page and my site map, 2 hits. Then left..

anybody knows this spider? I checked out the ip address but it didnt' give me much info

MkSlim

wilderness

2:52 am on May 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



How about a full log line?
There must be more info available than "crawler.de"?

volatilegx

3:16 pm on May 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Here's what I have on crawler.de

# Crawler.de
# UA "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)"
80.182.226.190
195.20.224.72
195.20.224.73

wilderness

4:53 pm on May 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



SetEnvIf User-Agent MSIECrawler keep_out

Span

5:16 pm on May 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



MSIECrawler is not a true web robot. It is usually to be found in site logs if someone bookmarks a page whilst offline. Internet Explorer will then download the page and all links related to it, including links, images, JavaScript and Style sheets, when the user is next online.

[msdn.microsoft.com ]

wilderness

6:06 pm on May 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



MSIECrawler is not a true web robot

I didn't feel it needed an explantion ;)

It's in most UA deny lists.

For the benefit of those not aware?
There are a variety of tools similar to this for keeping bookmarks current.
The majority of browser users are not even sure how to confgure such things and are even less aware that their computer is verifying links in the background, unbeknown to their current activity.

From a webmasters point-of-view, we have a tool (bot) using or bandwidth unnecessarily. Perhaps even for a page the visitor will never return to.

Don

MkSlim

1:45 am on May 10, 2005 (gmt 0)

10+ Year Member



In other words, don't get excited but don't worry about it either... unless it chews up a lot of your bandwidth...

thanks
Mk