Mobile & Desktop chat app, often plugged into Facebook, but why is it crawling a web site? Also took favicon.ico for each page request.
lucy24
8:48 pm on Jun 22, 2016 (gmt 0)
Also took favicon.ico for each page request.
Often this means that the human at the other end will see a favicon that they can click on instead of just text. It's one of those subliminal things that you generally want to encourage because it tends to make the link look more attractive to the potential visitor.
Was it a full top-to-bottom spidering or did they ask for selected pages out of the blue?
Does this specific site's HTML include an explicit favicon meta on each page? (If not, asking for a favicon each separate time is a bit dopey because it's obviously going to be the same for the whole site.)
keyplyr
8:58 pm on Jun 22, 2016 (gmt 0)
I agree with that, however this was a bot. Asked for just a few pages, possibly shared by members of the WhatsApp community.
I also used to think a favicon request gave credence to the possibility the request was human... but go over to GitHub and test drive some of the offerings. Much of the free scraping software includes favicon requests :)
keyplyr
8:06 pm on Aug 2, 2016 (gmt 0)
I've been seeing the same WhatsApp hitting HTML pages. I assume users are sharing these pages and the WhatsApp UA is verifying. Does not appear to be a linear crawl.
