| 7:20 pm on May 29, 2012 (gmt 0)|
Are they actually still crawling, I haven't seen them for a very long time.
Maybe that's why their search results are so poor, at least for the keywords I'm most familiar with.
| 7:30 pm on May 29, 2012 (gmt 0)|
I see scoutjet almost every week sometimes performing a full crawl, but usually just a dozen or so pages.
| 7:46 pm on May 29, 2012 (gmt 0)|
Blekkobot/Scoutjet turned up for the first time in a long time on several sites in the last 24 hours or so.
Another new one was the ahrefsbot.
| 9:30 pm on May 29, 2012 (gmt 0)|
|Another new one was the ahrefsbot. |
I've had that one blocked for a long time due to misbehavior w/r/t robots.txt, so it was a surprise seeing the occasional "ahrefs" graphic in the top corner of wmt pages. What's it for? First showed up in November, with a peculiar habit of first getting pages and then getting robots.txt to, presumably, see if it was allowed to get the pages it had already gotten.
| 10:08 pm on May 29, 2012 (gmt 0)|
You can blame this new UA on me ;)
I suggested they change it because ScoutJet didn't point people to using Blekko and IMO they were wasting literally millions of opportunities daily to let webmasters know they exist.
Anyway, they still honor the ScoutJet UA obviously so you don't have to change anything unless you serve robots.txt dynamically like I do, as I had to change my script just a bit so I didn't keep telling Blekko it was denied LOL!
| 10:19 pm on May 29, 2012 (gmt 0)|
Unusually for a bot, there's no appended version number.
Blekkobot; ScoutJet; in
Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot) with no appended version info cf.
| 11:06 pm on May 29, 2012 (gmt 0)|
Unless you have multiple active bots that have different criteria that need to be met, I really don't see the need for the version number except as a courtesy. It's actually a PITA for people that test for exactly specific UAs that suddenly get broken when the version changes.
| 11:11 pm on May 29, 2012 (gmt 0)|
I don't test for specific UA versions, but it is useful to see when the version number changes as it gives a big clue that the bot behaviour may have changed in some way too.
[edited by: g1smd at 11:28 pm (utc) on May 29, 2012]
| 11:25 pm on May 29, 2012 (gmt 0)|
I agree and you can see it's evolving and not some dead bot just running from a server until the end of time just wasting our bandwidth. However, it also makes things break for both novices and those doing strict matching was my point.
| 9:04 pm on May 30, 2012 (gmt 0)|
Wish you'd warned us, IncrediBill! :)
I test bot UAs against declared or otherwise known IPs, sometimes shortened UAs, sometimes (as in scoutjet/blekko) the full UA. This helps in prohibiting bad UAs from good bot IPs, which I'm seeing a lot more of lately.
I usually pick up changes fairly quickly but this UA change slipped by me for at least a couple of weeks.