DuckDuckBot

(Am I hopelessly late to this party?)

[Pfui]

Has anyone seen DuckDuckBot? If yes, does it behave itself? I'm a big fan of DuckDuckGo, but dislike its wrapping of Bing's results.

"What Is DuckDuckBot and What Does It Do?"
(includes info & 12 IPs)
[help.duckduckgo.com...]

[lucy24]

I occasionally see something calling itself DuckDuckBot, but they don't necessarily match up with their reported IP ranges, so I continue to block them. Or rather
:: quick consultation of htaccess and logged headers ::
I have not elected to poke holes for assorted header-and-IP deficits. Besides, I count a whopping total of 12 (twelve) robots.txt requests over the past 13 months--nine of them with a patently bogus referer--meaning they have received nine minimalist “everything is disallowed” and three full-length “you, DuckDuckBot, are disallowed by name”.

otoh I do regularly see, and permit, DuckDuckGo-Favicons-Bot--which requires hole-poking of its own because it requests / root before favicon, sends an auto-referer with all requests, and doesn't request favicon if / is blocked. (There is no earthly reason for it to keep requesting the root, but the icon accompanying search results does make your page more attractive.) I don't see a lot of people from DDG search, but the ones that do show up tend to be “quality” visitors--the kind who stick around and go on to additional pages.

[dstiles]

I allow specific IPs for ddg according to their web site [help.duckduckgo.com...]

I don't see a lot of hits for the bot and don't normally check referers for ddg

[nickZ]

Their (DDG) SERPS are kinda off anyway,
When looking for a term which about letting reconvert websites, the Sitebuilder sites, where the sites getting reconverted from, are prominently displayed.

[tangor]

DDG is general is an okay to very okay search for me (mostly bing, of course), but they are my fall back when bing gets silly and starts taking on all kinds of krap to the search result that it won't connect to the site. Thus ... DDG works.

As for DDG as a bot, they honor my robots.txt and their favicon requests are generally ignored as they are only once a month or so.

[SumGuy]

Got a couple hits today from 40.88.21.235 which seems to be a DDG bot IP address, the user-agent being:

Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)

It just requested my landing page and favicon.ico a couple times. Didn't seem to request robots.txt.

Now the header field right after the UA, I'm not sure - is that the cookie field? I normally don't see anything there. But for these hits I see this:

TS(8-character-hex)=(a_big_hex_string); CFTOKEN=(much_smaller_hex_string); CFID=(9-digit-number)

I don't send cookies, so I'm not sure if this was trying to set one or send one.

I also don't see any organic site access within the same time frame as these hits to indicate that they were part of a DDG search.

It's kinda odd that the IP doesn't resolve to a host name. You would think that DDG would have that under control.

I guess it's a "favicons-bot" - what ever that is. I scanned for past hits from 40.88.0.0/16 - I see about a dozen past hits from this exact same IP, mostly in the second half of 2020, a few in 2021 and 1 in 2022. None of those had the cookie thing going on. Actually no other http/https activity from that entire /16 except for 40.88.21.235.

[lucy24]

The faviconbot is exactly what it says: it scoops up a current favicon so it can display it alongside the SERP. Try any old random DDG search and you can see what it looks like. A handful of other search engines--Yandex, I think?--do the same thing. I think it does make the result look more appealing--or less appealing, if everyone but you has an icon. But of course that just means people see it; there's no guarantee that they will click on it among the however-many hits there are on the page.

On the other hand, I don't think anyone has ever figured out why it has to pick up the root Every. Darn. Time. Nor yet why it insists on sending auto-referers, just as if it were a spambot.

[dstiles]

> pick up the root

It has to pick up some page and the root is the easiest.Without a page header, which CAN specify its location, it might miss where the favicon lives, which is not always in the root folder.

> sending auto-referers

You could ask them? May be something the designer thought was neccesary. :)

[blend27]

@SumGuy

CFTOKEN=(much_smaller_hex_string); CFID=(9-digit-number)

These are ColdFusion Server-side cookie contents sent with request back to requester.

Example: CFTOKEN=(5af323b545bb531d-B5AA756A-DB7C-0882-08914B004C4C3E6D ); CFID=(123456789)

As to why you would have these in Request Headers -- beats me. Unless DDG is running ColdFusion Instance to scrape you site... which is...naaaahhh

[Peter_S]

Hi,

It's been months since the last DDG bot visit on my site. I see requests identifying themselves as :

DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)

However, the page http://duckduckgo.com/duckduckbot.html does NOT exist. The real page is https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/

And, none of the requests are matching the IP listed on this page.

So what's going on at DDG?

[edited by: not2easy at 11:37 am (utc) on Jul 23, 2023]
[edit reason] unlinked for readability [/edit]

[brotherhood of LAN]

>months since the last DDG bot visit

Bing's API price increases for all the metas that used it seems to have shook up who is doing what. Not that DDG ever seemed to have any kind of independent index beyond Bing (and pre Ukraine invasion, Yandex).

Does anyone here actually know what their bot does, beyond favicon collection.

[Peter_S]

Hi,

Indeed , I only see the fav icon bot. other DDG bot hits are coming from IP ranges, that they are NOT listing on their official page. However all these other hits are coming from a Microsoft IP range. Legit? No way to tell.

[nickZ]

DDG now only runs over Bing, even their adverts goes over Bing