What is the freshbot doing?

Getting very old pages and looks like normal crawler activity

[TheDave]

I was having a closer look through my logs and I noticed Freshbot getting pages which are very old. I'm talking about the oldest, least modified page on my web, a page which I dropped all links to a couple of months ago. So why would freshbot being doing this? Another thing I have noticed is freshbot got a page which actually was fresh about a month ago, but then came back for it just recently, even though it is unmodified. Thirdly, I noticed freshbot relentlessly wasting a lot of time going through my showproduct pages (200 in about 2 days), which are in no way fresh (I add new products, but it was the oldest products being crawled), and the only thing that stopped this activity was me excluding those pages with robots.txt (which I was going to do anyway).

So what I am wondering is does the IP 64.68. always mean the freshbot?

[vitaplease]

Freshbot is still learning.

Do not forget, to find a real new, fresh page, freshbot has to know where it is located. Hence it looking at the most established (often most linked to pages in your site) to find possible new links towards real fresh pages.

[DavidT]

Same here, it goes for the most minor pages with 3 words of text on them and ignores new pages with hundreds of words. Also requesting pages I deleted months ago.

[HitProf]

>So what I am wondering is does the IP 64.68. always mean the freshbot?

No I don't think so. 64.68.84.xx is reported to be a deepcrawl bot.

There is a list somewhere in this forum.