Informational Thread on Freshbot

Can we keep this thread facts only about freshbot?

[Clark]

Maybe this has been done before, but I never noticed it, so...how about a thread with pure info on the Freshbot. I'm really curious as to why Freshbot chooses certain pages to be indexed over and over and not others. In my logs, there is no rhyme or reason to them. Also would like to know how to offer up an index of fresh content for the freshbot.

If anyone has studied the freshbot algo, would love to hear from you.

These are the "facts" I've gleaned from WW so far:

1. The higher PR pages will more likely get visited by the freshbot, more pages indexed more often.

2. If your content changes frequently, you have a better chance to get freshbotted.

3. News sites with well-formed indexes and Blogs do well with freshbots.

[g1smd]

I see no rhyme nor reason in the freshbot sometimes. It does what it does, when it feels like, how it feels like, and if it feels like it.

[Jesse_Smith]

4. The freshbot can get a new site listed with in a few days after being created. Though it dumps you a few days later.

5. If you have a site with a good PR and you finally make a change that allows the googlebot to crawl for the first time, the freshbot will act like the deepcrawl and deepcrawl your site over and over so it looks like it never dumped you.

6. The freshdate has nothing to do when the dance will occure. Some times it's up when the dance starts and some times it's not.

[twilight47]

Yes, but if freshbot freshes and then later dumps a site before the update, will the SERPs during the fresh period be similar to one's once the site is updated and indexed?

[suggy]

What I know of Freshbot

1) Site does not need a high PR. My PR4 is indexed almost every other day by Freshbot.

2) Pages show in the results typically within 3 days

3) Freshbot does not rely on high ranked (PR6 is often quoted) links to your site. My highest inbound link is a PR5.

4) Freshbot gets a hunger for sites that are frequently updated. You can keep her coming back for more with daily updates.

Suggy
ps: I luv Freshbot!

[Clark]

Hi Jesse_Smith,

Per your item 5, What is a good way to prepare your site for freshbot? Is it different than for the deepbot? (I would guess that freshbot likes to see links more on the front page whereas the deepbot needs a sitemap somewhere, but not necessarily the front page. But that's conjecture, nothing I specifically noticed).

[Jesse_Smith]

I got the #5 tip from my message boards. For two years it wasn't geting very many pages deepcrawled, about five files indexed. I hurd about the session ID not being liked by search engines so when it started the deepcrawl in January I got rid of the session ID and BANG! I started geting deepcrawled and shortly after that all the main board indexes were listed. The dance came and there was around 1,600 pages listed! Now there are over 7,000 files indexed, and this www-sj thing that's going on shows another 1,000 listed.

session ID = Google Death Penalty

[Clark]

I guess freshbot has found me. Over 1400 pages indexed in the last 24 hours.

BTW, am I the only one that thinks it's kind of silly that freshbot data leaves the index after a couple of days? Spidering is a page is spidering a page. Even if the page is given a priority for a day or two and then given a temporary PR1 until such time as the update happens, having extra (freshbotted) pages in the index can result in some answers on queries where there are no other pages that match.

[amazed]

fresh links from high page rank sites seem to attract freshbot

I don't see rhyme nor reason sometimes either like freshbot taking a page where I placed a new link as fresh but ignoring the actual fresh page

might have a time schedule, might go for a walk on holidays when not working overtime somewhere else

[DavidT]

Not a big thing by any means but annoying for me anyway. Comes every few days to my penalised site (I imagine because I keep piling on the links) and absolutely refuses to follow any redirects, the only robot I've seen with this problem.

[j_h_maccann]

I put up an entirely new site at a newly-registered .info URL (2000 static pages of scanned documents, interlinked with 49,000 internal links). On April 01 I opened up its robots.txt and submitted the URL to Google, but no visit.

On April 15, I got links to the new site from three popular unaffiliated sites in the same field, two of which get freshbotted regularly themselves.

Freshbot came by my new site within 24 hours to look at the top few levels. Within another 24 hours the new homepage appeared in Google at #1 for its title, and a second-level page at #2 (and the #3 and #4 sites were the two freshbotted sites linking to my new site, each with different link text). Obviously, the title of the new site is NOT targeted competitively by others.

The site has never changed position since, and remains now in the same #1 position on www, www-sj, and www-fi.

Deepbot came by within the first week and indexed all 2000 pages. These interior pages are still not visible on any of www, www-sj, or www-fi.

So freshbot is very powerful magic for a new site. By contrast, although other searchbots have stopped by for minor visits, the site is still not visible in any search engine other than Google or Google's customers, more than a month later.

[brotherhood of LAN]

Some of the fundamentals were posted here [webmasterworld.com] by Lazerzubb, feb10, 2003.

Should cover at least some of it :)

[Clark]

One thing I noticed but am unsure of, freshbot does not seem to behave like a crawler in the sense that it gets a page, finds the links, then crawls those links. Maybe on the first visit. But I'm seeing it visit pages for which there are no longer any backlinks, yet not visit a single new page that is linked from a page it did crawl. Anyone else notice similar?