Fresh Bot How to

Does anyone.....

[jjdesigns4u]

Can someone give me an understanding of freshbot.

I know this spider is used to collect fresh information but I do not have a clear function of how it collects and then places the info.

And what can be done to get more attention

and anything else you want to share

[takagi]

Hi jjdesigns4u,

A visit by freshbot is like a user visiting your site with a browser. The information of the visited pages are saved and checked for links to unknown pages and compared with the previous data (if available). Freshbot could decide to follow the links to new pages. The data is then distributed to the 9 data centers. After some delay (2 days or so), the new information can be found in the SERPs, sometimes with the date the pages was spidered. This date will disappear after a few days. And sometimes the complete page disappears from the SERPs until the next update. Pages with updated data could cause a higher ranking in the SERPs. The PageRank is only recalculated once a month, so new pages will start without PR. You can find more information in the Freshbot [webmasterworld.com] page of the WebmasterWorld Glossary [webmasterworld.com].

> And what can be done to get more attention
Those pages on your site that change more often, are more likely to be visited. These changes should be more than just adding one word or so. Maybe freshbot is also more interested in pages with internal links (site map) or external links (links page).

> and anything else you want to share
The difference between freshbot and deepbot seems to get smaller (see Is Freshbot now Deepbot? [webmasterworld.com] thread).

[jjdesigns4u]

takagi

That was exactly what I was looking for thank you very much

[jeyval]

Can you tell me how do you know that your have a received a visit from freshbot?

Not that I have checked very often on the past but frankly I can only remember having read "googlebot". Does it reads differently in the logs when such visit is from fresbot?

Thanks

[Stefan]

Freshbot and deepbot have traditionally been differentiated by their IP#'s; freshie is 64.68.x.x, deepbot is 216.239.x.x.

Hope that helps

[g1smd]

Just like that:

2003-05-30 08:26:04 64.68.82.57 - W3SVC9999 255.255.255.255 80 GET /robots.txt - 200 2 4184 181 31 HTTP/1.0 Googlebot/2.1+(+http://www.googlebot.com/bot.html) - -

Comes back every day.

[jeyval]

Thanks both of you

Thats quite odd because I never seem to get an ip address when talking about googlebot but instead my logs display this way

crawler11.googlebot.com - - [30/May/2003:07:41:52 -0400] "GET / HTTP/1.0" 200 48431 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Any answers having this into consideration?

[g1smd]

My logs are from an IIS server, maybe yours is running Apache or someting else?

[kpaul]

crawler11.googlebot.com - - [30/May/2003:07:41:52 -0400] "GET / HTTP/1.0" 200 48431 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Any answers having this into consideration?

crawlerX. == Freshie
crawlX. == DeepBot

Although there are changes underway and who knows what will happen in the next few weeks ;)

[jeyval]

Thanks very much

In that case, would you consider "strange" that fresbot is visiting my site despite its been banned (or at least gone from the index) for almost a week now?

Thanks again