Soooo, Has Anyone Seen Deepbot?

Or deepbot-like behavior?

[Critter]

I've noticed freshie poking around my site, but nothing like deepbot behavior.

Was wondering if anyone's seen deepbot proper (from the old deepbot IP addresses).

Peter

[Stefan]

Not here, just freshie barely looking at the site. If we're moving from a regular deepcrawl to just freshbot changes, then I expect to never have new pages added to the index again. I get the impression that freshie likes changes, but my index and main pages are what they are... I'm not changing them every day to keep a bot happy. The new stuff, pages every day, are new, not there before, and not being found by freshbot. My frustration continues, (especially cause Fast, Ink, and the whole kit and kaboodle are finding and adding everything very quickly).

[MrSpeed]

No deepbot. Freshbot seems to be doing it's typical thing. Getting a lot of visits from scooter.

[jimbeetle]

Hmmm. No deepbot but freshbot has taken all pages plus in the last couple of weeks despite no new content, acting much more like deepbot.

Think I saw another thread on different behaviour here somewhere but can't find it now. Things do seem to be changing somewhat.

[mfishy]

I really can't tell the difference in the way freshbot is acting. Fresh has always hit all my pages.

For those who are seeing a change in the way freshbot crawls, I am interested in hearing exactly what this means.

GoogleGuy said that we should expect a traditional update (crawl/index) cycle. If they are still planning on using deepbot from 216....this is not good news as I would have hoped to have seen it by now and this would mean a long delay for the next update.

[Mozart]

Here freshbot found a few new pages from links from older pages, but it did not follow the links on those new pages to find even more...

[hotice_2002]

No deepbot here too. The freshbot visited my site a moment ago.
64.68.82.48 - - [30/May/2003:07:52:26

[wanderingmind]

Same here.

Freshie comes everyday, takes the changes in the homepage and refreshes the cached page in google.

One change though - earlier, after a few days, the homepage freshie took would revert back to the deepcrawled page. Not happening this time. The page seems to remain in cache even after the Freshbot date is removed from the SERP. Now that is more like deepbot. Hmmmm...

[cheethebee]

No deepbot, only freshbot - which has indexed my whole site again...

[wanderingmind]

Freshbot indexed all pages? Has never happened to me. freshbot has taken only a look at the homepage, and even though other updated pages are linked to from the homepage, the only thing that has been taken is the homepage. Wish I were in your shoes :)

[annej]

Always before Google updated new articles within a day or two. Now even thought freshbot seems to be dropping by almost every day the new pages I've put up in the last month don't come up in the serps. Instead the serps show the index page that tells about the new pages. I even linked one new article from my homepage in hopes that would help freshie pick it up with no luck.

[Pricey]

Freshbot has only visited a handfull of my pages over the last few months, and I have not see deepbot at all.

[trillianjedi]

Looking at my logs I'm pretty sure that freshbot is still crawling deepbot collected links.

I believe this is also why Swamper's site was bought down from the earlier SID problem - he's fixed his URL's but freshbot is using deepbot April data (the old SID URL's) and not the corrected ones from the site.

So I think we still have a while to go. It seems google is quite intent on using freshie on the April data at the moment.

The other thread I started on this with googleguy's comments also refers (lost the link).

TJ

[Stefan]

Glory be... I just looked at the logs again. Over the last few hours 64.68 has been hitting me and finding my newest pages... one just put up yesterday no less... this is the most it's done here in a while. There are still unindexed pages to be found... go freshie go.

[jomaxx]

I have seen deepbot (i.e. Googlebot from IP 216.239.*) take hundreds of pages from my site every day for at least the past week. But Freshbot has been more active.

[mfishy]

jomaxx

Wow! Haven't heard or seen that from anybody else yet. I have a PR 7 page that has not seen deep yet - just the usual fresh

[jomaxx]

It's kind of like a Freshbot pattern, though. Coming back every day and crawling 800-1200 pages.

When the true deep crawl comes, Googlebot usually hits ~15,000 pages in a couple of days. That has not happened yet in May.