Welcome to WebmasterWorld Guest from 34.204.194.190

Forum Moderators: open

Message Too Old, No Replies

Is Freshbot now Deepbot?

The line is getting drawn ever thinner

     
4:18 pm on May 22, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7256
votes: 3


I've seen several postings about this now in the last few days, although this is my first actual experience of it.

I'm being hit very hard by google's freshbot at the moment, and going deep too. At first glance at what is currently going on with the little guys, I had to check and double check that the IP's were 64.... (they are).

It's behaviour, in terms of hard hitting and depth of crawl (it's going through the entire site) is more like the character of the old deepbot.

In fact, it's identical behaviour to deepbot the last time it crawled this site back in April.

I'm interested in hearing from others who are seeing the same.

TJ

11:31 am on June 2, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 19, 2002
posts:2139
votes: 0


Picking up new pages? So what? Freshbot has always picked up any new page I have put up.
12:38 pm on June 2, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 3, 2003
posts:58
votes: 0


Just for the record, last month I raised the idea here that Deep Crawl might now be taking place with IPs that were previously thought to be Fresh Bots, but no one was willing to contemplate this scenario (see [webmasterworld.com ]). However, after reading this discussion, it sounds like people may be warming up to the idea. That said, my site has been pounded by 64.* for the past ten days. I've not added up the daily totals, but suffice it to say that it's approaching 1 million pages and still shows no sign of slowing down. In parallel with this, normal Fresh Bot behavior has been taking place, with Freshie eating up around 50k pages during the same period.
12:43 pm on June 2, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 19, 2002
posts:2139
votes: 0


uber_boy,

You may be right and that is good info. My point is that it is not useful to say that the deepcrawl is happening because new pages are being picked up. It certainly is a strong possibility that the 64 range is now deepbot.

I can't think of any reason why GoogleGuy can't confirm or deny this.

12:58 pm on June 2, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7256
votes: 3


It is possible that googlebot is crawling from freshbots old IP range - but I have not yet seen any reasonable explanation as to why they would do that.

What is a fact, and has been shown by several webmasters logs (including matching of SID's where applicable) is that freshbot has crawled the original deepcrawl index from April.

It may be that has been the case for you also. In doing so, inevitably, the behaviour looks like deepbot because it is, in effect, following deepbots footsteps.

TJ

1:22 pm on June 2, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 2, 2003
posts:398
votes: 0


I think Google's bots must be on a diet or something. They just nibble a bit here and there and come back a few days later... Before, they would eat the whole plate and you didn't see it come back until a few weeks later...

Must be that new algo again

1:26 pm on June 2, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 17, 2003
posts:384
votes: 0


Looks like it. The bot is tending to eat the robots.txt and then come back for half the site after about 30 mins then the rest 30 mins after that and then anything it missus about an hour later.

Kinda...!

3:07 pm on June 2, 2003 (gmt 0)

New User

10+ Year Member

joined:June 2, 2003
posts:39
votes: 0


do general public ever use 64. etc and 216. etc as I get google proxy wap visits everyday? and get 64. etc most days from various places yet i have no listing on the web of my site etc as it is new?

Dayo_UK

6:37 pm on June 2, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


Domin2

Normal surfers can come in on 64 or 216 etc and often do as you have stated.

Dididudu

Following Fridays crawl my pages were live Saturday/Sunday.

Following todays I would expect them to be live tomorrow/wendesday..

Looks like normal freshbot activity to me :)

12:40 pm on June 10, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 19, 2002
posts:2139
votes: 0


It would help G to do an accurate/deep crawl, if for no other reason, just to get rid of the old pages that don't work anymore and have no links pointing to them for months.

While freshbot adds pages, it does not take them away and this badly affects searches.

9:51 am on June 3, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Feb 14, 2003
posts:236
votes: 0


freshbot was very active the entire last week on my sites, but i did not see any new additions and a lot that was added 10 days ago dropped out. i did not see any freshtags throughout all serps that i have checked. hm..
12:11 pm on June 3, 2003 (gmt 0)

Senior Member from ES 

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1381
votes: 0


Funny thing I've noticed,

I've been hit by freshy, and if I view my sites listing the fresh stuff is there, but on the serps our fresh stuff is awol, but our competitors isn't. We just lost another spot ;/

3:34 pm on June 3, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Sept 7, 2001
posts:212
votes: 0


Same thing here - a new site and new pages on existing sites were added by freshbot last week, then dropped a few days ago. Dónde está el UPDATE?

Dayo_UK

3:43 pm on June 3, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


Rocco

Re: Freshbot was very active last week but I did not see any new additions.

Fresh data was certianly added however fresh tag have not always been shown :)

3:55 pm on June 3, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Feb 14, 2003
posts:236
votes: 0


Dayo_UK - i am not sure about the addition of fresh stuff, my site is pr8 and all the fresh pages (which are crawled daily) are not getting listed. the fresh pages that were listed are out now.

i mean it is normal behaviour for fresh pages to be listed and dropped again. but while lately almost everyday new pages were added (with fresh tag) for the last 4-5 nothing was added.

i am probably just spoilt expecting google adding everyday new pages of mine ;)

4:01 pm on June 3, 2003 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 14, 2003
posts:4319
votes: 42


I had new pages show up today in Google. (site:domain.com -asd). These pages have been up for 4 months but have had minor changes. My URL's were not being visited by google at the april deepcrawl time because they were too long. This was fixed about a month ago. It's kind of random which pages were added. Of course every time I do a site:domain.com -asdf today I get a different number of links. They are doing something.
This 211 message thread spans 15 pages: 211
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members