homepage Welcome to WebmasterWorld Guest from 54.167.182.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 40 message thread spans 2 pages: 40 ( [1] 2 > >     
Most aggressive crawl I have ever seen.
Gbot pulled 4700 pagesso far today; Request 3 pages a second
vabtz



 
Msg#: 28237 posted 9:48 pm on Feb 22, 2005 (gmt 0)

I recently changed my site to reduce the page size drastically and clarified the links structure a bit.

Today gbot has pulled 4700 pages (so far) requesting 3 pages a second.

I verified its a real bot. Is this normal? My server can easily handle the load but just seems a little frightening and exciting at the same time.

 

sandor

10+ Year Member



 
Msg#: 28237 posted 2:58 am on Feb 23, 2005 (gmt 0)

yeah, i've noticed it crawling quite a lot lately. wonder what's up!

itloc

10+ Year Member



 
Msg#: 28237 posted 6:27 am on Feb 23, 2005 (gmt 0)

Same here ...

Never saw a crawl like that...

itloc

walkman



 
Msg#: 28237 posted 7:10 am on Feb 23, 2005 (gmt 0)

huge crawl on 3 sites. Barely any visits on the one that matters most though :). Of well.

estaquieto

10+ Year Member



 
Msg#: 28237 posted 7:33 am on Feb 23, 2005 (gmt 0)

Me six. About 1,500 pages crawled on mine so far. Unprecedented.

victor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28237 posted 7:57 am on Feb 23, 2005 (gmt 0)

Googlebot started rampaging across one of my sites at that sort of rate a few months ago.

I emailed Google and told them to behave or be banned.

They emailed back an apology, and all has been well since.

Their bots can be as buggy as their SERPs. If you don't tell them the problem, they may not notice it themslves.

midlifecrisis

5+ Year Member



 
Msg#: 28237 posted 8:05 am on Feb 23, 2005 (gmt 0)

On a brand new site (setup only last Friday) the bot pulled almost 3000 pages on Sunday. All of them already show up in the index.

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28237 posted 8:15 am on Feb 23, 2005 (gmt 0)

I also got the full treatment. Anyone notice any 304's around the same time? I posted here about it, it's got me wondering

[webmasterworld.com...]

eyezshine

10+ Year Member



 
Msg#: 28237 posted 8:58 am on Feb 23, 2005 (gmt 0)

I seen googlebot spidering my cgi pages deep today. Normaly they only spider one level deep into my cgi generated pages but today I seen it spidering up to 10 levels deep just like they were standard html pages?

Maybe they are trying to build an even bigger database? I never thought I would see them digging that deep on cgi generated pages?

mcavill

10+ Year Member



 
Msg#: 28237 posted 9:48 am on Feb 23, 2005 (gmt 0)

wonder if that's to correct the url only listings I'm seeing around the place.

GoldFish

5+ Year Member



 
Msg#: 28237 posted 10:23 am on Feb 23, 2005 (gmt 0)

I have the exact same thing. In this month so far, Googlebot has crawled about 20 times (!) the number of files than it did in january. IMHO, this probably has everything to do with their new algorythm, i.e. the Allegra update.

Liane

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28237 posted 10:33 am on Feb 23, 2005 (gmt 0)

Its seems we are all experiencing the same thing. Its "that time of the month" for Googlebot to do its thing. Maybe we'll have another update around the end of the month.

It'll be really interesting to see if the 302 redirect problem has been solved. I have a gut feeling it just may be! :)

GranPops

10+ Year Member



 
Msg#: 28237 posted 10:43 am on Feb 23, 2005 (gmt 0)

Massive crawl everywhere, every last nook and cranny, looking good for another shuffle

taps

10+ Year Member



 
Msg#: 28237 posted 11:48 am on Feb 23, 2005 (gmt 0)

Seeing a very deep crawl too.

Lost a lot of positions after renaming all URLs and Allegra. My 301-redirects seem to work properly. Hope to be back soon. :-)

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28237 posted 12:07 pm on Feb 23, 2005 (gmt 0)

The pattern for my website has always been a deep crawl a couple of days after an update and a couple of days before an update. My guess is that we will see more reshuffling this weekend.

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28237 posted 12:07 pm on Feb 23, 2005 (gmt 0)

>>wonder if that's to correct the url only listings I'm seeing around the place

I've got a couple of pages that went URL only on a site. One was definitely something wierd on their end - it was a duplicate listing of the same URL/page on the SERP - repeated twice, so one was normal and the other was URL_only. I suspect that's a side effect of something they're doing "differently" in some way.

The other page is my fault for sure - accidentally put the same text on two pages, widgets-2.html and widgets-3.html instead of the final copy of the pages. All they did was have the widgets-2.html page go URL_only and the other is fine, as is the rest of the site. Rather, will be fine if it ever gets out of the "sandbox."

My gut feeling, which I can't shake and can't figure out, is that they're doing something different in the way they're crawling sites.

xrtza

5+ Year Member



 
Msg#: 28237 posted 1:38 pm on Feb 23, 2005 (gmt 0)

Massive crawl here also. For about the past week 2-3 pages per second from 10 or more ip addresses. I want to believe all the attention is a good thing, but of the over 16,000 pages indexed by google none are rankng for anything including distinct company name. Very little traffic from G and intestingly more traffic from google.br than google.com (seems odd to be since it is .com and located in US). The not ranking may be in part due to a major rewrite of all URLs. Only a handful of pages have PR at this point. Hopefully a bigger better update is coming. As much as I have read about Allegra I don't quite understand exactly what it did other than make some happy and some sad. I can't help but to believe if Google feels my site is important enough to crawl and index that many pages it must be important enough to rank somewhere. Does anyone have any relative data to manage ranking expectations. What is reasonable? If on page one MSN page three Yahoo where would be realistic expectations for Google? page 10? 20? 1000? I would be curious to know if anyone has any data on this.

Just Guessing

10+ Year Member



 
Msg#: 28237 posted 2:10 pm on Feb 23, 2005 (gmt 0)

Yes I saw huge crawls from the Mozilla agent before and after Allegra, and again now over the last couple of days.

Could also be a panic crawl to sort out a problem.

I have a site where a site:www.example.com search seems completely broken - loads of URL only listings, only returns 600 results even with &filter=0 and doesn't return the home page or any top level pages even though they are indexed and ranking well in other Google searches.

midlifecrisis

5+ Year Member



 
Msg#: 28237 posted 2:11 pm on Feb 23, 2005 (gmt 0)

"If on page one MSN page three Yahoo where would be realistic expectations for Google? page 10? 20? 1000?"

I have one site that is #1 at Y and #2 at M for the same kw1 kw2 combo (500K SERPS). Also first page positions at Jeeves, AV etc. It's at a solid #75 at G. Sigh.

trillianjedi

WebmasterWorld Senior Member trillianjedi us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28237 posted 2:15 pm on Feb 23, 2005 (gmt 0)

they're doing something different in the way they're crawling sites.

I know exactly what you mean Marcia - I felt the same way about it. Couldn't spot any specific pattern though.

Actually, it looks like old-skool deepbot of about 2 years ago, but with some tweaks. I wonder if they dug some old code out?

Back to a monthly re-index maybe?

TJ

midlifecrisis

5+ Year Member



 
Msg#: 28237 posted 9:02 am on Feb 24, 2005 (gmt 0)

Another massive crawl yesterday: Googlebot more than 7000 pages, Teoma 10000 pages. Luckily the server didn't explode.

taps

10+ Year Member



 
Msg#: 28237 posted 10:24 am on Feb 24, 2005 (gmt 0)

Almost 40 percent of all my traffic this month results from bots. Lotta activity.

Jakpot

10+ Year Member



 
Msg#: 28237 posted 12:03 pm on Feb 24, 2005 (gmt 0)

Now what are they going to do with this crawl results?
More surprises?

midlifecrisis

5+ Year Member



 
Msg#: 28237 posted 12:19 pm on Feb 24, 2005 (gmt 0)

Given the fact that SE spamming, scraper sites etc. are such a widespread problem, could this be a sign of a concerted action?

Perhaps the beginning of something like the SEAA, Search Engine Association of America ;-)

xrtza

5+ Year Member



 
Msg#: 28237 posted 12:56 pm on Feb 24, 2005 (gmt 0)

I just wish I could feel certain they will use all the pages the have crawled. Googlebot has settled down to a more "normal" rate for our main site now. Cached daily and deep crawled more than once this month. Last week was the big burst that the title of this thread describes. Still waiting for the update of all updates.

MrSpeed

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28237 posted 1:30 pm on Feb 24, 2005 (gmt 0)

Actually, it looks like old-skool deepbot of about 2 years ago, but with some tweaks. I wonder if they dug some old code out?

Back to a monthly re-index maybe?


I can't help but wonder if it's in response to Yahoo's crawling activity. It seems as though Yahoo is much more eager to crawl a new site or pages. Perhaps Yahoo saw an opportunity for sites that Google sandboxes?

Seo1

10+ Year Member



 
Msg#: 28237 posted 1:37 pm on Feb 24, 2005 (gmt 0)

My two thoughts

With the index doubling in size recently, and as Mr Speed said with Yahoos crawls, and now from what I see MSN and AskJeeve page counts in the server stats Google needs to do this.

On subdomains the crawl is even crazier multiple visits per day

On a client site Googlebot ate up 13k pages.

Sweeeeeeeeeeettttt

grail

5+ Year Member



 
Msg#: 28237 posted 1:45 pm on Feb 24, 2005 (gmt 0)

Yeah "me too". Googlebot had a real spurt, I like to think it was in direct response to my post on here just before it. 'Spider penalty' 'Googlebot on go slow'. :-)

Still has someway to beat Yahoo which is all day everyday. Infact I just started to put up some new stuff and yahoo was taking it before I had even finished tweaking the pages, i'm frankly amazed it found it it's almost like it spiders real time as soon as it finds a link off it goes. Rather than collecting a list of links which it intends to crawl later.

trillianjedi

WebmasterWorld Senior Member trillianjedi us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28237 posted 1:50 pm on Feb 24, 2005 (gmt 0)

Consider Block Rank:-

[webmasterworld.com...]

.... alongside furious deep-crawl behaviour.

TJ

Dayo_UK

10+ Year Member



 
Msg#: 28237 posted 1:51 pm on Feb 24, 2005 (gmt 0)

Lets hope Google can start ranking pages like Yahoo - based on content again.

Googlebot is crawling like it did in November - just before the index size increased to 8 Billion.

This 40 message thread spans 2 pages: 40 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved