homepage Welcome to WebmasterWorld Guest from 54.161.175.231
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 176 message thread spans 6 pages: < < 176 ( 1 2 3 4 5 [6]     
Gbot running hard
ncw164x




msg:167902
 9:04 am on Sep 23, 2004 (gmt 0)

googlebot requesting between 2 - 5 pages a second, not seen this type of spidering for a long time

 

cdog863




msg:168052
 9:14 pm on Oct 1, 2004 (gmt 0)

I have been getting hit hard too by the googlebot..

Recently (it's a new site) they have sucked up over 250 pages. (even my forum)

The only other thing I'm wondering now is when am I gonna get a PR... jeesh

defone




msg:168053
 10:06 pm on Oct 1, 2004 (gmt 0)

It seems that gbot is little bit settled down, spidering seems normal again. What do you think?

jnmconsulting




msg:168054
 10:20 pm on Oct 1, 2004 (gmt 0)

One of my sites have been hit everyday for the last 5 days by multiple G bots, I see 12 different IP addys.

walkman




msg:168055
 1:29 am on Oct 2, 2004 (gmt 0)

Gbot i staking a break...still spidering but not as much. I get about 200 hits on a 3-4000 page site.

BillyS




msg:168056
 2:14 am on Oct 2, 2004 (gmt 0)

Now I'm getting worried. The last time I saw a fresh tag for one of my websites was September 29th. I actually saw this tag appear on the 29th.

Now fresh tags are missing for me - two days running. I have more pages showing up in google than ever before for this website when I query "site:www.widgets.com widgets" But I just checked my logs, no googlebot for 2 days straight - ARGHHHHHHHHHHHHHHHHHHHH!

Thoughts? (no black hat, in case you're wondering)

smoke signals




msg:168057
 6:36 am on Oct 2, 2004 (gmt 0)

Hi there all,
First Iíd like to take a moment to introduce myself, as this is my very first post to your website. Iím Casey and live down Scottsdale, Arizona way.
I'm relatively new to web design, (but not just dropped off the turnip truck new) and will be putting my efforts toward a Native American Art website, so Iíll be turning to this section here a lot in my efforts to learn more about Google and how it operates, along with all the other valuable information on SEO to be found too. However, as for right now I feel rather inadequate with this vast amount of knowledge reeling by me at the turn of every page. I will try to find answers to my questions first by going to your Search Area, but hope you will bear with me if I ask some elementary questions now and again.
Sincerely Southwest,
Casey

jeremymgp




msg:168058
 10:41 am on Oct 2, 2004 (gmt 0)

Welcome, Casey. :)

BillyS




msg:168059
 1:03 pm on Oct 2, 2004 (gmt 0)

I just posted this over in the robots forum too (awaiting review). More stuff from Google (I think)

66.92.186.101 - - [02/Oct/2004:08:31:18 -0400] "GET /robots.txt HTTP/1.0" 200 484 "-" "stat (statcrawler@gmail.com)"
66.92.186.101 - - [02/Oct/2004:08:31:19 -0400] "GET / HTTP/1.0" 200 19826 "-" "stat statcrawler@gmail.com"

Anyone else see this one yet? I could not find anything on the web. Looks like it resolves to Speakeasy?

[Server: whois.arin.net]

CustName: SFO BRIDGED CIRCUITS
Address: 440 Mission Court
City: Fremont
StateProv: CA
PostalCode: 94539
Country: US
RegDate: 2001-11-09
Updated: 2001-11-09

NetRange: 66.92.186.1 - 66.92.186.255
CIDR: 66.92.186.1/32, 66.92.186.2/31, 66.92.186.4/30, 66.92.186.8/29, 66.92.186.16/28, 66.92.186.32/27, 66.92.186.64/26, 66.92.186.128/25
NetName: SPEK-SFO-BR-44
NetHandle: NET-66-92-186-1-1
Parent: NET-66-92-0-0-1
NetType: Reassigned
Comment:
RegDate: 2001-11-09
Updated: 2001-11-09

TechHandle: AS3414-ARIN
TechName: Stollar, Andreas
TechPhone: +1-206-728-9770
TechEmail: abuse@speakeasy.net

OrgTechHandle: AS3414-ARIN
OrgTechName: Stollar, Andreas
OrgTechPhone: +1-206-728-9770
OrgTechEmail: abuse@speakeasy.net

# ARIN WHOIS database, last updated 2004-10-01 19:10
# Enter? for additional hints on searching ARIN's WHOIS database

Critter




msg:168060
 1:12 pm on Oct 2, 2004 (gmt 0)

It's a non-Google bot that's owned by someone that happens to have a gmail account.

ownerrim




msg:168061
 5:18 pm on Oct 2, 2004 (gmt 0)

Weird, I typically get what I'd guess you'd fresh tags every 2-3 days, but I just checked and the cache for my home page has reverted to what was retrieved on 9/12/04

g1smd




msg:168062
 1:27 pm on Oct 4, 2004 (gmt 0)

I watch a SERP with about 30 results in it, several times per week (it is a result for some incorrect information that is printed on other sites that they have been asked to remove).

The SERP has been stable for months, except for the reduction from about 50 results where sites gradually comply with the removal request.

The SERP was rearranged a bit a few days ago, with several results dropping out even though they haven't yet made the requesed change; but today there is a major change. The results have been almost turned upside down. Looks like Google has built and published a new index based on the massive spidering that they did a week or so ago.

sri_gan




msg:168063
 2:46 pm on Oct 4, 2004 (gmt 0)

I probably think, Google and MSN are testing their new Crawlers.

Possibly Google wanted to test and identify cloaking pages through the Mozilla User Agent which it could track maximum redirections as well crawl other application page (ex. Flash, MS word docs etc...)

MSN is about to demonstrate its search capabilities to a panel of people so possibly they increased the spiders to apply their alogirthm on maximum pages.

The other ip I saw in the previous page here is not from MSN or Google.

Pass the Dutchie




msg:168064
 2:55 pm on Oct 4, 2004 (gmt 0)

Possibly Google wanted to test and identify cloaking pages

Yea you could be right if these are all new IP's Gbot are using.

webdude




msg:168065
 2:56 pm on Oct 4, 2004 (gmt 0)

Looks like Google has built and published a new index based on the massive spidering that they did a week or so ago.

This is what I am seeing too. In my areas, it looks like the index has been rebuilt from the ground up. I still think this has to do with a cloaking and metarefresh/302 problem the G was having.

New IPs. New datcenters? New Index? Don't know yet for sure yet.

newwebster




msg:168066
 3:27 pm on Oct 4, 2004 (gmt 0)

Looks like a bunch of junk to me. Google just went out and try to find some new fresh pages to add to their index. It does not look like any ranking factor has been introduced as of yet. These results will change over the next few weeks as the weighting factors are applied.

RichD




msg:168067
 7:30 pm on Oct 5, 2004 (gmt 0)

Just to help clear up an earlier discussion in this thread on the number of servers at Google between Hanu and Lord Majestic, I just came across this pdf: - [research.ibm.com...]

Page 39 shows the number of servers and queries at various points between Nov-98 and Aug-02. Charting these figures in excel, and using the 200M queries/day stat from page 35, makes it look like there would be about 12,000 servers for search now + maybe a few for adwords/adsense/gmail/etc.

The last stat they give for number of servers in Aug-02 (handling 150M queries) is shown as >10,000 but looking at the data in excel sugests it shouldn't be much over that unless they were hitting some scaling issues.

ncw164x




msg:168068
 7:53 pm on Oct 5, 2004 (gmt 0)

At Pubcon 6 there was a well know google employee who said the official figure that they admit to was 14,200 or 14,500 I cant remember now, but then gave his normal big beaming smile ;)

Now us lot being on the outside of googleplex...well...the figures i have heard of now exceeds 100,000...who knows?

dataguy




msg:168069
 2:14 pm on Oct 6, 2004 (gmt 0)

I run a very small, very niche search engine (for kids) which has under 500 listings. Gbot has performed 17,000 queries of my search results pages in the past week. You think they got the information they are after?

george123




msg:168070
 2:34 pm on Oct 6, 2004 (gmt 0)

FRESH TAGS... i monitor 5 times a day the SERPS in my industry let me tell you the TAGS are all fresh in the top 10 5/10/04.

Nuttzy99




msg:168071
 2:34 pm on Oct 6, 2004 (gmt 0)

Gbot has performed 17,000 queries of my search results pages in the past week.
Are you running a forum or portal that assigns session ids? That can cause bots to freak out, b/c they think each new sid is a new page.

-Nuttzy

dataguy




msg:168072
 2:54 pm on Oct 6, 2004 (gmt 0)

>Are you running a forum or portal that assigns session ids? <

Nope, just a querystring with the search term and page number...

kosar




msg:168073
 3:31 pm on Oct 12, 2004 (gmt 0)

i have gotten 18,000 hits over the last 4 days for an 800 page site.

darqSHADOW




msg:168074
 5:10 pm on Oct 12, 2004 (gmt 0)

I'm up to 34k hits this month alone from GoogleBot, and almost a gig of bandwidth used. Google used to track 35 pages for me (as I said before), it then jumped to about 600, last nite it was 900, today its 750, so Google seem to be all over the board on their results lately. Its almost like things change on a minute by minute, search by search basis for me. (The GoogleDance Tool shows the same results for all datacenters, yet all datacenters change throughout the day.)

DS

dataguy




msg:168075
 5:41 pm on Oct 12, 2004 (gmt 0)

You know, traffic is picking up steadily, I think I will retract anything that sounded like a complaint and ask the Gbot to come back for more....

kosar




msg:168076
 5:46 pm on Oct 12, 2004 (gmt 0)

ds, what is the google dancetool you are reffering too?

CarlC




msg:168077
 6:31 pm on Oct 12, 2004 (gmt 0)

ds, what is the google dancetool you are reffering too?

He is referring to a tool available on a popular SEO website that allows you to see SERPS from different Google datacenters side by side for comparison.

Go to google and type "google dance tool". The number one result should take you there. Hopefully it hasn't changed since this post time. ;)

This 176 message thread spans 6 pages: < < 176 ( 1 2 3 4 5 [6]
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved