homepage Welcome to WebmasterWorld Guest from 54.237.78.165
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 41 message thread spans 2 pages: 41 ( [1] 2 > >     
Is Google spidering tomorrow?
Google spidering
cazgh

10+ Year Member



 
Msg#: 15473 posted 8:42 am on Jul 23, 2003 (gmt 0)

According to my logs, the Google Bot comes every 10 days and is due tomorrow.

Anyone agree?

 

mipapage

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 15473 posted 1:00 pm on Jul 23, 2003 (gmt 0)

cazgh,

For some perspective, the bot comes to my site pretty much every day, though I wouldn't call her regular.

ciml

WebmasterWorld Senior Member ciml us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 15473 posted 1:32 pm on Jul 23, 2003 (gmt 0)

If you have access to a server with many sites on, then you get to see how Googlebot' patterns are very different from site to site; page to page.

Sometimes daily, sometimes more like monthly.

ILLstyle

10+ Year Member



 
Msg#: 15473 posted 2:48 pm on Jul 23, 2003 (gmt 0)

I would agree I monitor a few sites. Most get hit every day but a few don't.
It seems the sites that get hit every day do better in serps have better PR and so on.
I think sites that have a higher PR get more attention form the bots.

rfgdxm1

WebmasterWorld Senior Member rfgdxm1 us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 15473 posted 3:16 pm on Jul 23, 2003 (gmt 0)

My sites typically get hit daily. I'd be alarmed if Googlebot was absent for more than a few days.

Hobbyist

10+ Year Member



 
Msg#: 15473 posted 3:27 pm on Jul 23, 2003 (gmt 0)

Mine too! Am i one of the big boys now? :P

GranPops

10+ Year Member



 
Msg#: 15473 posted 3:39 pm on Jul 23, 2003 (gmt 0)

Getting hit virtually every day, and updated. Now 127 No. 1's on G, wish I was young like you lot and had the energy to do this for a living.

Granpops

trillianjedi

WebmasterWorld Senior Member trillianjedi us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 15473 posted 3:48 pm on Jul 23, 2003 (gmt 0)

I think sites that have a higher PR get more attention form the bots.

Correct.

Also sites that have regular fresh content will get hit more regularly.

TJ

Imaster

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 15473 posted 3:50 pm on Jul 23, 2003 (gmt 0)

My sites get hits from Google almost everyday, but since the last 3 days I am noticing a huge number of hits from Googlebot, so that calls for a new update soon :)

Stujoe

10+ Year Member



 
Msg#: 15473 posted 4:16 pm on Jul 23, 2003 (gmt 0)

Googlebot has been gulping up pages from my site for about the last week. I did have a couple of important pages show up a few days ago for the first time but they disappeared yesterday. Second time that has happened in a week.

bobosse

10+ Year Member



 
Msg#: 15473 posted 4:32 pm on Jul 23, 2003 (gmt 0)

Could you guys told me what you think about this?

I rebuilt a site a month and a half ago (completly rebuilt) and it got crawled by Googlebot at the beginning of the month during 6 days in a row. Since then, nothing, no Googlebot.

Well, in the SERPs, I could find the old pages, but since the past 10 days, it came down from 20 pages to 2 pages only available today.

Would that mean that I will get all my new pages in the SERPs soon?

Thanks for the feedback.

Bobosse

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 15473 posted 8:37 pm on Jul 23, 2003 (gmt 0)


I have a site (online at the end of March) that hasn't been freshed for many many weeks. It is at #1. However, at #2 is a page that points to that site, and that page has been fresh many times, but not recently.

Another site, first online at the beginning of May, and indexed on 2003-06-16 was freshed on every page, every single day from 2003-07-07 to 2003-07-19, but hasn't been freshed since then. [Exception: a newer page was freshed once on 2003-07-21 and just that page].

xlcus

10+ Year Member



 
Msg#: 15473 posted 9:07 pm on Jul 23, 2003 (gmt 0)

I would agree I monitor a few sites. Most get hit every day but a few don't.

Yeah, I run quite a lot of sites and I see quite a variation in the frequency of Googlebot accesses...

I run a script to check the last bot access for each site and order the list by date to see which sites were most recently indexed.

Here are the current results...

Jul 23 21:56 www.site-g.com/robot_log
Jul 23 21:48 www.site-j.com/robot_log
Jul 23 19:22 www.site-b.com/robot_log
Jul 23 19:22 www.site-o.com/robot_log
Jul 23 16:56 www.site-q.com/robot_log
Jul 23 11:37 www.site-a.com/robot_log
Jul 23 06:56 www.site-p.com/robot_log
Jul 23 05:05 www.site-e.com/robot_log
Jul 21 02:42 www.site-m.com/robot_log
Jul 21 01:30 www.site-d.com/robot_log
Jul 20 20:30 www.site-h.com/robot_log
Jul 14 18:47 www.site-r.com/robot_log
Jul 11 15:02 www.site-l.com/robot_log
Jul 11 10:00 www.site-i.com/robot_log
Jul 09 16:13 www.site-c.com/robot_log
Jul 09 00:09 www.site-t.com/robot_log
Jul 05 03:09 www.site-n.com/robot_log
Jul 05 03:05 www.site-s.com/robot_log
Jul 05 03:05 www.site-k.com/robot_log
Jul 04 23:53 www.site-f.com/robot_log

athinktank

10+ Year Member



 
Msg#: 15473 posted 9:52 pm on Jul 23, 2003 (gmt 0)

I would agree I monitor a few sites. Most get hit every day but a few don't

I concur. All over the map for my sites. Most all of the PR is the same on all of my sites so can not comment on the hight PR gets more attention theory.

mipapage

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 15473 posted 9:44 am on Jul 24, 2003 (gmt 0)

Hmmm...

Just a point of interest - as I mentioned above we get hit pretty much every day, but yesterday (the 'tomorrow' referred to in the subject) we got hammered. Checked back in the logs and sure enough there is a rough ten day pattern as to when we get a real thourough visit from g-bot.

Imaster

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 15473 posted 9:49 am on Jul 24, 2003 (gmt 0)

As I mentioned earlier, yes Google has been aggresively crawling the sites since a week or so. Good days are ahead :)

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 15473 posted 7:07 pm on Jul 24, 2003 (gmt 0)

Missing fresh tags for the past 4 days are back today. Lots of fresh seen.

JonB

10+ Year Member



 
Msg#: 15473 posted 7:15 pm on Jul 24, 2003 (gmt 0)

heavy spidering today from gbot.....

Dayo_UK

10+ Year Member



 
Msg#: 15473 posted 7:16 pm on Jul 24, 2003 (gmt 0)

This thread is worrying me.

I had seen Googlebot everyday but now no visit since 22nd :( - while others are reporting heavier activity I am reporting lighter - oh dear.

Dayo_UK

10+ Year Member



 
Msg#: 15473 posted 7:38 pm on Jul 24, 2003 (gmt 0)

Hi All - Now I know y'all are gonna say this is a coincedence but with reference to my last post above - I was getting desperate for a Googlebot visit and pressed the smiley face on the toolbar - and guess what 10mins later Googlebot started visiting - lol.

pchristensen

10+ Year Member



 
Msg#: 15473 posted 8:20 pm on Jul 24, 2003 (gmt 0)

My site is only about a two months old, but Google was spidering my page every two days or so.

Then, a few days ago, every cached page on the site now dates back to two months ago. What is happening?

-Paul

Napoleon



 
Msg#: 15473 posted 8:25 pm on Jul 24, 2003 (gmt 0)

I don't want to hear any of this. We had a disaster yesterday... the site has been offline for 36 hours and has another 18 to go.

I fear the worst. I just don't want to think about it!

Someone tell me you can survive 54 hours downtime!

bcolflesh

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 15473 posted 8:30 pm on Jul 24, 2003 (gmt 0)

I hope you have an uptime contract with the host!

Napoleon



 
Msg#: 15473 posted 8:45 pm on Jul 24, 2003 (gmt 0)

>> I hope you have an uptime contract with the host! <<

Oh... don't start me! It's been an absolute and total nightmare.

You know the feeling... distress, frustration and anger all rolled into one. Knowing that there is nothing you can personally do about it, and cursing everything that moves. Praying that Googlebot will stay away for a few days, but knowing it won't.

When it is back up tomorrow, the next phase kicks in. Worrying about the implications. Checking Google every 5 minutes for signs of impending disaster.

Someone tell me it will be OK... or you'll see a grown man cry!

kevinpate

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 15473 posted 9:53 pm on Jul 24, 2003 (gmt 0)

> Someone tell me you can survive 54 hours downtime

The small nfp site I agreed to service 18 months back, when I was absolutely greener than popeye's spinach, survived near on 3 weeks worth of dead time in April '02.

The prior hosting arrangment went dark sans warning, but they were sending word to the nfp org everything would be straightened out. Being both a newbie webhead and a volunteer, I had little choice but to defer to those who accepted the explanations. I did begin to start hunting alternatives just in case, a wise course of action as it turned out. All was well again in short order.

Excluding a couple of self-inflicted newbie hiccups, the general pattern over the past 15 months has been steadily improving visibility and traffic for our regularly expanding content base.

I'm sure it helped some that for our most common 2 word phrase the typical number of results returned are a tad over 1.4 million. It likely would have played differently in a field where the returns total 2, 4 or 6 fold that level.

incywincy

10+ Year Member



 
Msg#: 15473 posted 10:02 pm on Jul 24, 2003 (gmt 0)

sorry to say this but googlebot absolutely hammered my site today. 2277 pages out of a site total of 3241 pages, you had it right for me cazgh.

Stujoe

10+ Year Member



 
Msg#: 15473 posted 3:26 am on Jul 25, 2003 (gmt 0)

Seems googlebot has pretty much left my site alone today. Just a handful of hits and they were all requests for pages that have been deleted from my site and google results for a couple months.

xlcus

10+ Year Member



 
Msg#: 15473 posted 12:21 pm on Jul 25, 2003 (gmt 0)

I've only had a few Googlebot accesses today. Even on my higher PR pages. This is the first day in ages that I've seen a lull.

Napoleon



 
Msg#: 15473 posted 12:56 pm on Jul 25, 2003 (gmt 0)

Well... the site is back at its new host.... and you guys are telling me that crawling has scaled down! I think I need to get drunk!

too much information

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 15473 posted 1:00 pm on Jul 25, 2003 (gmt 0)

Dayo_UK, your trick didn't work for me. :(

I havn't seen GB in a LONG time. Last time she was around, I was editing some text (dynamically generated site) and I think I made her mad, because before that she was around all the time.

Does anyone think that sending flowers might help? ;)

This 41 message thread spans 2 pages: 41 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved