homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 41 message thread spans 2 pages: 41 ( [1] 2 > >     
Is Google spidering tomorrow?
Google spidering

 8:42 am on Jul 23, 2003 (gmt 0)

According to my logs, the Google Bot comes every 10 days and is due tomorrow.

Anyone agree?



 1:00 pm on Jul 23, 2003 (gmt 0)


For some perspective, the bot comes to my site pretty much every day, though I wouldn't call her regular.


 1:32 pm on Jul 23, 2003 (gmt 0)

If you have access to a server with many sites on, then you get to see how Googlebot' patterns are very different from site to site; page to page.

Sometimes daily, sometimes more like monthly.


 2:48 pm on Jul 23, 2003 (gmt 0)

I would agree I monitor a few sites. Most get hit every day but a few don't.
It seems the sites that get hit every day do better in serps have better PR and so on.
I think sites that have a higher PR get more attention form the bots.


 3:16 pm on Jul 23, 2003 (gmt 0)

My sites typically get hit daily. I'd be alarmed if Googlebot was absent for more than a few days.


 3:27 pm on Jul 23, 2003 (gmt 0)

Mine too! Am i one of the big boys now? :P


 3:39 pm on Jul 23, 2003 (gmt 0)

Getting hit virtually every day, and updated. Now 127 No. 1's on G, wish I was young like you lot and had the energy to do this for a living.



 3:48 pm on Jul 23, 2003 (gmt 0)

I think sites that have a higher PR get more attention form the bots.


Also sites that have regular fresh content will get hit more regularly.



 3:50 pm on Jul 23, 2003 (gmt 0)

My sites get hits from Google almost everyday, but since the last 3 days I am noticing a huge number of hits from Googlebot, so that calls for a new update soon :)


 4:16 pm on Jul 23, 2003 (gmt 0)

Googlebot has been gulping up pages from my site for about the last week. I did have a couple of important pages show up a few days ago for the first time but they disappeared yesterday. Second time that has happened in a week.


 4:32 pm on Jul 23, 2003 (gmt 0)

Could you guys told me what you think about this?

I rebuilt a site a month and a half ago (completly rebuilt) and it got crawled by Googlebot at the beginning of the month during 6 days in a row. Since then, nothing, no Googlebot.

Well, in the SERPs, I could find the old pages, but since the past 10 days, it came down from 20 pages to 2 pages only available today.

Would that mean that I will get all my new pages in the SERPs soon?

Thanks for the feedback.



 8:37 pm on Jul 23, 2003 (gmt 0)

I have a site (online at the end of March) that hasn't been freshed for many many weeks. It is at #1. However, at #2 is a page that points to that site, and that page has been fresh many times, but not recently.

Another site, first online at the beginning of May, and indexed on 2003-06-16 was freshed on every page, every single day from 2003-07-07 to 2003-07-19, but hasn't been freshed since then. [Exception: a newer page was freshed once on 2003-07-21 and just that page].


 9:07 pm on Jul 23, 2003 (gmt 0)

I would agree I monitor a few sites. Most get hit every day but a few don't.

Yeah, I run quite a lot of sites and I see quite a variation in the frequency of Googlebot accesses...

I run a script to check the last bot access for each site and order the list by date to see which sites were most recently indexed.

Here are the current results...

Jul 23 21:56 www.site-g.com/robot_log
Jul 23 21:48 www.site-j.com/robot_log
Jul 23 19:22 www.site-b.com/robot_log
Jul 23 19:22 www.site-o.com/robot_log
Jul 23 16:56 www.site-q.com/robot_log
Jul 23 11:37 www.site-a.com/robot_log
Jul 23 06:56 www.site-p.com/robot_log
Jul 23 05:05 www.site-e.com/robot_log
Jul 21 02:42 www.site-m.com/robot_log
Jul 21 01:30 www.site-d.com/robot_log
Jul 20 20:30 www.site-h.com/robot_log
Jul 14 18:47 www.site-r.com/robot_log
Jul 11 15:02 www.site-l.com/robot_log
Jul 11 10:00 www.site-i.com/robot_log
Jul 09 16:13 www.site-c.com/robot_log
Jul 09 00:09 www.site-t.com/robot_log
Jul 05 03:09 www.site-n.com/robot_log
Jul 05 03:05 www.site-s.com/robot_log
Jul 05 03:05 www.site-k.com/robot_log
Jul 04 23:53 www.site-f.com/robot_log


 9:52 pm on Jul 23, 2003 (gmt 0)

I would agree I monitor a few sites. Most get hit every day but a few don't

I concur. All over the map for my sites. Most all of the PR is the same on all of my sites so can not comment on the hight PR gets more attention theory.


 9:44 am on Jul 24, 2003 (gmt 0)


Just a point of interest - as I mentioned above we get hit pretty much every day, but yesterday (the 'tomorrow' referred to in the subject) we got hammered. Checked back in the logs and sure enough there is a rough ten day pattern as to when we get a real thourough visit from g-bot.


 9:49 am on Jul 24, 2003 (gmt 0)

As I mentioned earlier, yes Google has been aggresively crawling the sites since a week or so. Good days are ahead :)


 7:07 pm on Jul 24, 2003 (gmt 0)

Missing fresh tags for the past 4 days are back today. Lots of fresh seen.


 7:15 pm on Jul 24, 2003 (gmt 0)

heavy spidering today from gbot.....


 7:16 pm on Jul 24, 2003 (gmt 0)

This thread is worrying me.

I had seen Googlebot everyday but now no visit since 22nd :( - while others are reporting heavier activity I am reporting lighter - oh dear.


 7:38 pm on Jul 24, 2003 (gmt 0)

Hi All - Now I know y'all are gonna say this is a coincedence but with reference to my last post above - I was getting desperate for a Googlebot visit and pressed the smiley face on the toolbar - and guess what 10mins later Googlebot started visiting - lol.


 8:20 pm on Jul 24, 2003 (gmt 0)

My site is only about a two months old, but Google was spidering my page every two days or so.

Then, a few days ago, every cached page on the site now dates back to two months ago. What is happening?



 8:25 pm on Jul 24, 2003 (gmt 0)

I don't want to hear any of this. We had a disaster yesterday... the site has been offline for 36 hours and has another 18 to go.

I fear the worst. I just don't want to think about it!

Someone tell me you can survive 54 hours downtime!


 8:30 pm on Jul 24, 2003 (gmt 0)

I hope you have an uptime contract with the host!


 8:45 pm on Jul 24, 2003 (gmt 0)

>> I hope you have an uptime contract with the host! <<

Oh... don't start me! It's been an absolute and total nightmare.

You know the feeling... distress, frustration and anger all rolled into one. Knowing that there is nothing you can personally do about it, and cursing everything that moves. Praying that Googlebot will stay away for a few days, but knowing it won't.

When it is back up tomorrow, the next phase kicks in. Worrying about the implications. Checking Google every 5 minutes for signs of impending disaster.

Someone tell me it will be OK... or you'll see a grown man cry!


 9:53 pm on Jul 24, 2003 (gmt 0)

> Someone tell me you can survive 54 hours downtime

The small nfp site I agreed to service 18 months back, when I was absolutely greener than popeye's spinach, survived near on 3 weeks worth of dead time in April '02.

The prior hosting arrangment went dark sans warning, but they were sending word to the nfp org everything would be straightened out. Being both a newbie webhead and a volunteer, I had little choice but to defer to those who accepted the explanations. I did begin to start hunting alternatives just in case, a wise course of action as it turned out. All was well again in short order.

Excluding a couple of self-inflicted newbie hiccups, the general pattern over the past 15 months has been steadily improving visibility and traffic for our regularly expanding content base.

I'm sure it helped some that for our most common 2 word phrase the typical number of results returned are a tad over 1.4 million. It likely would have played differently in a field where the returns total 2, 4 or 6 fold that level.


 10:02 pm on Jul 24, 2003 (gmt 0)

sorry to say this but googlebot absolutely hammered my site today. 2277 pages out of a site total of 3241 pages, you had it right for me cazgh.


 3:26 am on Jul 25, 2003 (gmt 0)

Seems googlebot has pretty much left my site alone today. Just a handful of hits and they were all requests for pages that have been deleted from my site and google results for a couple months.


 12:21 pm on Jul 25, 2003 (gmt 0)

I've only had a few Googlebot accesses today. Even on my higher PR pages. This is the first day in ages that I've seen a lull.


 12:56 pm on Jul 25, 2003 (gmt 0)

Well... the site is back at its new host.... and you guys are telling me that crawling has scaled down! I think I need to get drunk!

too much information

 1:00 pm on Jul 25, 2003 (gmt 0)

Dayo_UK, your trick didn't work for me. :(

I havn't seen GB in a LONG time. Last time she was around, I was editing some text (dynamically generated site) and I think I made her mad, because before that she was around all the time.

Does anyone think that sending flowers might help? ;)

This 41 message thread spans 2 pages: 41 ( [1] 2 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved