Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: open

Message Too Old, No Replies

Is Google spidering tomorrow?

Google spidering

     
8:42 am on Jul 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 17, 2003
posts:76
votes: 0


According to my logs, the Google Bot comes every 10 days and is due tomorrow.

Anyone agree?

1:00 pm on July 23, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1375
votes: 0


cazgh,

For some perspective, the bot comes to my site pretty much every day, though I wouldn't call her regular.

1:32 pm on July 23, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 22, 2001
posts:3805
votes: 2


If you have access to a server with many sites on, then you get to see how Googlebot' patterns are very different from site to site; page to page.

Sometimes daily, sometimes more like monthly.

2:48 pm on July 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 16, 2003
posts:167
votes: 0


I would agree I monitor a few sites. Most get hit every day but a few don't.
It seems the sites that get hit every day do better in serps have better PR and so on.
I think sites that have a higher PR get more attention form the bots.
3:16 pm on July 23, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member rfgdxm1 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 12, 2002
posts:4479
votes: 0


My sites typically get hit daily. I'd be alarmed if Googlebot was absent for more than a few days.
3:27 pm on July 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 11, 2001
posts:44
votes: 0


Mine too! Am i one of the big boys now? :P
3:39 pm on July 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 15, 2003
posts:187
votes: 0


Getting hit virtually every day, and updated. Now 127 No. 1's on G, wish I was young like you lot and had the energy to do this for a living.

Granpops

3:48 pm on July 23, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7242
votes: 0


I think sites that have a higher PR get more attention form the bots.

Correct.

Also sites that have regular fresh content will get hit more regularly.

TJ

3:50 pm on July 23, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:960
votes: 0


My sites get hits from Google almost everyday, but since the last 3 days I am noticing a huge number of hits from Googlebot, so that calls for a new update soon :)
4:16 pm on July 23, 2003 (gmt 0)

New User

10+ Year Member

joined:July 14, 2003
posts:22
votes: 0


Googlebot has been gulping up pages from my site for about the last week. I did have a couple of important pages show up a few days ago for the first time but they disappeared yesterday. Second time that has happened in a week.
4:32 pm on July 23, 2003 (gmt 0)

New User

10+ Year Member

joined:July 10, 2003
posts:25
votes: 0


Could you guys told me what you think about this?

I rebuilt a site a month and a half ago (completly rebuilt) and it got crawled by Googlebot at the beginning of the month during 6 days in a row. Since then, nothing, no Googlebot.

Well, in the SERPs, I could find the old pages, but since the past 10 days, it came down from 20 pages to 2 pages only available today.

Would that mean that I will get all my new pages in the SERPs soon?

Thanks for the feedback.

Bobosse

8:37 pm on July 23, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0



I have a site (online at the end of March) that hasn't been freshed for many many weeks. It is at #1. However, at #2 is a page that points to that site, and that page has been fresh many times, but not recently.

Another site, first online at the beginning of May, and indexed on 2003-06-16 was freshed on every page, every single day from 2003-07-07 to 2003-07-19, but hasn't been freshed since then. [Exception: a newer page was freshed once on 2003-07-21 and just that page].

9:07 pm on July 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 5, 2002
posts:89
votes: 0


I would agree I monitor a few sites. Most get hit every day but a few don't.

Yeah, I run quite a lot of sites and I see quite a variation in the frequency of Googlebot accesses...

I run a script to check the last bot access for each site and order the list by date to see which sites were most recently indexed.

Here are the current results...

Jul 23 21:56 www.site-g.com/robot_log
Jul 23 21:48 www.site-j.com/robot_log
Jul 23 19:22 www.site-b.com/robot_log
Jul 23 19:22 www.site-o.com/robot_log
Jul 23 16:56 www.site-q.com/robot_log
Jul 23 11:37 www.site-a.com/robot_log
Jul 23 06:56 www.site-p.com/robot_log
Jul 23 05:05 www.site-e.com/robot_log
Jul 21 02:42 www.site-m.com/robot_log
Jul 21 01:30 www.site-d.com/robot_log
Jul 20 20:30 www.site-h.com/robot_log
Jul 14 18:47 www.site-r.com/robot_log
Jul 11 15:02 www.site-l.com/robot_log
Jul 11 10:00 www.site-i.com/robot_log
Jul 09 16:13 www.site-c.com/robot_log
Jul 09 00:09 www.site-t.com/robot_log
Jul 05 03:09 www.site-n.com/robot_log
Jul 05 03:05 www.site-s.com/robot_log
Jul 05 03:05 www.site-k.com/robot_log
Jul 04 23:53 www.site-f.com/robot_log

9:52 pm on July 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 18, 2002
posts:126
votes: 0


I would agree I monitor a few sites. Most get hit every day but a few don't

I concur. All over the map for my sites. Most all of the PR is the same on all of my sites so can not comment on the hight PR gets more attention theory.

9:44 am on July 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1375
votes: 0


Hmmm...

Just a point of interest - as I mentioned above we get hit pretty much every day, but yesterday (the 'tomorrow' referred to in the subject) we got hammered. Checked back in the logs and sure enough there is a rough ten day pattern as to when we get a real thourough visit from g-bot.

9:49 am on July 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:960
votes: 0


As I mentioned earlier, yes Google has been aggresively crawling the sites since a week or so. Good days are ahead :)
7:07 pm on July 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Missing fresh tags for the past 4 days are back today. Lots of fresh seen.
7:15 pm on July 24, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 10, 2001
posts:562
votes: 0


heavy spidering today from gbot.....

Dayo_UK

7:16 pm on July 24, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


This thread is worrying me.

I had seen Googlebot everyday but now no visit since 22nd :( - while others are reporting heavier activity I am reporting lighter - oh dear.

Dayo_UK

7:38 pm on July 24, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


Hi All - Now I know y'all are gonna say this is a coincedence but with reference to my last post above - I was getting desperate for a Googlebot visit and pressed the smiley face on the toolbar - and guess what 10mins later Googlebot started visiting - lol.
8:20 pm on July 24, 2003 (gmt 0)

New User

10+ Year Member

joined:June 10, 2003
posts:36
votes: 0


My site is only about a two months old, but Google was spidering my page every two days or so.

Then, a few days ago, every cached page on the site now dates back to two months ago. What is happening?

-Paul

8:25 pm on July 24, 2003 (gmt 0)

Senior Member

joined:Nov 20, 2000
posts:1336
votes: 0


I don't want to hear any of this. We had a disaster yesterday... the site has been offline for 36 hours and has another 18 to go.

I fear the worst. I just don't want to think about it!

Someone tell me you can survive 54 hours downtime!

8:30 pm on July 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


I hope you have an uptime contract with the host!
8:45 pm on July 24, 2003 (gmt 0)

Senior Member

joined:Nov 20, 2000
posts:1336
votes: 0


>> I hope you have an uptime contract with the host! <<

Oh... don't start me! It's been an absolute and total nightmare.

You know the feeling... distress, frustration and anger all rolled into one. Knowing that there is nothing you can personally do about it, and cursing everything that moves. Praying that Googlebot will stay away for a few days, but knowing it won't.

When it is back up tomorrow, the next phase kicks in. Worrying about the implications. Checking Google every 5 minutes for signs of impending disaster.

Someone tell me it will be OK... or you'll see a grown man cry!

9:53 pm on July 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 2, 2002
posts:1167
votes: 0


> Someone tell me you can survive 54 hours downtime

The small nfp site I agreed to service 18 months back, when I was absolutely greener than popeye's spinach, survived near on 3 weeks worth of dead time in April '02.

The prior hosting arrangment went dark sans warning, but they were sending word to the nfp org everything would be straightened out. Being both a newbie webhead and a volunteer, I had little choice but to defer to those who accepted the explanations. I did begin to start hunting alternatives just in case, a wise course of action as it turned out. All was well again in short order.

Excluding a couple of self-inflicted newbie hiccups, the general pattern over the past 15 months has been steadily improving visibility and traffic for our regularly expanding content base.

I'm sure it helped some that for our most common 2 word phrase the typical number of results returned are a tad over 1.4 million. It likely would have played differently in a field where the returns total 2, 4 or 6 fold that level.

10:02 pm on July 24, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Feb 5, 2002
posts:333
votes: 0


sorry to say this but googlebot absolutely hammered my site today. 2277 pages out of a site total of 3241 pages, you had it right for me cazgh.
3:26 am on July 25, 2003 (gmt 0)

New User

10+ Year Member

joined:July 14, 2003
posts:22
votes: 0


Seems googlebot has pretty much left my site alone today. Just a handful of hits and they were all requests for pages that have been deleted from my site and google results for a couple months.
12:21 pm on July 25, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 5, 2002
posts:89
votes: 0


I've only had a few Googlebot accesses today. Even on my higher PR pages. This is the first day in ages that I've seen a lull.
12:56 pm on July 25, 2003 (gmt 0)

Senior Member

joined:Nov 20, 2000
posts:1336
votes: 0


Well... the site is back at its new host.... and you guys are telling me that crawling has scaled down! I think I need to get drunk!
1:00 pm on July 25, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 16, 2003
posts:897
votes: 0


Dayo_UK, your trick didn't work for me. :(

I havn't seen GB in a LONG time. Last time she was around, I was editing some text (dynamically generated site) and I think I made her mad, because before that she was around all the time.

Does anyone think that sending flowers might help? ;)

This 41 message thread spans 2 pages: 41
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members