homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

GoogleBot - freshdeepbot...
Is the bot staying regular during all this indigestion?

 7:52 am on Jun 26, 2003 (gmt 0)

Something that I've noticed in our logs: Since the weekend, lets say Saturday the 21st, we have been visited by Googlebot a mere 3 times, and all on that day. When the little bot came around it managed to stir up two 404's and one 200. We would normally be visited by the 'bot twice every three days. The site is about 40 pages, nothing that large...

Saturday also marks the first day with a new version of our site, finally dynamic. The url's have been rewritten so that they appear the same as the old site, however some pages were 'lost' (ie never to return) with the new site.

Now, there are fresh tags out there, so the bot is working, but has there been a noticeable decrease in gbot activity while this update business settles down? Or have we done something wrong....?

We have been 'relatively' hammered by Slurp and the Fastbot, as well as that MSN bot, which seems to have increased our traffic from MSN.



 11:11 am on Jun 26, 2003 (gmt 0)

We would normally be visited by the 'bot twice every three days.

ie: get many hits from gbot on two separate occasions in a three day period.


 11:30 am on Jun 26, 2003 (gmt 0)


It looks like GoogleBot is having a nap... or busy doing something else. :)

She used to crawl a few pages of my sites every day, but I couldn't find any trace in the logs for the last 3 days.
A few other webmasters are facing the same "non-activity".



 11:42 am on Jun 26, 2003 (gmt 0)

Yeah, I have new site I'd hoped would be spidered and indexed by now, but I can only get googlebot to crawl pages that are directly linked from external, pre-indexed pages. In other words, it will only go one level deep from known pages. From my experience, even the old freshie would've done a lot more by now.

Its really casting a pall on the continuous update/freshdeepbot theories.


 11:49 am on Jun 26, 2003 (gmt 0)

Thanks for the replies, I'm glad that I'm not the only one.

Its really casting a pall on the continuous update/freshdeepbot theories.

Not so sure that is a theory, I believe that pre-esmeraleda GG mentioned that this was the case - ie it's fact...


 11:53 am on Jun 26, 2003 (gmt 0)

Right...freshdeepbot is freshdeepbot, its just not acting as I would expect it to in that role.

Its neither going deep nor enticed by the fresh content of new sites, anyway.


 12:29 pm on Jun 26, 2003 (gmt 0)

Right, I agree.

Not sure that it needs to be mentioned, but the fact that one bot is doing what was the work of two is different than the existence of the rolling-update.

GG said that in the future the update as we know it (once a month w/ dances etc.) will change.

However we have already seen freshdeepbot in action during the last deep crawl.


 12:30 pm on Jun 26, 2003 (gmt 0)

Freshbot not doing much but don't you think Google has some other, more pressing concerns at the moment?

Seems they are having serious problems with the transition to using freshdseep. Mainly missing pages/losing data. I would gladly sacrifice a few fresh visits a week to get ONE solid, reliable crawl.


 12:43 pm on Jun 26, 2003 (gmt 0)

Sure thing mfishy,

but there was an observable change in my logs wrt a vanishing gbot, tied with fresh results yesterday that our site is normally part of but wasn't this time, and the fact that we put up a new site just before gbot disappeared.

Put that all together and I needed to know if I either was in 'good company' or up the creek with the redesign.


 12:53 pm on Jun 26, 2003 (gmt 0)

I'm not much good at access log analysis, but I was scrolling through yesterday and observed Googlebot behaving oddly.
It came several times in the last 5 days, and it requested our robots.txt each time-- about 5 times.
Then it left.
Every time, it came back and requested robots.txt multiple times.
I don't think we have one, but that's never stopped the bot in the past. The site's been up probably 5 years at this point, with little fundamental change.
Finally, the last time (yesterday I think) it requested robots.txt 4 or 5 times, and then it downloaded one image-- a spacer.gif, I think-- and departed.

So I have no clue what's up. It's not spidering the site. Two weeks ago it did the whole site. It generally comes by every couple of weeks and noses around, making sure we're still there, doing most if not all of the site-- I'm not up on the various IP addresses, but we had several different ranges of IP address for them. The whole site's been spidered a million times; I haven't changed anything in months, as I've been working on an entire replacement site (boss finally realized how many inquiries we were getting through our website-- might be good to have more than a glorified brochure up there, no?). That's about to be uploaded, and will be almost entirely different-- but you can bet I'm leaving the old pages in place, with either a link or a redirect to their new counterparts! No way am I taking a chance on Googlebot spidering my new site... it's behaving too oddly!

swampy webber

 1:10 pm on Jun 26, 2003 (gmt 0)

Same thing here - robots.txt then leaving. What is it thinking? Come on in Googlebot....


 1:19 pm on Jun 26, 2003 (gmt 0)

same thing here - googlebot is just skimming my site. havent had a deep crawl for over a month now.


 1:19 pm on Jun 26, 2003 (gmt 0)

I was deep crawled this morning on 3 of my sites. Its up to something.


 1:26 pm on Jun 26, 2003 (gmt 0)

I was thinking Fredbot was perhaps slowly going deep but only on deep pages that have been changed since the last visit...so much for that theory.


 6:36 pm on Jun 26, 2003 (gmt 0)

I normally expect to see googlebot anywhere from 1000 - 2000 times a day. 6/20 - 7000 visits, but since then the average is down to about 30!

Most of the visits are just picking up index and robots - most strange, I was getting worried that I had somehow offended the bot but as others are seeing the same thing I'm not too worried - but curious.


 6:43 pm on Jun 26, 2003 (gmt 0)

Freshbot not doing much but don't you think Google has some other, more pressing concerns at the moment?

mfishy, lets hope G shares your viewpoint.


 6:47 pm on Jun 26, 2003 (gmt 0)

So far today, we've had 10 of our 36 pages hit by freshbot.


 11:43 pm on Jun 26, 2003 (gmt 0)

Last seen: 2003-06-21 22:36:32 UTC but I haven't checked the logs for 2003-06-26 yet (still 16 minutes of that day left).

I also saw the robots file attemped 2 times, about 20 minutes apart, nearly a week ago. That isn't usual.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved