Welcome to WebmasterWorld Guest from 54.221.30.139

Forum Moderators: open

Message Too Old, No Replies

GoogleBot - freshdeepbot...

Is the bot staying regular during all this indigestion?

     

mipapage

7:52 am on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Something that I've noticed in our logs: Since the weekend, lets say Saturday the 21st, we have been visited by Googlebot a mere 3 times, and all on that day. When the little bot came around it managed to stir up two 404's and one 200. We would normally be visited by the 'bot twice every three days. The site is about 40 pages, nothing that large...

Saturday also marks the first day with a new version of our site, finally dynamic. The url's have been rewritten so that they appear the same as the old site, however some pages were 'lost' (ie never to return) with the new site.

Now, there are fresh tags out there, so the bot is working, but has there been a noticeable decrease in gbot activity while this update business settles down? Or have we done something wrong....?


We have been 'relatively' hammered by Slurp and the Fastbot, as well as that MSN bot, which seems to have increased our traffic from MSN.

mipapage

11:11 am on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We would normally be visited by the 'bot twice every three days.

ie: get many hits from gbot on two separate occasions in a three day period.

hetzeld

11:30 am on Jun 26, 2003 (gmt 0)

10+ Year Member



mipapage,

It looks like GoogleBot is having a nap... or busy doing something else. :)

She used to crawl a few pages of my sites every day, but I couldn't find any trace in the logs for the last 3 days.
A few other webmasters are facing the same "non-activity".

Dan

Dolemite

11:42 am on Jun 26, 2003 (gmt 0)

10+ Year Member



Yeah, I have new site I'd hoped would be spidered and indexed by now, but I can only get googlebot to crawl pages that are directly linked from external, pre-indexed pages. In other words, it will only go one level deep from known pages. From my experience, even the old freshie would've done a lot more by now.

Its really casting a pall on the continuous update/freshdeepbot theories.

mipapage

11:49 am on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for the replies, I'm glad that I'm not the only one.

Its really casting a pall on the continuous update/freshdeepbot theories.

Not so sure that is a theory, I believe that pre-esmeraleda GG mentioned that this was the case - ie it's fact...

Dolemite

11:53 am on Jun 26, 2003 (gmt 0)

10+ Year Member



Right...freshdeepbot is freshdeepbot, its just not acting as I would expect it to in that role.

Its neither going deep nor enticed by the fresh content of new sites, anyway.

mipapage

12:29 pm on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Right, I agree.

Not sure that it needs to be mentioned, but the fact that one bot is doing what was the work of two is different than the existence of the rolling-update.

GG said that in the future the update as we know it (once a month w/ dances etc.) will change.

However we have already seen freshdeepbot in action during the last deep crawl.

mfishy

12:30 pm on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Freshbot not doing much but don't you think Google has some other, more pressing concerns at the moment?

Seems they are having serious problems with the transition to using freshdseep. Mainly missing pages/losing data. I would gladly sacrifice a few fresh visits a week to get ONE solid, reliable crawl.

mipapage

12:43 pm on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Sure thing mfishy,

but there was an observable change in my logs wrt a vanishing gbot, tied with fresh results yesterday that our site is normally part of but wasn't this time, and the fact that we put up a new site just before gbot disappeared.

Put that all together and I needed to know if I either was in 'good company' or up the creek with the redesign.

dragonlady7

12:53 pm on Jun 26, 2003 (gmt 0)

10+ Year Member



I'm not much good at access log analysis, but I was scrolling through yesterday and observed Googlebot behaving oddly.
It came several times in the last 5 days, and it requested our robots.txt each time-- about 5 times.
Then it left.
Every time, it came back and requested robots.txt multiple times.
I don't think we have one, but that's never stopped the bot in the past. The site's been up probably 5 years at this point, with little fundamental change.
Finally, the last time (yesterday I think) it requested robots.txt 4 or 5 times, and then it downloaded one image-- a spacer.gif, I think-- and departed.

So I have no clue what's up. It's not spidering the site. Two weeks ago it did the whole site. It generally comes by every couple of weeks and noses around, making sure we're still there, doing most if not all of the site-- I'm not up on the various IP addresses, but we had several different ranges of IP address for them. The whole site's been spidered a million times; I haven't changed anything in months, as I've been working on an entire replacement site (boss finally realized how many inquiries we were getting through our website-- might be good to have more than a glorified brochure up there, no?). That's about to be uploaded, and will be almost entirely different-- but you can bet I'm leaving the old pages in place, with either a link or a redirect to their new counterparts! No way am I taking a chance on Googlebot spidering my new site... it's behaving too oddly!

swampy webber

1:10 pm on Jun 26, 2003 (gmt 0)

10+ Year Member



Same thing here - robots.txt then leaving. What is it thinking? Come on in Googlebot....

KDE_Head

1:19 pm on Jun 26, 2003 (gmt 0)

10+ Year Member



same thing here - googlebot is just skimming my site. havent had a deep crawl for over a month now.

ncsuk

1:19 pm on Jun 26, 2003 (gmt 0)

10+ Year Member



I was deep crawled this morning on 3 of my sites. Its up to something.

skipfactor

1:26 pm on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I was thinking Fredbot was perhaps slowly going deep but only on deep pages that have been changed since the last visit...so much for that theory.

why2kit

6:36 pm on Jun 26, 2003 (gmt 0)

10+ Year Member



I normally expect to see googlebot anywhere from 1000 - 2000 times a day. 6/20 - 7000 visits, but since then the average is down to about 30!

Most of the visits are just picking up index and robots - most strange, I was getting worried that I had somehow offended the bot but as others are seeing the same thing I'm not too worried - but curious.

Kirby

6:43 pm on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Freshbot not doing much but don't you think Google has some other, more pressing concerns at the moment?

mfishy, lets hope G shares your viewpoint.

kmtell

6:47 pm on Jun 26, 2003 (gmt 0)

10+ Year Member



So far today, we've had 10 of our 36 pages hit by freshbot.

g1smd

11:43 pm on Jun 26, 2003 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Last seen: 2003-06-21 22:36:32 UTC but I haven't checked the logs for 2003-06-26 yet (still 16 minutes of that day left).

I also saw the robots file attemped 2 times, about 20 minutes apart, nearly a week ago. That isn't usual.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month