Welcome to WebmasterWorld Guest from 54.145.53.251

Forum Moderators: open

Message Too Old, No Replies

GoogleBot - freshdeepbot...

Is the bot staying regular during all this indigestion?

     
7:52 am on Jun 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1375
votes: 0


Something that I've noticed in our logs: Since the weekend, lets say Saturday the 21st, we have been visited by Googlebot a mere 3 times, and all on that day. When the little bot came around it managed to stir up two 404's and one 200. We would normally be visited by the 'bot twice every three days. The site is about 40 pages, nothing that large...

Saturday also marks the first day with a new version of our site, finally dynamic. The url's have been rewritten so that they appear the same as the old site, however some pages were 'lost' (ie never to return) with the new site.

Now, there are fresh tags out there, so the bot is working, but has there been a noticeable decrease in gbot activity while this update business settles down? Or have we done something wrong....?


We have been 'relatively' hammered by Slurp and the Fastbot, as well as that MSN bot, which seems to have increased our traffic from MSN.
11:11 am on June 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1375
votes: 0


We would normally be visited by the 'bot twice every three days.

ie: get many hits from gbot on two separate occasions in a three day period.

11:30 am on June 26, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Dec 5, 2002
posts:219
votes: 0


mipapage,

It looks like GoogleBot is having a nap... or busy doing something else. :)

She used to crawl a few pages of my sites every day, but I couldn't find any trace in the logs for the last 3 days.
A few other webmasters are facing the same "non-activity".

Dan

11:42 am on June 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 20, 2003
posts:390
votes: 0


Yeah, I have new site I'd hoped would be spidered and indexed by now, but I can only get googlebot to crawl pages that are directly linked from external, pre-indexed pages. In other words, it will only go one level deep from known pages. From my experience, even the old freshie would've done a lot more by now.

Its really casting a pall on the continuous update/freshdeepbot theories.

11:49 am on June 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1375
votes: 0


Thanks for the replies, I'm glad that I'm not the only one.

Its really casting a pall on the continuous update/freshdeepbot theories.

Not so sure that is a theory, I believe that pre-esmeraleda GG mentioned that this was the case - ie it's fact...

11:53 am on June 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 20, 2003
posts:390
votes: 0


Right...freshdeepbot is freshdeepbot, its just not acting as I would expect it to in that role.

Its neither going deep nor enticed by the fresh content of new sites, anyway.

12:29 pm on June 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1375
votes: 0


Right, I agree.

Not sure that it needs to be mentioned, but the fact that one bot is doing what was the work of two is different than the existence of the rolling-update.

GG said that in the future the update as we know it (once a month w/ dances etc.) will change.

However we have already seen freshdeepbot in action during the last deep crawl.

12:30 pm on June 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 19, 2002
posts:2139
votes: 0


Freshbot not doing much but don't you think Google has some other, more pressing concerns at the moment?

Seems they are having serious problems with the transition to using freshdseep. Mainly missing pages/losing data. I would gladly sacrifice a few fresh visits a week to get ONE solid, reliable crawl.

12:43 pm on June 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 11, 2003
posts:1375
votes: 0


Sure thing mfishy,

but there was an observable change in my logs wrt a vanishing gbot, tied with fresh results yesterday that our site is normally part of but wasn't this time, and the fact that we put up a new site just before gbot disappeared.

Put that all together and I needed to know if I either was in 'good company' or up the creek with the redesign.

12:53 pm on June 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 18, 2003
posts:400
votes: 0


I'm not much good at access log analysis, but I was scrolling through yesterday and observed Googlebot behaving oddly.
It came several times in the last 5 days, and it requested our robots.txt each time-- about 5 times.
Then it left.
Every time, it came back and requested robots.txt multiple times.
I don't think we have one, but that's never stopped the bot in the past. The site's been up probably 5 years at this point, with little fundamental change.
Finally, the last time (yesterday I think) it requested robots.txt 4 or 5 times, and then it downloaded one image-- a spacer.gif, I think-- and departed.

So I have no clue what's up. It's not spidering the site. Two weeks ago it did the whole site. It generally comes by every couple of weeks and noses around, making sure we're still there, doing most if not all of the site-- I'm not up on the various IP addresses, but we had several different ranges of IP address for them. The whole site's been spidered a million times; I haven't changed anything in months, as I've been working on an entire replacement site (boss finally realized how many inquiries we were getting through our website-- might be good to have more than a glorified brochure up there, no?). That's about to be uploaded, and will be almost entirely different-- but you can bet I'm leaving the old pages in place, with either a link or a redirect to their new counterparts! No way am I taking a chance on Googlebot spidering my new site... it's behaving too oddly!

1:10 pm on June 26, 2003 (gmt 0)

New User

10+ Year Member

joined:Mar 3, 2003
posts:24
votes: 0


Same thing here - robots.txt then leaving. What is it thinking? Come on in Googlebot....
1:19 pm on June 26, 2003 (gmt 0)

New User

10+ Year Member

joined:June 26, 2003
posts:4
votes: 0


same thing here - googlebot is just skimming my site. havent had a deep crawl for over a month now.
1:19 pm on June 26, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 17, 2003
posts:384
votes: 0


I was deep crawled this morning on 3 of my sites. Its up to something.
1:26 pm on June 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 22, 2003
posts:1483
votes: 0


I was thinking Fredbot was perhaps slowly going deep but only on deep pages that have been changed since the last visit...so much for that theory.
6:36 pm on June 26, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 7, 2003
posts:90
votes: 0


I normally expect to see googlebot anywhere from 1000 - 2000 times a day. 6/20 - 7000 visits, but since then the average is down to about 30!

Most of the visits are just picking up index and robots - most strange, I was getting worried that I had somehow offended the bot but as others are seeing the same thing I'm not too worried - but curious.

6:43 pm on June 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 1, 2003
posts:1201
votes: 0


Freshbot not doing much but don't you think Google has some other, more pressing concerns at the moment?

mfishy, lets hope G shares your viewpoint.

6:47 pm on June 26, 2003 (gmt 0)

New User

10+ Year Member

joined:Apr 17, 2003
posts:26
votes: 0


So far today, we've had 10 of our 36 pages hit by freshbot.
11:43 pm on June 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Last seen: 2003-06-21 22:36:32 UTC but I haven't checked the logs for 2003-06-26 yet (still 16 minutes of that day left).

I also saw the robots file attemped 2 times, about 20 minutes apart, nearly a week ago. That isn't usual.