Forum Moderators: open

Message Too Old, No Replies

Why has Googlebot lost it's appetite?

Month over month, googlebot visits are WAY down...

         

WebFusion

11:46 am on Apr 1, 2004 (gmt 0)

10+ Year Member



Hello all,

About 4 weeks or so ago, I had one of my PR7 sites experience the mysterious "dropped pages" syndrome that has been discussed here, goign from over 100k indexed pages to less than 3000.

Further, reported backlinks went from 1200+ to 10 (!)

A little background:

-Site is non-commercial with NO ourgoing affiliate links visible (any of the few sponsor links sprinkled thoughout the site use cgi-based redirects)

-Although we have exchanged some links with other sites in our genre, they are ALL on topic, with no more than 15 outgoing links on any one page (and NO bad neighborhoods)

-The site in question is updated everyday with ne non-commercial, on-topic content (mostly articles).

-The site does very well in Yahoo, ask, etc.

I've never done anything dodgy with the site, nor do I employ automated link exchanges, etc.

Having said all that...and despite STILL having a PR7, freshbot no longer visits, and googlebot visits in general have dropped to a handful per day for the last 3 weeks. In fact, the Yahoo/Slurp and Ask Jeeves/Teoma crawlers have grabbed 1000% more pages than google in the last 30 days...

So i guess my question is...what's up with the googlebot?

Schneewittchen

1:00 pm on Apr 1, 2004 (gmt 0)

10+ Year Member



welcome to the club. when i read your post so my last optimism to come back after "few" days in G serps is away. i have lost 99,9% of indexed pages, G-traffic is going down to zero. is it a kind of penalty? my PR is still 6, all backlinks listet but no crawling (only robots and index page) ,no cache update, my url*s nowhere be found in G. now, today is day 15 of horror for me. a never ending story? should i have no chance to come back without changing my url (after 4 years!) if... my site triggered a filter and i change my site now....Google never knows it, because bot is not crawling the changes.

MikeBeverley

4:06 pm on Apr 1, 2004 (gmt 0)

10+ Year Member



The loss of your backlinks will most likely have dropped your bot traffic considerably. This may not be completely Google's fault.
Many webmasters have started using redirects (like you say you do) in their linking to avoid PageRank leak, linking to bad neighbourhoods accidentaly, having too many outbound links on a page etc.

It may be that alot of your reciprocal linkers are no longer linking clearly to you.

Another reason you're hits are down could be Google's current tendancy not to deep crawl sites. Does your PageRank spread evenly over your site (PR6, PR5's and so on) or is it just your main page that is high?

WebFusion

9:43 pm on Apr 1, 2004 (gmt 0)

10+ Year Member



Until recently, Pr was spread preety much evenly across the site, with hte main page having a PR7, and the successive pages all Pr6, Pr5, Pr4 and so on. The lowest Pr was generally a PR4.

It should be noted that any actual "sites of interest" I linked to were direct links. Only redirect were used for the few affiliate/advertising links (mostly for tracking purposes).

Further, the site has a good deal of pages that are deep-lnked from other sites (articles, etc.), and still has over 700 PR5+ sites linking to it (not to mention 2 Pr 8's, and dozens of Pr6's etc.). After checking over 60% in the last 3 weeks or so (I've kept track of incoming links), all are still linking directly.

I'm leaning towards this being a (hopefully) temporary glitch, but I'm not holding my breath on that one. Luckily, we receive enough traffic from Yahoo/MSN and sites linking to us that all is not lost in terms of traffic...but a 50% drop is a splash of cold water, that's for sure.

Ah well...to be honest, I've given up on trying to pander to google's whims, and I'll stick to the tried and true: Clean Html and good, basic design. I think it's time I stopped letting google dictate deisng decisions, and focus on what got us here in the first place.

Having said that...if google fails to realize that it's "experiments" in algo changes belong in the lab, and not released in the wild until it's ready for prime time, then perhaps their days are truly numbered...

Yahoo's results are looking pretty good ;-)

Symbios

9:48 pm on Apr 1, 2004 (gmt 0)

10+ Year Member



Maybe you need to find a way to fresh up the pages, it looks like googlebot actively goes after new things, its been pretty busy on my sites so maybe U'm partly to blame.

Schneewittchen

10:22 pm on Apr 1, 2004 (gmt 0)

10+ Year Member



thats a new algo from G$$gle named
"normal-site-filter" or "this-page-is-too-relevant-for-listing-filter" you dont know? this is no longer a search engine. in 1 year, google have so many super-anti-spam-filters,.... the one and only listing is this: "BUY [some stupid keyword] AT EBAY.COM! CLICK HERE!"

Spine

10:39 pm on Apr 1, 2004 (gmt 0)

10+ Year Member



Here's something else I've seen on one of my sites that
Googlebot visits almost daily.
My stats software shows how many times Gbot has visited, and how many pages it has read. Usually there are 3-10 page views for each 'visit' from the bot, but for the past 4 days, it's been 5 visits and zero page views.

My HTML checks out with the W3c validator, and my links seem OK with the link checker. Scooter seems to still be spidering the site the same as ever.

I'm looking into possible causes, since another site I watch closely still has the normal ratio of visits to page views from Gbot.

Powdork

3:23 am on Apr 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Might the five visits represent get requests for robots.txt?

Spine

6:26 am on Apr 2, 2004 (gmt 0)

10+ Year Member



I don't have a way of knowing, but my robots.txt is there for the finding.

Powdork

7:28 am on Apr 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Your stat program may not count robots.txt as pageviews.

ChrisKud5

8:10 am on Apr 2, 2004 (gmt 0)

10+ Year Member



I put up a new link to a new page from my home on a PR 6 over 2 weeks ago and gbot has not even TRIED to look for the linked file. The index page is freshed daily, but does not want to read or follow links.

Maybe a little less focus should have be given to that lengthy moon office garbage april fools and a little more time kicking gbot in the butt to get his ass in gear.

zgb999

12:19 pm on Apr 2, 2004 (gmt 0)

10+ Year Member



From the pages that you lost how many of them have similar content in form of
- same title
- same description
- same first paragraph
- other content that is the same as on other pages?

Do you see a difference between the pages that you did not loose in comparison to the ones you lost?

Macro

12:56 pm on Apr 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have the same problem on one of my sites. Google has lost it's appetite.

We have hundreds of good links in, some of them from authority sites that have home pages of PR9 and PR10 (Though the links are from PR5 & PR6 pages). We have a fair amount of content (about 1000 pages) but all the major revisions we've made to content over the last few weeks is just not getting picked up. There has been no change in IP, robots.txt, and no (my educated guess) bad neighbourhood penalty, no duplicate content/title/description/first para/text (apart from the gif Nav Bar) and no loss of PR. In fact several of our pages went up to PR6 in the last update. The changes I refer to above were updating of content and didn't involve major changes to structure/linking/design.

Google please come back.

WebFusion

3:27 pm on Apr 2, 2004 (gmt 0)

10+ Year Member



From the pages that you lost how many of them have similar content in form of
- same title
- same description
- same first paragraph
- other content that is the same as on other pages?
Do you see a difference between the pages that you did not loose in comparison to the ones you lost?

In a word...none. Every single page is unique in both first paragrpah, description, title, meta tags...and so on. Other than utilizing the smae basic template (for both esae of use and graphical attractiveness), there is not a single thing about these pages that could be considered duplicate content.

As for the current # of pages...it continues to drop, and has fallen below 2000 as of this writing (despite our maintaining a PR7!?!)

Ah well...I've gorwn so tired of the rolloercoaster that it's just not worth worrying over anymore. I've been doing this since 1996, and the same thing hold true now that held true then. Develop a good resource, and the traffic follows. While google may not be able to get it's collective head out of it's rear, Yahoo/MSN/ASK are now sending us about 70% of the traffic levels we enjoyed with google ranking our site as it is desered to be ranked. Since it is (for all practical purposes) a non-profit, google is becoming a non-issue.

Kind of nice, really. A few short months ago, it may have been a catastrophe with the googlopoly on search, but with all the "new" players picking up the slack it's alot easier to sleep at night.

Schneewittchen

3:38 pm on Apr 2, 2004 (gmt 0)

10+ Year Member



@webfusion. same happened to me. (every day G drop more and more sites, from 200.000 to 100.000, to 60.000....and now i see only 9 (in words: nine!)) the end is...Gbot get the robots.txt and any images since 2 weeks now. no update and no "come back". hard times for us. the problem is: nobody knows why, nobody can help - the only what we can do is wait....still wait and hope.

Macro

4:25 pm on Apr 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



same thing happend to me....every day G drop more and more sites, from 200.000 to 100.000, to 60.000

Seriously!? How many sites have you got. Maybe that's a reason - too many sites - possibly automated creation of pages?

Schneewittchen

4:39 pm on Apr 2, 2004 (gmt 0)

10+ Year Member



[Seriously!? How many sites have you got. Maybe that's a reason - too many sites - possibly automated creation of pages?]
1.)
i have a community, it is dynamic content like:
www.example.com/index.php?Profile=Username1
www.example.com/index.php?Profile=Username2
www.example.com/index.php?Profile=Username3
and so on.

since 3 years it wasnt a problem.
2.)
then, i have a mirror @ www02.example.com- google knows it and crawl it like the www-domain in past. same PR (6)
3.) i have more than one page designs, every design is another subdirectory like this:
www.example.com/design1/ is the same as:
www02.example.com/design1/
4.)
my domain have 2 aliases:
www.ex-ample.com and www.exmple.com

i dont know how many sites i have...there are all dynamic...every day my site become 50 - 100 new members, and so 50-100 new sites.

it is a page like millions of others.

Macro

4:55 pm on Apr 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Ah, I see.

Schneewittchen

5:10 pm on Apr 2, 2004 (gmt 0)

10+ Year Member



when google drop all pages like: /index.php?Username=XX or guestbooks etc.... no problem for me. but G also drop my (STATIC) index-page and all other static sites. so you cant find my page with a sitename-search like: mypagename.com. many users came in past over google, because they have hear the name from friends and search for the page. now.. a lot of spammers with sites like: www.anotherdomain.com/mysitename.html become MY visits now. and i cant do anything against it...