Forum Moderators: Robert Charlton & goodroi
The index growing on 64.233.179.104 does seem to be largely a Mozilla Googlebot generated index - and this new index is being built for the future - so can we say Mozilla Googlebot is now taking over from normal Googlebot.
OK ignore supplimentals etc for a moment - as all DCs have this problem and have a look at the cache dates for pages that are indexed...... some of these pages have only been fetched by Mozzilla Googlebot (even on the same day as normal Googlebot visited)
Eg. On the test DC I have a homepage cached 30th November at 5:40 - fetched by Mozilla Googlebot - while on the other DCs it is cached on 30th November at 3:40 - fetched by normal Googlebot.
So in many ways this does look like building a whole new index parrellel to the existing index - with largely Mozilla Googlebot crawl data.
Some pages appear very old - eg another page is cached on the test dc on 6th November - but on the other dcs it has cache in December - checking the logs - 6th November was the last time Mozilla Googlebot visited this page.
OK - there are pages in the test DC only visited by normal Googlebot - however, pages crawled by Mozilla Googlebot do not appear on other DCs.
The newest pages on the DC crawled by Mozilla Googlebot seem to be in November - eg no pages crawled by Mozilla Googlebot in December have made it to the index yet.
Some pages crawled by Mozilla Googlebot in November have not made it to the index - so I dont know if G are working with a sample data size......
For confirmation that this is a whole new build of the index MC said on his blog:-
"the test data center certainly has some different crawling and indexing characteristics."
OK - folks remember also that MC said that this index will roll out in months and is in a test state so I guess no need for early panic stations and slagging of Google in this thread.
Now 301s, 302s, Canonicals - for me a lot more 301s Google has crawled and indexed correctly. 302s - still lots in the index (mainly supplimentals) - not seeing any new 302s that show the url of the linking site but the content of the destination site (seeing the newest at about August 2005 time) - no doubt others may find some.
What are other observations people have seen with the new crawling and indexing on this test dc.
My site went missing (URL only, then supplemental) at the earliest stages of Jagger.
I really hope I make it through the filter this time around... I've been good. I added meta info to my pages... I removed duplicate words in title
Formerly:
"Domain.com - Article Name"
Currently:
"Article Name"
I just don't know what G wants from me in terms of getting these site penalties removed.
I will be crossing my fingers and watching this Test DC far more than I should be. Hoping for the best - and the best of luck to you all as well.
However there is still some old crap in the results that with cache dates from Nov 2004.
Out of the 60 or so well paying, high traffic articles I have (well... pre-jagger they were anyways) - 1 of them is showing up supplemental.
I see this as a good thing - as the article is a far more competitive sub-topic of my niche than what I typically go for. I wrote it, threw it up... thought it sounded a bit spammy, but left it anyways. Sure enough... supplemental.
I wonder what filter is causing this. It seems that some filters are already in place in the test DC... but which ones.
Is there something I am missing or should I expect a massive reduction in traffic?
Any help would be appreciated.
Is there something I am missing or should I expect a massive reduction in traffic?
Could google really get rid of so many pages, or has the Test DC got a limit on the total number of pages in its index?
Two sites i know of have directory sites that request links back to the main site. They will offer things like two or three links for one etc.
The main site cleans up with loads of one way links to internal pages etc and the two sites in question doing this are racing to the top of the serps for almost every related keyword in their commercial sector.
In effect its like a link farm type operation, but google cant spot it. I read matts blog and they claim they can spot linking farms and link buying structures of this type but evidence indicates to me that sites doing this using this method are cleaning up - well the two currently actively doing this are.
Meanwhile, early indications at 179.104 show the two sites in question at 1 for everything, so clearly crime does pay!
1. Similar content within the site
Many here have reported large sites with template pages - although each page may contain something unique, like a product description, the template content and the meta tags are very similar and outweigh the unique content on the page. These sites are reporting that many of their pages are going Supplemental.
2. External websites with "duplicate" content, such as scrapers or content thieves. The more I look around, and read that people are reporting this content still exists around the web, hmmm. I think that these pages may not be going supplemental, but have an overall site penalty applied near the end of the re-indexing. That may be why scrapers show up and last a while, because it takes a bit of time for the algo to spider and re-spider and double check however it makes the judgement, but eventually it catches up. I haven't been able to figure out if the site which originally published the content is also possibly penalized in the end?
.
In the other thread on the test DC update it was mentioned that Mozilla Googlebot is an anti spam bot.
I would have agreed with that statement a few months ago - however the fact remains that Mozilla Googlebot is adding pages to the index on the test DC.
So perhaps the purpose of the bot has changed over the last few months - the crawling activity has changed and the bot seems to be crawling in a more hierarchical manner.
>>I would have agreed with that statement a few months ago - however the fact remains that Mozilla Googlebot is adding pages to the index on the test DC.<<
But the questions are still:
- Mostly its "unhealthy sites" that Mozilla Googlebot or what I call "spam detective" visits.
- Wonder whether those added pages you mentioned are ranking at all!
And of course: wish you Dayo_UK a happy "uncanonical" 2006 :-)
They rank if the site is ok (eg not suffering from Canonical problems)
I agree that Mozilla Googlebot might be looking at sites that have had problems (not necessarily spam - eg Canonical problems amongst other things.)
www.foo_foo.com/Help/
Well, I've got nothing on my website with that address and a quick check in Google indicated they haven't ever seen that URL either. So what is that nasty little pest checking for?!?!
Now I must admit that three months ago my website might have returned a 301 that pointed to the closest page on my site - but I've learned that is a Google no-no. Well, my server fed that insect with a 404 and it went away.
Now if it would only go back and give my site the clean bill of health that it deserves...
One thing I know is that 2006 has to be better than 2005 when it comes to me and Google. With 500 referals a month (<-- no typo there) on average since October 1st, I don't think it can go downhill from there. That being said...
May you all be blessed with a prosperous 2006.
All of this is in my humble opinion (IMHO), not presented as fact but based on 15 years of experience working in network engineering & programming, combined with a bit of common sense.
Is it the Bot, Or The Algo?
What have the search engine bots been created for?
Would that be to crawl webpages, following links, and copying the contents of the pages back to somewhere where the Algo can then slice and dice, count links, anchor text, figure word density, word proximity, the content of tags, make a determination on what is spam, and what is not, and then make decisions on how to rank them for display in the Serps?
Or,
The Bot is written to make these decisions before the Algo even gets to look at it. (there is not enough bandwith for this to even be a considerstion)
IMHO (notice I use IMHO)
If this is the new bot that Google is working on to be used to crawl the web in the future, it is an attempt to be better possibly more efficient, at copying the contents of the webpages, and following links, possibly following javascript to find redirects and such(and in this case it might make sense to not index, and even flag a page for no-index for the redirects). It Might be even an attempt to be able to read and follow flash files.
Back To Watching
Newbies, Do Not Confuse The Number Of Posts, With The Quality Of Posts.
WW_Watcher
Luckily I have an idea of what the problem might be because I can see that it's spidering certain parts of the site. These pages may have caused the penalty because they were php error pages. What is the best way to remove them? Noindex tag or a 404?
Even from the one ip it added pages to the test dc (still not showing test data) - although only a sample of pages in my case.
MC said the test dc has different indexing and crawling charateristic - pretty heavy hint there IMO.
Happy new year to everybody!
I think they first place the partly fixed codes/results in the serps and then make the real update to give those lost sites there PR and ranking again.
Yes, that is the question - G may have fixed some canonical issues - but where is the rank gone for these pages/when does the rank return to these pages.
PR supposed to be updated continually behind the scenes (so I heard/it was quoted) - but sites that have had the canonical problem for a while seem to have had the PR soul ripped out of them.
Hopefully you are correct in the next stage.
I think when there is a mozilla-googlebot-from-one-IP-address activity, then and only then it would be a spam detective that always result in a site penalty.
Are you sure that your pages indexed by the mozilla-one-ip-googlebot werent indexed before by the old googlebot?
Looks to me like mozilla googlebots retake a look at the pages indexed previously by googlebots and determine their quality.
That's the scenario for me..
- Site is published
- Googlebots crawl the site heavily
- Site pages go into google index and serps
- Thousands of google referrals for a few weeks
- Mozilla googlebot crawl the site heavily again
- A penalty
I have seen a similar routine. I had noticed that the incoming links are not being calculated until after the initial inclusion. 10 days or so later when the incoming links are calculated ... Smack! ... Penalty Time.
My main problem is that when sites are old and have been at the top of the serps for many years, they have a load of rubbish incoming links from webwasters who just publish google results to boost their page relevancy. I assume google thinks that I am in league with these sites and slaps me with the penalty ... nothing could be further from the truth. Never had to mutual link and never paid for hits etc.
I think this is the link assassination technique that has been discussed in other forums ... Even if google think it can't happen. I'm fully expecting to loose my PR again in a week or two.
Cheers
Col
I have a site that was launched a month or so ago and has only been deep crawled by Mozilla Googlebot - 100% certain of that.
Those pages are showing in the test DCs with the cache date of the Mozilla Googlebot crawl. Pages are not showing in any other dcs.
www.mydomainname.co.uk does not exist; there is no cache for this page
Has this page been "banned" - if yes, why are the inside pages ranking (and rankling well)
Any ideas please....?
