Forum Moderators: Robert Charlton & goodroi
1.Link to a new page added to site map - Google cashes the site map but doesnt follow the listed link and cashe the new page off it. Why might Google cashe a page yet ignore the content on it?
2. A PR5 page with lots of links to it from authority sites (3 PR7s links to it for a start off), the page is about blue widgets and should rank high for the blue-widget search term yet, another page about blue widgets on the site (but not as detailed and has zero backlinks to it)features at the top of the serps rather than the one thats more specific that has loads of authority backlinks.
Any ideas please post, im at a loss trying to second guess this?.
... unless Google has raised their unique content filter to "must have at least 90% unique content" I don't have an explanation.
If it has become 90%, is it that unreasonable? It would be a very effective way of cleaning out pages that aren't genuinely unique. I'm not saying it's the problem, of course...
Apart from that, and although I know I really shouldn't refer to an earlier post of mine in the previous instalment of this thread: if my rather callous comment on the absence of a certain member resulted in a very valuable contribution soon afterwards, I don't regret having written it, even though, once again, my sense of discretion was entirely lacking. So it goes. It's good to see him onboard when people are desperate.
For instance, my robots.txt looks like:
User-agent: *
Allow: /
User-agent: mediapartners-Google
Allow: /
User-agent: Googlebot
disallow:/
I posted two messages on MC's blog early this week, but he removed them because my questions were very much related to BH SEO. I guess Black Hats aren't supposed to post on his blog. haha. Anyhow, here's my thought on this: If they're going to have a proxy, they better OBEY the rules specified in my robots.txt. For fearful reasons (Yeah, Google banning my Adsense account), I don't allow Googlebot to crawl my site, but I definitely allow Mediapartners-Google to display ads. I only allow non-Googlebot bots to enter my site including mediapartners. I'm fine mediapartners doing duty work as long as they STRICTLY follow my robots.txt rules. I have a few concerns over this issue:
If I don't allow Googlebot in, then there's a possibility that my site's content might not be fetched which will allow adsense to display relevant ads. Let's consider the following scenario: Googlebot comes in, my site says, "Googlebot, I don't let you in, but I allow Mediapartners in", so Googlebot goes back tells proxy that the site didn't allow me. Now, when I ask Adsense to display ads, Adsense will check with proxy to find if that content is already there for that URL. In this case, proxy says "no, the site denied entering". IF adsense completely relies upon proxy, then it will say the same thing to me... "Sorry bud, I don't have any ads for your page since you didn't let me (all services/proxy) in and it displays public service ads." But if the proxy says "Hey Adsense, no, the site denied letting Googlebot in, but the robots.txt says to allow Mediapartners in, so you might want to go there again and see if you can find anything and if you do, show them the relevant ads." Now as mediapartners gets the page content, then I EXPECT (and NOT HOPE) the proxy to mark that page as "mediapartners-only" page and not show that page in Google's search index.
I also asked Matt Cutts if they are going to consolidate how the User Agents are currently identified as while crawling. I read somewhere that Google UA's are now identified as "Mozilla... something". If this is going to be their ultimate outcome, they better give us some documentation on allowing/disallowing Google-service-specific bots, how they plan on obeying robots.txt AND whether or not they are strictly going to follow robots.txt for all subdomains/domains across the planet. They will undoubtedly save tons of bandwidth with the new proxy system, but they will need more computational power, which I think they don't have any problems with.
I used the following my robots.txt in my scenario above:
#BOF#
User-agent: *
Allow: /
User-agent: mediapartners-Google
Allow: /
User-agent: Googlebot
disallow:/
#EOF#
What is your take on this?
Thanks for your input and sorry for the long post.
All of my other website pages also dropped. I write articles & my content is unique.
This explains what I have been witnessing on a very small, very niche, single product book site a client of mine has.
Just before BigDaddy we uploaded 300 pages of unique images from the book complete with captions. This had the desired effect with all 300 being sucked up by Google and traffic to the site multiplying nicely - bear in mind the site only had 15 pages before that.
BigDaddy then happened and all 300 new pages simply disappeared. Just today, I notice four of the image pages have found there way back in.
This is also consistent with a comment Matt Cutts made recently about crawl depth being a factor of PR; the healthier your link development, the more appetite G has for your pages.
So the crawl behaviour appears to be more discerning now as a consequence of these changes. Which makes sense from Google's perspective, even if it is frustrating in the meantime. So, back we go to making sure our sites are loved by other sites :-)
Oh and RichTC, I reckon your first point is exactly the issue MC was referring to about crawl depth and PR; it doesn't matter how much content you make available to Google, it's appetite to crawl it is independent of knowing it exists and this includes any data you give it by way of Google SiteMaps. Didn't used to be like this.
Steve
Note "classic" and "has used". My feeling is that BD has changed the rules and other trust-based factors are dictating crawl depth. Still comes back to good quality linking strategies though.
For fearful reasons (Yeah, Google banning my Adsense account), I don't allow Googlebot to crawl my site, but I definitely allow Mediapartners-Google to display ads
pixellion, if I were Google I would definitely stop displaying ads on your site and make it obligatory to let in both: googlebot and mediapartners bot.
Why?
Because, AdSense is not a never-ending money river. this money comes from somewhere, and this 'somehwere' are most probably some other guys who pay for advertising. if adsense quality decreases even more, google will lose its revenue stream.
so it is natural they won't display proper ads on your site and similar ones.
in fact, as an advertiser, i am happy about it.
Or...if we (and hopefully Google) could just be sensible about this for a second, we could recognise that "unique" is NOT a magic good versus evil metric. Any search service, including Google and a growing number of vertical (meta-) search services will, by definition, be comprised of largely duplicate content. Clearly, these service are not spam.
Is a Website providing a technical service (such as a sophisticated vertical search engine) really to be considered less valuable than a Website comprising a bunch of "original" text.
Anyway. I don't think uniqueness has anything to do with the bug we are seeing. Anyone else started to leak pages again? We are starting to lose most of the new pages that were added in the last week or so.
This is also consistent with a comment Matt Cutts made recently about crawl depth being a factor of PR
Could be true PR 6 homepage lost 50% of pages. PR 5 90%.
I said this because I believe it to be as important as your actual PR value. If your links are increasing regularly, then Google will be more interested in crawling.
But then I didn't mean to imply that this was going to be that simple to work out, just that if they are ramping up this kind of contingency, then we will see some significant changes on sites which, amongst other things, have stale links.
Steve
This morning I see 4380 results and page 7-the end of the google listings shows all supplemental results.
Still being crawled furiously too.
so how can new sites ever get off the ground with the fear of the sandbox & now poor crawling due to low LP. plus this also doesn't explain how PR4/5 sites have dropped 1000's of pages
Prior to March 9, I had 42,500 sites in BD--all of them with caches from July & August 2005. I contacted Google and received a reply within 3 hours notifying me that they were referring my email to the engineering team. Within a week, the number of sites with fresh cached pages went to 400,000. But still the same rank.
Then I found that the same programming firm that triggered the dup content penalty incorrectly used 302s instead of 301s. I cleaned that up a month ago. Still the same rankings.
I have read that Google penalties last from 1-6 months. I certainly hope so since my pain has been almost unbearable for 4 1/2 months.
Any ideas/suggestions?
I have a common design to my site, and the common elements, tables, etc would easily account for 20% of the site.
I could understand 90% unique text, but not 90% total content.
Hardly a year goes by that Google doesn't have 3 or 4 algo fiasco's that obliterate webmasters in a zero sum game. Over the last 3 years, if I can get 2 to 3 months of Good rankings from Google, then that's about all I can expect from them. If I don't get hit by an algo change, then I get hit by a FUBAR they can't fix and on average, my pages are in the Google toilet 9 or 10 months out of the year.
With one exception: I have 1 website that has "authority" status in Google [because of all the .gov, .edu, .org and newspapers, etc. that link to it]. With that site, I can get in the top 10 or 20 for just about any term I want - just because it has "authority" status. But since it's in just one niche area, from a professional standpoint I'm limited on what I can add to it.
Google SEO can be boiled down to that one fact: Get a website with "authority" status and you can then milk google like one used to be able to when all you had to do was get 100s of non relevant links - which worked from 1999 through Oct. 2003.
Sadly, that's all you need to know about SEO for Google. In my sarcastic opinion (IMSO)
But with my other sites, 2 or 3 months out of the year of mediocre to good rankings is about all I can expect with Google. Their FUBARS have gotten so bad and so commonplace, that my attitude is now: If I write this content or build this new website, can Ask, MSN and Yahoo give me enough traffic to make it worth it?
I checked his site, he has 301 to www in place, has no supplemental pages and has a PR5 (for what its worth) now, he has always checked to see when new pages get into G using the site: command, has for the last 3 years he says, he swears that those pages were in G for atleast 3 months and had PR.
He now wishes he had known what I told him so he could have looked for which pages vanished and when, as his downsizing was in definate stages, not the odd one or two pages, he remembers thinking when he did site: instead of up/down by 1 or 2 it seemed as though "folders falling of a shelf" his words.
So the question is, has Google lost the ability to crawl php? when it could and did in the past or is google forcing the ReWrite rule
its not just php my site that has dropped 70% of its pages is all html and I've had pages up for weeks that are still sat uncrawled - fortunately MSN & Y has crawled these pages a good few times and are now sat and ranking well - so come on G sort your dam act out!
When you are talking about 90% unique content to avoid duplicate content, does this exclude HTML tags, scripts and layout.
I was thinking of it as just the text content, not all the tags etc. Again though, I don't know if this is a factor in things. Really, I don't what the hec is doing it, although part of it is canonical, no doubt. But I do know that BD not only didn't affect me, it helped (just like Florida, Jagger, and the rest). I'm not trying to crow about it, truly (I've read tens of thousands of update posts since I joined), and granted my site is niche (although the canonical problems were taken care of 3 years ago), but I rock on some very competitive searches, and some of the less succesful of them went from #2-3 to #1 in BD. So, it's not a total systemic problem, and not random. It's site-specific. Maybe it's a temporary crawling/indexing glitch that is triggered by certain factors on a site, which will soon be corrected, and all those missing pages will soon return, or maybe not (I haven't seen any reports from those who were affected by the Sandbox suddenly telling us they're at number one on their intended serps, and the Florida casualties never really bounced back). If it's a "not", some folks are going to have to rethink their game-plans, and move on.
About a week ago or so I noticed results dropping, all I had was the main page and 58 "supplementals" which were all pages from July/August 2005 (they had cached pages back then but just never included them in the index) Anyway as of today all the supplementals are gone and a site:xyz.com search produces 1 result - my main page and it's not the text from my main page in the description but the description from dmoz and the google directory which is showing. So basically I'm back in "sandbox purgatory" or possibly even worse, plain old "I don't really exist hell".
The thing is I can't work out why, for the life of me. It's a non-profit fansite for an actor who is a household name. The content is original, in fact so original that I can honestly say there is a lot of material on the site which can't be found anywhere else on the web - because we actually scanned it, or wrote it. We have inbound links from PR 9's like IMDB and DMOZ as well as lots of 6's and 7's. We have won awards for being a great site for content. We are on Page 1 for both Yahoo and MSN for searches on our main keywords - we are if not the best, certainly in the top 2 sites for our content type, which is why we have plenty of inbounds from peoples blogs, livejournals, forums etc to our main page and also deeper pages. I just wanted to post that I concur with the view that google may be broken in some way.
What may be the problem in our case is almost all of our pages are no archive - because we have had problems with plagiarism so we have a few ip ranges banned and we don't want the site cached. It is interesting that after BD came in the only pages that showed for our site were those from before we made the pages no-archive, that's why we had a bunch of supplementals that were cached in July/August 2005. I tried adding a sitemap hoping it might help - interestingly it came up and said pages were "partially-indexed" referring to the fact they are "no-archive" and then in a further explanation they said we "try to fully index pages" so I wonder does BD react differently to no-archive? are we being penalised? do you no longer have the ability to opt out of the cache and still appear in the index? Anyone have any idea or experienced any problems with no-archive pages?
I would appreciate any feedback if anyone has experienced anything similar. Thanks.
Yes, you do have company -- and no, it doesn't seem to make much sense.
