Forum Moderators: Robert Charlton & goodroi
With deep-inside, low-level pages now being excluded from Google's index (the site: search reports a loss of 1/3+ of my pages from the index), is it no longer true that Content Is King?
I have ideas to create dozens and dozens of new, tightly focused, detail-oriented pages. Trouble is, it is only natural--given my site design--to link these pages four or five levels deep.
Post-Big Daddy, it seems that the Google crawl/index depth is based on PR. With my site having "only" a PR5, this implies that these new four- or five-levels-deep pages will never be crawled, or if crawled, never be indexed.
From a SERPs standpoint, what's the use of creating these new pages? If SE visitors will never find these new pages, why bother?
Whatever happened to Brett Tabke's oft-repeated dictum: "Build one page of quality content per day. Google loves content, lots of quality content. Broad based over a wide range of keywords..."
If a tree falls in a forest with no one to hear it, does it make a sound?
As many here will probably agree, the thing to do is look for traffic outside of the Googleplex and keep creating content. Your mileage may vary depending on how much time you have.
I would also suggest you assume this is a temporary thing; either because there's some bug in the system, or until some new information comes to light that suggests a means of combating it.
What else can you do? Google need content and in the best designed websites the actual detail stuff, e.g. your page on green plastic widgets from Japan or whatever, will have their own special place on your site. If it is properly designed this will be along the lines of:
root/products/widgets/japanese.html
This is usually reflected in your navigation, with unrelated links (unrelated to your current page that is) hidden away. These are the principles of good design, so if Google are penalising for this they're just going to encourage flat websites. And who needs that?
There have been a number of suggestions that content too far from your homepage won't do well. However it doesn't really make sense that Google would drop the pages. So you'd expect them in a site: command, even if they rank poorly. The evidence suggests a bug or mistake they don't know how to fix.
What I fear is that this represents new policy, a misguided attempt to fix their "huge machine crisis"--as in maybe they face a hard limit on the size of their index, and their algorithmic, guiding principle is to rank pages, hence accept them into the index, according to link level.
As for: "users will appreciate it". There are limits to my altruism. A man's gotta eat.
Given Google's knowledge of the web it wouldn't make sense to just cull things because of their distance from a main page.
It's easy to jump to conclusions about Google screwing up, but it is hard with the drop in pages to imagine a good reason for it.
Opt out if you don't want to get our advertisingsays the spammer.
Include these meta tags if you don't want to get indexedsays the search engine.
So you do.
Wait! I am still getting your "advertising"!
Wait! I am still being indexed by your search engine!
This is why I am so against opt-out solutions. It NEVER works in the long run.
There is an opt-in solution which works. Google's Sitemap is a clear proof of that. So why not abandon automatic spidering, and rely on "Sitemap" solutions? The initial momentum is already there as far as volume, so no need to consume everything.
Wonder if they have some sort of virus....
Imagine google would compare websites with books!
But, in many cases a better analogy would be comparing with a newspaper. How many pages does a newspaper generate in a day? NYT? WSJ?
It is quite possible to generate thousands of unique, valuable pages of information in a short time, depending on the number of writers one has available.
WBF
There is an opt-in solution which works. Google's Sitemap is a clear proof of that.
I have used sitemaps for the last six months. My sitemaps are complete and impeccably clean. Google reports no problems with my sitemaps.
Yet Google has dropped 1/3+ of my pages (a week ago it was 1/2) from their index.
Is this "clear proof" that the sitemap solution "works"?
I thought that the main purpose behind sitemaps was to help ensure that all of my pages got crawled and indexed. If not, what's the point of maintaining sitemaps?
Whatever happened to Brett Tabke's oft-repeated dictum: "Build one page of quality content per day. Google loves content, lots of quality content. Broad based over a wide range of keywords..."
Google is still indexing new pages.
That doesn't mean glitches can't occur, or that every new page will get crawled and indexed quickly.
In my own experience, pages that are linked from the home page or another top-level page will be indexed more quickly than pages without such highly-visible links. (In other words, page 1 of a new article may be indexed in 24 to 48 hours, but it will take longer for pages 2, 3, etc. to make their way into Google's index.)
If you rely on the Internet as a primary revenue source, I suppose there's not much choice but to continue building content. But, if you have other options, then I think it simply makes sense to pursue them. The Internet is still important to me, but only as a secondary thing to support non-Internet business endeavors.
Right now, adding new content is too likely to be a dry hole. It probably will change. Meanwhile, I'm "elsewhere".
If I can write an article and get paid a couple of hundreds bucks on delivery, guaranteed, why should I waste my time writing for "nothing", in the hope that eventually I'll get indexed and can monetize the work?
Or write a book that will pay me for years.
If I can write an article and get paid a couple of hundreds bucks on delivery, guaranteed, why should I waste my time writing for "nothing", in the hope that eventually I'll get indexed and can monetize the work?Or write a book that will pay me for years.
I suppose if you go after "other" business endeavors you still essentially have to promote yourself in some way online or off. Gotta find someone to write for. Gotta market your ebooks. Gotta market yourself and find the BEST publisher who can get your writing the best exposure...etc.
Pretty much you have the same situation. Gotta look at other traffic and promotional sources to make up for any Google losses. It sucks and yes it is a bit more work than just dishing out content and hit/missing the free search engine traffic ride. Content is king. If you serve up blank pages you will go nowhere fast. You HAVE TO HAVE something to promote whether it is one page or 20,000 pages. The quality is what makes you GREAT in the eyes of the visitor. Combine QUAITY with a strong ABILITY to market and promote the site despite any slips and downfalls is what will make your site a LONG STANDING POWERFULLY BRANDED entity.
If I can write an article and get paid a couple of hundreds bucks on delivery, guaranteed, why should I waste my time writing for "nothing", in the hope that eventually I'll get indexed and can monetize the work?
To each his own. I've found that it's a lot more profitable to be an author-publisher and to make money whether I'm working, traveling, or even going nuts with boredom in a hospital bed. Also, it's a lot less stressful to take a long-term view instead of obsessing about how quickly Google indexes my new pages. Instant gratification is nice, but it isn't (or shouldn't be) essential.
newspapers real value lasts only a day
Only true of some newspaper content. But, accepting your argument, how about magazines? Is Scientific American only valuable for a few days or month?
The point is, it is possible to generate large amounts of valuable, unique content in a short time - depending on one's resources.
WBF
The PR thing may be deceptive, if G is now discounting links for various reasons they could still show up in PR calculations, so while you may show a 5 it's effectively a 3/4.
Instead of playing the guessing game on this issue, have any of you been studying the current #1 ranking sites under your preferred keywords? If not, you should be. If you have, what are you finding?
Are you finding sites with:
1. More content than your website has?
2. Content that is linked to from the home page?
3. Content that is several levels deep within the site?
4. More backlinks than your website has?
These are all factors to consider. I am finding a trend with a lot of websites that seem to be ranking higher that have a massive number of pages of content linked to directly from the home page. I am also finding websites that rank high with a mix of all of the above.
Figuring out what Google is doing is really nothing more than figuring out what they like about the sites that they are ranking high. Doing the research on this would save all of us from a lot of guessing.
I believe that content is still "king", and backlinks are still "queen". However, I also believe that how the hierarchy of your website kingdom is structured has a lot to do with whether or not it will reign supreme, or end up cast into obscurity at the bottom of the pack.
Google is still indexing new pages.
Indeed. I put up four new pages several weeks ago, and they were crawled and indexed within the week. I was pleased that two of the pages ranked mid-teens in the SERPs.
Then, several days later, Google dropped them from the index. They have entirely vanished from Google's SERPs.
As have 100+ pages of old, established content that Google saw fit to index for months and years but has now decided to drop. These are quality pages on a par with the several hundred other pages that Google has decided to keep in its index--for now, at least.
So I am left to wonder: If I create any new pages, will they be indexed? Indexed then dropped a few days later? Indexed, then dropped months or years later?
That doesn't mean glitches can't occur, or that every new page will get crawled and indexed quickly.
It's not just new stuff that's effectively banished from the index.
I can only hope that this dropping of pages--old and new--is just a "glitch" that Google will solve sooner rather than later.
Then, several days later, Google dropped them from the index. They have entirely vanished from Google's SERPs.
I've had pages disappear from the index for months and then reappear. (Right now, some of those pages are #1 for competitive keyphrases, but they were MIA for quite a while. I have no idea why, since they're similar to other pages that have always done well and never went missing.)
I was hit with a duplicate content penalty for this site a year ago. I dealt with it, and was restored to the SERPs several weeks later. I have no reason to believe that any of these recently dropped pages are affected by any duplicate content penalty.
In my own experience, pages that are linked from the home page or another top-level page will be indexed more quickly than pages without such highly-visible links.
What if the link is in multiple folders? For example on my blog, Index will lead to /post_id/postname/ (that's my permalink setup)
Is that bad?
So far my site's been up a week so i'm not *too* worried. My index has finally been listed, cache shows the update as of around Sunday, but still only 1 page listed. Should I be patient, or change the permalink to something different like /postname-post_id/
Thanks!
Note: I do have a second-level-deep site_index.html page that links to every one of the site's pages. (This is not a recent innovation; this page has been there since the beginning.)
Also, I had a third-level page that linked to ~80 fourth-level pages. I added a link to that one page on every site page (adding it to the footer menu), including the home page, making that page effectively second-level, and those ~80 pages newly third-level.
So, I can't say that any of this disproves the notion that Google now only crawls/indexes so many levels deep based on PR. Maybe those pages are back because of my personal sitemap, also because I tweaked the links to that one important page.
Or maybe it was all just a temporary glitch...
For example, my site where new pages added to the homepage and toplevel pages every day - google used top crawl and rank them in 48 hours.
Suddenly, google stops indexing them - still crawling, and not in the index. Linking from anywhere doesn't make any difference.
So it's not linking alone or PR either.
A few pages which are linked from top level pages vanish suddenly from index. Why? No clue. Then, 10 % of them come back after 3 weeks. Why? No clue.
I would say, my naive logic says that if there is a new article on a site that is updated several times a day, a search engine should crawl it almost instantly. If a detailed authoritative article appears 5 levels deep, it too should be crawled and indexed and ranked.
But obviously google and others have other issues to tackle, and a little collateral damage is to be expected.
I am sure Google will come back and start crawling my new pages and indexing them in 48 hours again. But worrying about when thats gonna happen wont help.
And for some of us, it could be as simple a matter as 'a man's gotta eat'. His food is the collateral damage.
Of course there is no way to know, but I think indexing of new pages and their appearance in the serps may have to do with trust rank and perhaps PR as well.
I have seen it happen since the big daddy update. New pages were added to my own site and were indexed immediately. So far, they are ranking and staying in the index. New pages added to a friends site (same day as my new pages) did exactly what you described. They were indexed, disappeared and reappeared a couple of weeks later. His site has PR 3 mine (I think) is PR 5. Don't know 'cause I use Mac so don't have a toolbar.
Anyway, if Google indexes a new page and isn't sure if that page is of sufficient quality to remain in the index ... it is entirely possible that trust rank has a role in this odd behaviour of appearing, disappearing and reappearing after being indexed and put through the trust-o-meter.
Just a stab in the dark.
I would think that trust rank applies to entire sites, not to individual pages.
