| This 249 message thread spans 9 pages: < < 249 ( 1 2 3 4 5 6 7  9 ) > > || |
|Pages Dropping Out of Big Daddy Index|
Continued from: [webmasterworld.com...]
internetheaven, you said:
|I had 20,300 pages showing for a site:www.example.com search yesterday and for the past month. Today it dropped to 509 but my traffic is still pretty constant. I normally get around 4,500 - 5,000 to that site per day and today I've already got 4,000. |
So, either Google doesn't account for even a small percentage of my traffic (which I doubt) or the way Google stores information about my site has changed. i.e. the 20,300 pages are still there, Google will only tell me about 509 of them. As far as I can tell, I think the other pages have been supplemented.
That resonated with something that I was talking about with the crawl/index team. internetheaven, was that post about the site in your profile, or a different site? Your post aligns exactly with one thing I've seen in a couple ways. It would align even more if you were talking about a different site than the one in your profile. :) If you were talking about a different site, would mind sending the site name to bostonpubcon2006 [at] gmail.com with the subject line of "crawlpages" and the name of your site, plus the handle "internetheaven"? I'd like to check the theory.
Just to give folks an update, we've been going through the feedback and noticed one thing. We've been refreshing some (but not all) of the supplemental results. One part of the supplemental indexing system didn't return any results for [site:domain.com] (that is, a site: search with no additional terms). So that would match with fewer results being reported for site: queries but traffic not changing much. The pages are available for queries matching the supplemental results, but just adding a term or stopword to site: wouldn't automatically access those supplemental results.
I'm checking with the crawl/index folks if this might factor into what people are seeing, and I should hear back later today or tomorrow. In the mean time, interested folks might want to check if their search traffic has gone up/down by a major amount, and see if there are fewer/more supplemental results for a site: search for their domain. Since folks outside Google couldn't force the supplemental results to return site: results, it needed a crawl/index person to notice that fact based on the feedback that we've gotten.
Anyone that wants to send more info along those lines to bostonpubcon2006 [at] gmail.com with the subject line "crawlpages" is welcome to. So you might send something like "I originally wrote about domain.com. I looked at my logs and haven't seen a major decrease in traffic; my traffic is about the same. I used to have about X% supplemental results, and now I hardly see any supplemental results with a site:domain.com query."
I've still got someone reading the bostonpubcon email alias, and I've worked with the Sitemaps team to exclude that as a factor. The crawl/index folks are reading portions of the feedback too; if there's more that I notice, I'll stop by to let you know.
[edited by: Brett_Tabke at 8:07 pm (utc) on May 8, 2006]
>So my theory is correct. Google is mostly giving problems for new and major redesigned site
I also disagree I did a major rebuild for a couple site of sites late last year and as we assuming BD started life around November time both sites have gone through this fine, in fact one site is sat at number one for a very competitive keyword across both .com & UK so rebuilding a site will not trigger this, don't ask me what does though as I have no idea other than the one the bulk think here - G is broke
>>Another thought is that if Google is having problems with indexing pages of a site - wouldn't it affect the SERP's - number of pages of a site and lost backlinks thereby causing a SERP's change?
maybe hence the reason the emails from G are saying use the sm
I don't think you can infer much with the data available other than that there is a problem.
i have noticed enough consistent instances to give credit to your theory...it specifically involves major changes that were ongoing during the concept of BD..i.e from October 05 through March 06
More comments on this theory would be very helpful..
Please share your experiences and let's get a vote on this..
A vote? You're kidding right? This is like voting on reality and deciding that the winning vote reflects it.
I have followed this thread for a while as our site went from 30000 pages listed on google late March to less than 100 today. As with most, our traffic from them went the same way. As all here, we have been trying to find out why.
We posted our info and questions on mattcutts.com/blog early on and they were ignored. We emailed to the address specified by googleguy and never heard back. We contacted seo experts quoted on this and cuttsís sites and they did not know.
How then are we to know? Or is it that it does not matter to google? Certainly larger companies including foreign car manufacturers are told when they have a problem on their sites and promptly re-included. Why would this access be available only to the few? Why would small businesses that need it most not have the same access?
Everyone always complains about Microsoft but, with them, the worst case if you have a problem with one of their software is to you pay $250 for a support incident. In this case, they will stay with you until there is a resolution to your software trouble. Where is that level of service with google?
Instead, we get ignored or provided with evasive replies. Worst, many now feel obligated to jump and thank profusely google representatives for every evasive non-answer they provide us because they took time off their busy schedules to address our measly concerns. Never mind that some businesses already closed because of this lack of communications and that others are in danger of following the same path because of it.
That they don't charge us for those listings is really no reason for this carelessness. google's business model is about not charging for services so that they can charge others for data and advertising. They used our information to make money and gain size when they wanted to and now command a 50% market share of the search business. We are not asking them to share the tricks that would allow our sites to be first. We just want to know why they remove sites so that we could fix that. We are asking for the same transparency they already afforded larger companies. What will it take to get that?
Yikes, I just noticed today another few pages dropped from google. I was noticing drops from certain pages i've altered, assuming that was the case, but the pages I noticed dropped today were pages that I created in the past 3 weeks, they were listed for a few weeks with 0 changes to them, and now they're gone out of the blue. At least the other recent drops sort of had a reason, which was changes made to them.
wow... I have fallen victim to this..
I just redid my website about a month ago..
Because I do graphic design for a living and my site was geared towards that.... but now I also decided to start offering my photography services to people... so I redid my site design completely not knowing about any of Google's issues.
At first all of my new pages got indexed right away as I submitted to Google Sitemap... and I was so happy all 70-80 pages were indexed.
This morning... only 20 pages are indexed... and really it is all from the old site design in cached history.
What is the finally prognosis with all of this?
Will my new site be reindex or am i SOOL?
thanks for any help!
I still don't think this is down to rebuilding a site
right done my bit of stressing out over G today a bottle of red wine is calling so I'm off
|So my theory is correct. Google is mostly giving problems for new and major redesigned site |
This is not the case for us. We have two, older established sites that we have not made any changes to, that are suffering this dramatic loss of pages. We have lost around 10,000 pages on one and around 5,000 pages on the other. The pages lost are primarily from discussion boards. Our positions for our main key words are holding well, but we used to get a lot of traffic from more obscure searches that came through indexed discussion board pages.
Strangely we have another site, in a different sector, with a discussion board that seems to show an inflated page count now, but perhaps the pages were there but never properly indexed previously.
My experiences with site redesign.
The site I talk here of now looks NOTHING like it did six months ago. EVERY page has changed though urls's havent changed at all.
The page structure, layout, meta tags are ALL changed. I started these changes in November and finished around a month ago.
The main focus of my site has also changed. Pre Sept 2005 it was very focused and geared towards mobile / cell phone extras, now there is zero mention of this.
When this dropped pages issue began I looked towards duplicate issues, my pages when looking at the code are VERY VERY simillar, though to the ordinary surfer they aren't. I have changed common elements to make them as unique as I can (still working on it), trying variants to diferent sections to see whats working and whats not.
The more unique I make each one the better it ranks (with other optimization also a factor)
If you read my earleir post and had a quick look at the sites mentioned there you will have seen these recurring common elements in a high % of those sites.
Again my site isnt perfect, very high ranking pages have pagerank 0 whereby 3 months back they had at least a bit of green. Not all the 1300 pages are ranking high, though that is down to me tinkering and trying new stuff out.
Google certainly has problems, of some sort but I'm not convinced its all one sided.
Just as a light hearted footnote: when this particular site was aiming at mobile phone stuff it ranked ok. just yesterday i noticed it was #2 for a pretty competative phrase, these is absolutely no mention of this on the entire site never mind my homepage which is the ranking page. Google has a memory like an elephant lol, Im guessing it will be in relation to inbound links from sites of that theme but strange none the less.
There are simply too many options available to be able to comment with authority i would suggest.
I have seen my page count go from 700,000 to 900 and now back up to 14,000 which is probably more accurate than the 700,000 ever was, but at the same time my traffic has stayed fairly stable give or take 500 uniques a day.
The site had a redesign in nov / dec last year but kept exisiting page names and this hasn't effected any positioning, infact over the last few months the site has become more stable in it's rankings and ranked well for anything i put up.
I, like most people, have noticed a lot of pages simply dissapearing that are unique in content and structure, and that previously ranked in top spots, these pages have mainly been added over the last 3 - 4 months but have kept their pr.
Today, the site still has it's pages indexed, and still ranks in the top spot for my guide term on .co.uk uk only search, but check the .com or .co.uk all web search and it's dissapeared along with hundreds of my top terms.
I've done nothing different to it than usual and the serps in my sector look absolutely rubbish, being full of spam, cloaked pages, redirects and anything else you would consider black hat.
The only thing i can think is that G is simply broken, the question now is how long will it take for them to fix it and can they do it before i rip out all my hair and put a hammer through my pc!
>>The main focus of my site has also changed. Pre Sept 2005 it was very focused and geared towards mobile / cell phone extras, now there is zero mention of this.
if that's the case, then i'm not surprised your rank has dropped loads. because google rates all the links that point TO your site by seeing if they are relevant to the page content. if you've changed to a new subject then all of your old inbound links are going to be about completely different subjects.
so you are effectively starting from scratch again - you've got no decent inbound links.
|When this dropped pages issue began I looked towards duplicate issues, my pages when looking at the code are VERY VERY simillar, though to the ordinary surfer they aren't. I have changed common elements to make them as unique as I can (still working on it), trying variants to diferent sections to see whats working and whats not. |
I'd like to add that the two sites i've been monitoring, have a similar thing. Each and every page that's been dropping have been part of a template. That said, the only thing that looks like a template is the layout. To the ordinary surfer, a lot is different. The title, desc, <h1> text, the header logo, every alt tag, image names, and every page has a product review which I write 4-5 unique paragraphs per page.
The only thing that is in template form is the actual layout, and if i'm punished for trying to be smart about building a site and putting all repetitive html code on one simple control, that's ridiculous on googles part. It's not like I made a site where I make 2-3 changes to variables per page and it dynamically builds a completely unique, but similar template looking site.
so how can someone like me with a fairly unknown site targetted specifically at attracting people to my site who are interested in photography services get reindexed on Google?
Google was 50% of al my of my traffic based on search engine results, and now I don't get any google visitors to my site.
You might find Matt's today post of relevance to this thread:
May 16, 2006 @ 12:24 pm
|f that's the case, then i'm not surprised your rank has dropped loads. because google rates all the links that point TO your site by seeing if they are relevant to the page content. if you've changed to a new subject then all of your old inbound links are going to be about completely different subjects. |
so you are effectively starting from scratch again - you've got no decent inbound links.
sorry, let me add that my site lost ALL its rankings last sept. after much thought i decided that the mobile / cell phone related stuff had to go. the subject of each page didnt change, it was just they were focused at the phone market.
I redesigned the site to freshen it up, make it more user friendly and rid my on site association with anything to do with mobile phones. I still never had any rankings. My rankings returned when 1000+ pages disappeared. My remaining 200/300 regained there old positions.
Until recently a lot of the page elements were of the following type:
red widgets from my wonderful site
green widgets from my wonderful site
pink widgets from my wonderful site
I made an effort to rid many of these similarities. Be it in the title, description, keywords, h tags, body text.
Ive tried many many combinations to see what works best. All I know is that since Ive started changing these more of my pages started to return, now they all have been reindexed but some are ranking higher than others depending on what combinations Ive been messing with.
I just finished reading the blog, and what scares me is the outbound links. How do you know if google feels they're relevant enough to link to? I'm not linking to any ringtone site from a real estate site, but it's possible some of my outbound links may go to questionable sites.
I recently saw somewhere a tool to do a 'banned' check on google, should I download that and make sure every site I link to isn't banned? If so, can anyone recommend a more legit tool for that? I was unsure of the site I went to, so wasn't about to just install some random .exe file. I'd like a trusted tool, or even a search command to check for banned domains. Thanks!
|You might find Matt's today post of relevance to this thread: |
Wow, Iíll say; thatís a very comprehensive rundown on whatís happening. Certainly some very interesting comments he made about reciprocal links, link relevancy and affiliate sites.
You gotta give Google credit for communicating so openly, (wether you like the message or not). Certainly more information than you would ever get from other SEís.
Now off to figure out what it all means.
ooh this is a good idea... if you get such a tool, please share it with us!
Matt's post is interesting, both because it covers good territory, but also because it shows that folks at Google really don't have a good appreciation of the actual problems.
Matt thinks the crawl is better, while this bot is far weaker than the old one.
I sent an example of a site that has 300 inbound links to the main page, and 50+ to internal pages. Most inbound links are PR4 or higher, several PR5, a couple PR6. The links out are mostly to government sites. It has no affiliate links and makes not one nickel. Yet it dropped 2/3 of its 70 pages in the big drop last month. It now is fully indexed again, when I threw another PR6 link at it. The point is, the weakling bot is the culprit, not anything else. That's not to say what he mentioned doesn't cover MOST things, its just to point out that (as usual) something they think is working pretty good is actually not working nearly as well as they think.
The worst thing though, by far, is apparently they have done nothing about supplementals and the rotting effect they have on domains, other than removing the oldest half. To be blunt, what a dumb idea that is. Apparently a lot of us will have to wait another year and a half for Google to obey 301s that have been in place for eleven months.
"You gotta give Google credit for communicating so openly, (wether you like the message or not). Certainly more information than you would ever get from other SEís."
Sure..credit to Google for communicating openly. Having said that, I wish to give Forum30, The Mother of All Forums, credit too for motivating Google to communicate openly, IMO ;-)
Originally posted in wrong thread
I'm not sure how to respond Google employee 1's post.
Big Daddy is out and fully loaded.
Everything is fine and if your site is missing pages then it's because you have some type of penalty ie. too many recip links, no authority, poor linking out, etc.
And it seems Google is happy with their new index.
After doing some searches for personal matters like Joe Q. Searcher, I immediately switched to Yahoo and MSN.(where I found what I was looking for)
I suppose my searching skills are "too advanced" vs. Joe Q. Searcher for objective opinions. 4-5 word search stings, etc.
Joe Q. simply doesn't search for "widget reviews of unique article"?
They simple type in "widget" or "widget review" or "widget article"?
If that's the case, once again, why is Google considered the "best"?
Any basic SE can pull up the 10 most relevant sites for broad terms like that.
Google doesn't (and never) held any superiority for that.
Google USED to hold superiority when it came to "odd phrase in unique town" or "unique product for unique problem".
Seems that's no longer the case...
Fair enough, enjoy your new "algo" Google.
Rest on your laurels and Phds.
I don't support nor use inferior products simply because they have the biggest advertising budgets.
Unfortunately, most people do.
At least, that's what they are betting on....
|it's possible some of my outbound links may go to questionable sites |
Drop them if you arent 100% certain of them. Why jepordise your site for something you arent so sure of or have doubts about. Unless they bring you a ton of traffic which then means you need to choose between traffic from the links and building a site to please only Google.
Anyone know a good lawyer? Class action anyone :) ...?
"Anyone know a good lawyer? Class action anyone :) ...? "
Trust me. A good SEO will serve you better than a good lawyer :-)
everyone seems to have comments...
does anyone have answers or suggestions how to get a site back to being indexed?
"does anyone have answers or suggestions how to get a site back to being indexed? "
Clean your site and submit a reinclution request.
I did it and today my site is back after 17 months of heavy penalty.
In case you need more info on how to file a reinclusion request, Matt wrote very informative post here:
Based on Matt Cutts post..
We have a site that is half affiliate and half our own products with our own shopping cart...
Are we being penalized for having affiliate links?
If that's the case what should we do get rid of half of our income due to Google?
Just to stir this pot one last time I never filed a reinclusion request.
My pages reappeared of their own free will after making the changes I spoke of before, they are all two clicks from home page plus I use google sitemaps. Most of the pages that went missing have IBLs to.
So a reinclusion request isnt always necessary.
| This 249 message thread spans 9 pages: < < 249 ( 1 2 3 4 5 6 7  9 ) > > |