Google have De-Indexed this site

[DavidJudge]

I'm looking at a site which was removed from their index over a period of time starting on 8th April through to 6th May (this was when the last page was removed).

I have just gained access to Google webmaster tools and it unfortunately isn't really shedding any light on the subject.

They do have a bit of long winded robots.txt, but can't quite see anything wrong with it either.

They did receive an email from Google when they were running an Adwords campaign stating Google had removed the site from their index (see below).

The email was sent on 29th March (just before they showed as being de-indexed in webmaster tools).

At the time the issue occurred they were running an ad for a genuine give away which didn't sound like anything that would violate Google's policies.

I've had a good look through the site and I can't see anything that stands out as to why.

I have checked the basics:

- Webmaster Tools
- Backlinks
- Site Structure (including any outbound links,
- Robots.txt
- Sitemap.xml (just in case there was anything odd)
- Duplicate Content
- Broken Links

I can see a number of small issues with the site, but nothing is really jumping out for me.

I'm not sure if I'm ok to post the URL here, but any suggestions would be appreciated.

******

Thank you for your email. I hope you're doing well.

I reviewed your AdWords account 'xxx-xxx-xxxx' and noticed that you have created a campaign with ad and keywords directing to the website 'xxx.com' and since the website is violation our webmaster guidelines policy it has been disabled to run through AdWords.

And this policy is something which is not an advertising policy but a Google search related one.

When a site gets disabled in Google Search it would be disabled for Webmaster Guidelines policy. And unless it is re-enabled in Google Search your website cannot be allowed to advertise with AdWords.

A violation of our Webmaster Guidelines may lead to a site being removed entirely from the Google index or otherwise penalized. When a site is penalized, it may no longer show up in results on Google or on any of Google's partner sites.

******

[bhartzer]

Most likely it's not the technical aspects of the web site--it's the topic of the site.

[aakk9999]

You can try to send a reconsideration request and see if the answer will give you any clues.

[Robert Charlton]

Hi DavidJudge, and welcome to WebmasterWorld.

I have checked the basics

You describe the technical basics, but it sounds like you should be checking the Google Webmaster Guidelines, and, in particular, the Quality Guidelines section. See...

Google Webmaster Guidelines
[support.google.com...]

I'm not sure if I'm ok to post the URL here

Thanks for asking. No, it's not OK to post the url here. As the Google Forum Charter [webmasterworld.com] explains, for your own protection, and for the protection of others here, we don't allow site reviews in our public areas. What this means is no links, urls, keywords, or suggestions of any kind about how to find a site. Give the Charter a read... you'll find it helpful.

Also, take a look at the Hot Topics [webmasterworld.com] section, pinned to the top of the Google SEO forum home page, and look at the Google Update History [webmasterworld.com] section up near the top. It contains a monthly reference to updates, and you might be able to match up some dates with your AdWords email date.

Offhand, by the dates, it looks like you were hit by Panda 2.0... but I don't believe that should have taken you completely out of the index.

[DavidJudge]

Hey, thanks for your suggestions and for the welcome.

I've read through the suggested links and appreciate the heads up (I've book marked the Google Update History).

I'm not sure if the site has been re-submitted for consideration and wanted to make sure it was all good (from what I can control) before I did so.

I've gone through each of the points on guidelines and did not see any issues there, so perhaps the next step is to re-submit?

The site is a WordPress CMS (which obviously has it's weak points) but I've never seen this happen before.

When you log into webmaster tools and check under 'health' - 'index status' you see the number of indexed pages gradually go from 178 to 0 in just under a month. It also hasn't changed since then (no more or less pages indexed).

I thought it may have been a Panda update but there wasn't a mentioned update around this time frame as you mentioned Robert and would have expected a reduction in the pages indexed, but not complete removal.

The site does show up in the other SERP's so it looks like it's just Google that are unhappy with it.

The content is not adult based or sensitive in any way and the site was successfully indexed and online since early 2008 to earlier this year. The topic of the site site is web design, logo design and print services.

I even checked other sites on the same IP (more out of interest) and there are about 40 other sites on the same IP and a few I checked were all indexed.

Any other thoughts would be appreciated.

[Robert Charlton]

Panda 2.0

My bad. I was looking at the chart for roughly corresponding dates, and was looking at 2011. Again, Panda shouldn't have taken you out completely.

The site does show up in the other SERP's so it looks like it's just Google that are unhappy with it....

If Google's the only problem, then it's most probably something like backlinks or perceived spam... perhaps not in sync with the announced major updates, but one of the many algo changes that happen every day.

The topic of the site site is web design, logo design and print services.

Normally, we like to try to avoid site topics too, but let's let this one go. One possible idiosyncrasy of a web design site would be that many of them depend on global footer links for ranking. It's likely at some point that Google has discounted these. I can't imagine that the site you're dealing with would have been so dependent on them that it would have been removed from the index, unless they appear to be intentionally manipulative.

Just fishing for ideas... did the site itself design WordPress themes?

[bhartzer]

Without knowing the exact URL it is going to be difficult to give you any more help. However, you might look and see If the content is all original content, you might run it through copyscape to make sure it is not a duplicate of another site.

[DavidJudge]

Normally, we like to try to avoid site topics too, but let's let this one go.

Oops, my bad. So we don't go too deeply into it, I have seen this same template used elsewhere and it is current indexed with PR. I checked the footer links (using a backlink tool) and they've actually used their business name on each backlink.

When I look at the link building they're done to the site, nothings really standing out. Sure there are a couple of questionable links, but only a very small handful.

you might run it through copyscape to make sure it is not a duplicate of another site.

Yeah, did that too and the content shows up as unique.

I must admit, this one has me stumped which was the reason for the post here.

The only thing I can think it the robots.txt file.

It has a lot of disallows from third party spiders and also has -

User-agent: Adsbot-Google
Disallow:

User-agent: Googlebot
Disallow:

I believe this is only an issue when you have the trailing slash.

I must admit, I had never come across the letter I posted in my first post before.

This was from the Google Ad Team.

[TheMadScientist]

WOW, this one's fascinating ... Glad you let the topic through, because it doesn't make a ton of sense since it's 'completely generic' and not even close to what I'd normally expect the topic of a deindexed site to be if it's topic related.

If that's the robots.txt I'm not seeing anything wrong with it.
You aren't running any noindex or nofollow headers or metas are you?

And, welcome to WebmasterWorld! Sorry about the reason behind your joining us here ... Hopefully we'll find an answer for you somewhere and you'll hang around for cooler reasons.

###

Actually, noindex or nofollow makes no sense and it really can't be the robots.txt based on the e-mail ... Sorry ... It's in violation of the guidelines somewhere, not 'self removed' so robots.txt shouldn't matter, and it looks fine, because GBot will follow the most specific line and it allows access.

I read about the violation earlier when I first looked at the thread and figured it was one of those 'no brainer, quit selling v*gra and you'll get back in' answers (lol), but I forgot that when I initially posted, so totally disregard my question, but not the welcome and I'll come back to this later if I have a more constructive question or something.

###

And, it's later ... Are you selling ads at all?

[simonmc]

Is it possible that somebody entered a DMCA against the site because the content had been plagiarized? I know this happens a lot in web design and logo design.

[scooterdude]

Consider removing.

User-agent: Googlebot
Disallow:

Google rarely interprets anything the way you might expect, its not impossible that they view that as an unintentional incomplete robot instruction

[DavidJudge]

when I first looked at the thread and figured it was one of those 'no brainer, quit selling v*gra and you'll get back in'

lol, I really wouldn't touch that sort of site (not that there's anything wrong with that).

And, it's later ... Are you selling ads at all?

There are no ads on the site at all.

There is a twitter and a facebook feed.

There are the usual 'Social Links' to their other online properties and on the footer a few outbound links to sites that offer real value to the visitors of the site. These are 'follow' hypertext links and are legit trusted sites.

Is it possible that somebody entered a DMCA against the site because the content had been plagiarized?

This could be a possibility I guess, but there are only a few images on the site and they're pretty much stock photos I've seen on other sites.

I did paste a bunch of pages through my copyscape account and there were no references to external sites.

Consider removing.


User-agent: Googlebot
Disallow:

I had a good look and Google only remove if it has the trailing slash. Also they removed the site for a violation so as TheMadScientist mentioned it probably wouldn't matter.

I also had a good look on a popular backlinking service and it showed 32 referring domains, so it's not a massive number of links to potentially get banned. Every one of those sites are legit sites, although most of them are footer links to reference for their service.

There was also no message for un-natural backlink patterns.

The more I am looking at the information I have available to me it doesn't look like the site has been submitted for reconsideration.

[DavidJudge]

Is it possible that somebody entered a DMCA against the site because the content had been plagiarized?

The site also have a DMCA policy page where they ask you to outline if you have any issues with any content onsite along with a contact email address.

[atlrus]

Is it possible that somebody entered a DMCA against the site because the content had been plagiarized? I know this happens a lot in web design and logo design.

This is the first thing that popped into my head when I read the OP's first post. It's the only time I've heard Google actually de-indexing a site (besides self inflicted de-indexing, of course).

You were clearly told that the site was disabled in Google Search, so nothing technical here - someone at Google just removed the website manually.

I have never had to deal with DMCA, but I am sure there must be a way to check on this.

[lucy24]

Obligatory question at this point: Are you absolutely positive the site owners and everyone else involved are being completely forthright with you? It sounds as if you never set eyes on the site until after things went kerblooey, and then they came running to you for help. Is that right?

I'm thinking something analogous to a person withholding key information from the doctor because it's embarrassing and they really didn't think it could be relevant. And I can't help but notice that there seems to be a six-month hole in the middle of the narrative. What happened between now and last May?

[DavidJudge]

Are you absolutely positive the site owners and everyone else involved are being completely forthright with you?

Yeah, a really good question. Am I sure that they're being honest... not 100% but I did ask the question pretty much the same way you mentioned at the very beginning.

I also researched everything at the beginning and aside from what appears to be a very high number of facebook likes, everything else looks ok.

It looks like they've continued on during the gap except have just not used their website to generate business. They're looks like they're not really experienced in SEO so it went into the too hard basket.

Is it possible that somebody entered a DMCA against the site because the content had been plagiarized?

I did a little more research on this and Google are supposed to notify you via webmaster tools if their is a DMCA violation.

Webmaster tools was only recently added to the site, but I have noticed when adding webmaster tools that notifications sent prior to the addition of webmaster tools are present (I noticed this with the unnatural link patterns email they sent out).

I also used an image cross referencing tool and I did find a few of the images on other sites, but not one single site and each of the other sites were having issues.

[aakk9999]

Does the site use canonical link element and if so, where does it point to?
Have you tried "Fetch as googlebot" and checked what response you are getting? If Googlebot fetches the page, try clicking on "Submit to index"

[simonmc]

Has the site been de-indexed from Bing and Yahoo?

[jimbeetle]

Thank you for your email. I hope you're doing well.

I reviewed your AdWords account 'xxx-xxx-xxxx' and noticed that you have created a campaign with ad and keywords directing to the website 'xxx.com' and since the website is violation our webmaster guidelines policy it has been disabled to run through AdWords.

And this policy is something which is not an advertising policy but a Google search related one.

When a site gets disabled in Google Search it would be disabled for Webmaster Guidelines policy. And unless it is re-enabled in Google Search your website cannot be allowed to advertise with AdWords.

A violation of our Webmaster Guidelines may lead to a site being removed entirely from the Google index or otherwise penalized. When a site is penalized, it may no longer show up in results on Google or on any of Google's partner sites.

Is this real? Is there now a connection between the AdWords team and the Search team?

Since when does a site get "disabled" in Google search? And "re-enabled," where the heck did this terminology come from?

Have I missed something over the past few months?

[TheMadScientist]

This is like a great big game of 'Go Fish' lol

Did you replace: "lorem ipsum dolor"
With: "lorem ipsum dolor ABC"
To get the content to show as unique in copyscape? LOL

Seriously though:

... it showed 32 referring domains ...

My best guess right now is it's a combination of things ... Low link count + No strong links to overcome any 'spamminess' or 'low quality' associated with some unknown number of the links + Some type of low quality signals onpage = We don't want (need?) to index you.

It might not be those exact issues, but I think it's likely a combination and the low number of inbound links and not having any really solid links (from what it sounds like anyway) and IMO those could definitely have some influence on the situation, so my first 'gut instinct' after reading the '32 domains' number is to say get some good solid inbound links* and see what happenes...

* I wouldn't even worry too much about the text, just something quality links to flow some PageRank, because the ones you described don't sound like they'll pass much, if any, weight. (You can worry about the text and what you rank for after you get back in.)

So, in reading, it sounds like you only have a sliver of PR (at best) which won't overcome any 'negative signals' and that together with the site being on a heavily covered topic could, IMO, tip the scale to 'we don't need to index this one, because we have plenty of info on this topic listed already'.

[bhartzer]

What's the history of the domain? Was there content on the domain that got it banned in the past, they put up new content on the domain and that also gets banned?

[diberry]

The email says it's a Webmaster Guidelines policy. Typically, violating these policies results in a penalty, not de-indexing. There are sites which have copied many of my pages, photos and text, and DCMAs get each individual page removed with a notice in the SERPs, but the rest of the site remains indexed.

It may not take much (from a webmaster's view) to get a site penalized in Google, but it takes a LOT to get de-indexed.

I'd be thinking legal issues. Is there any way Google could think (even mistakenly) the site is promoting hate against a group, or piracy? I know based on the topic that's really unlikely, but as you've noted, this one's a real mystery.

Are you able to see independent caches of the site from the time before you were made aware of its problems? Just to confirm it wasn't a hate/porn/warez site which suddenly changed topics before asking for help? Again, I'm not suggesting this is likely, but I DO think it has to be something major like this.

[HuskyPup]

I reviewed your AdWords account 'xxx-xxx-xxxx' and noticed that you have created a campaign with ad and keywords directing to the website 'xxx.com' and since the website is violation our webmaster guidelines policy it has been disabled to run through AdWords.

The first part of this sentence indicates to me "arbitrage" which would indicate why it has been disabled...that's a possible guesstimate I'm making.

And this policy is something which is not an advertising policy but a Google search related one.

Google, it is very bad grammar to start a sentence with a conjunction however again it may be suggesting "arbitrage". What was the famous BMW demotion penalty a few years back?

When a site gets disabled in Google Search it would be disabled for Webmaster Guidelines policy. And unless it is re-enabled in Google Search your website cannot be allowed to advertise with AdWords.

They're trying to give you a hint here as to what they feel is wrong, it is a Webmaster Guidelines policy, there is something wrong with your site, seemingly there is not an advertising issue, presumably not arbitrage!

Ok I've stated the obvious therefore you need someone with fresh eyes to look at the site to see if there is anything obvious you may have missed ... it's always the blindingly simple things which are usually the culprit yet rarely can one see it oneself.

[TheMadScientist]

It may not take much (from a webmaster's view) to get a site penalized in Google, but it takes a LOT to get de-indexed.

It's actually not as much as it seems like sometimes ... I've seen some get removed when they're not doing any 'wrong', per se, but rather just 'don't have enough' to be in the index.

There was no notice for those though, but they didn't use AdWords/AdSense either. It's actually cool this one did and the notice was posted, because if it didn't all that would be known is it was deindexed.

I can't go into too much detail, but I've seen sites I know aren't 'messing up' be either not indexed or removed after a brief period of indexing and the only thing I (and others who have been doing this a while) can come up with is there's just not enough 'uniqueness' or 'difference' there for Google to warrant putting them in the index and a low number of inbound links does nothing to say, "Hey Google, people like this, it should be included...'

As a side note: I think we'll begin seeing more 'fickleness' WRT indexing from Google moving forward since they're committed to not being a search engine.

[edited by: TheMadScientist at 5:54 pm (utc) on Nov 12, 2012]

[Shaddows]

At this stage, I would go ahead and submit the reconsideration request.

You should include the fact YOU are now in charge of the website, that the previous webmaster is out of the picture, and that you have conducted a full review of the site with respect to the Webmaster Guidelines.

Go into specifics of what you have done to check compliance.

If the site really is clean, then you should be released immediately. Otherwise, as someone has already suggested, you may get more specific feedback.

Again, as others say, it is generally a legalistic issue that gets you "banned" - manipulation just gets you nasty penalties. At least, manipulation worthy of a ban should be glaringly obvious.

[Shaddows]

@TMS (Hi BTW)

Yeah, but too small for inclusion isn't the same a postively excluded, which is what the email implies.

[Sgt_Kickaxe]

The removal period was too gradual to be an actual ban, in my opinion. When they want your site out it goes all at once fairly quickly. To me this suggests a filter of some sort.

- remove the robots.txt file completely, for now.
- ask the site owner for an honest answer about their use of paid link services, you can't repair that easily or at all if they don't fess up with specifics. A definitive "no, we didn't use a paid link service" is just as good, you need somewhere to start.
- remove all links to the site Google states was against their web guidelines and ask for reconsideration.

[TheMadScientist]

Hey Shaddows ... Good to see you!

True, and that's good point ... I guess I'm 'grasping at straws', because what everyone here is saying about why sites normally get dropped is accurate and makes complete sense to me, so it seems like this one almost has to be something 'other than usual'.

David doesn't seem to me like one of those posters we get sometimes who's trying to work a spam site around a penalty, so if I throw out 'spammer' and assume we're getting the whole story about what's been looked at (it's possible there's something he missed - I know I certainly have had a few 'head slap moments' over the years - but assuming nothing was missed in this case), the only thing I come up with is 'not enough', or ...

It Could be what Robert said about an algo update of some type at the same time as the 'free giveaway' was running, where the combo of the 'giveaway' (maybe the wording or above the fold space taken by the giveaway info - something about it) and very low (possibly zero) 'quality link count' plus 'already have tons of sites on the topic' plus 'ROS inbound footer links' triggered a 'guideline violation', which could easily mean there's a 'time period' of penalization/deindexing, and if that's it, the issue may already be corrected but the 'penalty time period' not yet expired, so we could really be chasing our tails here.

BTW: I think the reinclusion request you suggested is probably a 'must do' at this point, because it should narrow it down to 'manual' or 'algorithmic' at the very least.

[bluntforce]

At the time the issue occurred they were running an ad for a genuine give away which didn't sound like anything that would violate Google's policies.

If the site is apparently clean now, but was de-indexed at a given point in time when something was different, I'd look at what was different.

[Robert Charlton]

I have seen this same template used elsewhere and it is current indexed with PR. I checked the footer links (using a backlink tool) and they've actually used their business name on each backlink.

Assuming that by "same template" here you're referring to my question about whether they've designed WordPress themes, this sets off some alarm bells.

At the time the issue occurred they were running an ad for a genuine give away which didn't sound like anything that would violate Google's policies.

What were they giving away? Was it by any chance the theme with their footer links?

Take a look at this discussion....

Google wants me to remove links from sponsored WordPress themes
http://www.webmasterworld.com/google/4445558.htm [webmasterworld.com]

Conceivably, there's enough similarity to sponsored themes that this site been similarly penalized. What's not completely clear is why the indexing withdrawal was gradual, why it appears that no notice was sent, etc. The timing isn't exactly what I'd expect from a Penguinized site, but, as noted, it does correspond to the giveaway ads.

One more possible strand... though I haven't followed it closely, I also remember hearing that the WordPress community was hit with backlink penalties. Has there been any participation by this site in the community that was hit?