|Does Google Ban or Filter Web Directories?|
I think the subject worth a thread itself. It's a suspision so far. Yet I don't see dmoz, yahoo nor any major web directory were banned/filter nor PRed zero as my web directory did. I tried to check it in Alexa (powered by google) and I see some results from my site. Appearently, Alexa brings old results from Google but something weird is that Alexa itself has PR0 now. But that's another story!
If you run a web directory, feel free to post your experience here.
Well it hasn't happened to me, partly because I don't have a directory site I suppose, and I doubt if I have a solution, but I was thinking.
If I was an evil virtual dictator on a power trip, and I wanted to keep my subjects guessing, I'd not implement one change, I'd implement two at the same time.
When a site owner is allowed to fix up their listing in a directory, they must be quite likely to put the exact same text in any directory they list under. Could trigger a dupe penalty for a phrase or two if I'm reading the Google patent right. Perhaps too many dupe phrase penalties on one site nukes it?
It might be that anything with the word 'directory' in a meta tag is more likely to be on top of the serps when a webmaster looking for directories to submit to is, well, looking.
Hope that helps.
They used to run search tests on their new search engine from their dorm rooms at Stanford back in 1998. That's how I first learned about Google, I kept seeing them in my former employer's visitor log ... actually I saw them even before back in 1997, but it was some weirder name than Google... (we were involved in some "bleeding-edge" technology - remember the battle of marketing superlatives in the Tech sectors back then... all the more to suck up your investment dollars)>
But Google was that name that caught my attention... as in "great, just what we need another search engine."
However, after seeing them a few weeks I finally gave their beta search engine a try myself, and "WOW!" This Google is GOOD!
Now after all their years of growth and most always featuring my "stuff" near the tops of appropriate search results, they completely delete my information from their data base.
"Where's the love & idealism we once shared in our youths, Google?" I bet if Larry or Segey discovered what was happening to me (moreover my information), some ears would be ringing at the Plex! Too bad they don't get in the trenches anymore... that's why Google has become just another soulless corporation IMHO.
My site is google indexed at 150,000 pages. PR 5. It has a small recip link directory (< 100 links). It has several thousand (~4K) pages with nearly identical content, and google api links at the bottom of the page. It has been banned.
Two OTHER sites are EXACTLY the same. PR 4 on both. Small link directory (< 100 links). Many pages of nearly identical content and google api links at bottom of page. Same code, even the same language in many areas... not banned.
None of the sites was a directory site! They all have recip link pages...
Here are the only differences I can see between these sites.
- Banned site had more traffic
- Banned site had more backlinks
- Banned site had more pages indexed by google (way more)
- Banned site was overly SEO optimized on the home page
If this was a simple algorithm flip of the switch, then why didn't the other two (lower traffic, lower indexed) sites get banned as well? Could it be that the algorithm has a pain threshold for amount of scraped/recip content?
The weird thing here is that my banned site saw more googlebot traffic this month than any month in it's history... why would they re-index almost my entire site and then boot it?
Its good to know that I am not alone in having a disappearing site. My main site has completely disappeared including the .com domain name today. It did have a directory, though not a ODP clone. My secondary site has not been hit as hard, although that is the one with the ODP clone directory.
I was wondering though if my sites diasappearance was related to a link I had just added to an affiliated company's site that has 5-10,000 pages linking to mine.
The way I see it, every time now Google makes a change it ends up hurting me instead of the old days when an update was eagerly awaited.
Big_Gig: Excellent point! I noticed a similar occurance on my banned site (it wasn't nearly as large as yours, but what I considered as the meat of my content, was in fact nothing more than a comprehensive niche market human-edited directory).
BUT, the similarity is that I noticed MUCHO googlebot slurping, even though the site was already completely deleted from their SERPs...
And the mystery keeps getting deeper...
"The weird thing here is that my banned site saw more googlebot traffic this month than any month in it's history... why would they re-index almost my entire site and then boot it?"
Same thing happened to me, well, actually not the same but very similar... This month, Google sent more traffic to my site than ever.
Question for those who were banned: We run some strictly opt-in newsletters that are sent out to thousands of subscribers. Do you have any kind of newsletters? Is it possible that google finds out somehow these newsletters and classify them as a spam?
Hey zoltan: I don't think that newsletters are the problem, as I never offered newsletters on my banned site.
("gosh darn" that is depressing - the term "banned" being associate with my site).
In fact, that was always something that I was considering to help increase return visitors... most find the information that they need and I don't see them again (except for those in the industry constantly "spying")
However... now my site has been actually banned by Google, so what do I know anymore? I went from "riches to rags" over-night....
Therefore I must be a bum, according to Google's treatment...
"Leper... Leper... Stay away from me or YOU will become infected and thrown out of the Google search engine community too!"
|Ok, so those that feel like they were executed – could you answer the following to see what they had in common? |
1. Dynamic or static pages?
2. How long were they online?
3. How many pages?
4. How were they originally seeded?
5. Were they free inclusion or paid listings?
6. Were they reciprocal link directories?
7. Template based – what changed in your template besides <title> tags and listings when going through the categories?
8. How many listings per page?
9. Was there other content besides the directory listings?
10. Did you offer/use ROS links?
11. What was the PR before they seem to have been penalized?
12. If you used an off-the-shelf script did you change the default paths etc?
The Contractor has just given the guidelines of running a successful directory. The only points I would add would be -
13. Does a related:www.MyDirectory.com search result in other directories you own?
14. Have you been making a lot of noice/promo about your directory?
Ok, here's another guess: How many of us run multiple sites with a similar theme as the one or two that we've had banned? Many of the posts here seem to suggest that this is common for site owners who have had a site banned. There is probably a high occurance of directory site owners with multiple directory sites. What I'm seeing is that there are some of us who have had sites banned that have mutiple similar themed sites which are not directories. I wonder if this is a warning shot to let us know that we should not try to make too many similar-themed sites?
I still don't hear anyone reporting on obvious known scraper sites. Have they been banned at the same proportion (roughly 25%) as the sites discussed in this thread? I can't be of help there because I don't watch for scraper sites, I barely watch my own sites in the serps.
My heart goes out to those of us who have lost our primary income with this filter, but my experience is that Google has been as reasonable as I could expect in these situations. Typically after they tighten up their filters they seem to manually adjust and eventually the good guys get their listings back. My advice is to not make major changes to your site, unless your site was overly optimized for search engines and under-optimized for human visitors. Now is the time to work on making your sites as useful as you can to human visitors... it will pay off eventually, if not through Google, through their competitors.
I've been at this before Google started and long before AdSense and I can survive without Google, though it would be tough. Fortunately the one site of mine that has been dumped is a minor site, and the loss in traffic has resulted in only a 20% reduction in AdSense revenue for the first day for that site. I would say that I feel personally insulted more than anything, though I know this filter has hurt others in more tangible ways.
|The Contractor has just given the guidelines of running a successful directory |
Maybe i am being thick but i don't see the guidelines - merely questions?
|Typically after they tighten up their filters they seem to manually adjust and eventually the good guys get their listings back. |
This seems true for the ordinary Google updates. After an update there are always many reports on this board of people who went from page #1 to #100, but after some weeks when the algorithm is tuned many come back slowely at reasonable positions.
The difference now is, that most people report a complete removal from the index, not a drop in the SERPs. This would indicate a manual ban which is much more difficult to reverse than a different position in the SERPs. It would have been far easier when they just removed rankings from the sites but let them stay in the index. In that case people here would just call it an update. There must be a reason that they now chose to perform a complete removal. I can only think of two reasons:
- Google is pretty sure that the sites that were removed do not improve their SERPs quality, so they are removed permanently and there is not much hope they will return in a few weeks.
- There is a technical issue/problem on Google's side which caused some domains to drop temporarily. If this is the case however, the ban should be visible on some data centers only, not on all simultaneously. Did people check if their site disappeared on all Google DC's?
>>>>>There is a technical issue/problem on Google's side which caused some domains to drop temporarily. If this is the case however, the ban should be visible on some data centers only, not on all simultaneously. Did people check if their site disappeared on all Google DC's?
Even if the sites have gone on all of the dcs at once - this suggests that Google still has a way of wiping the datacenters of sites in one go - so there may be a master list of domains/canonical urls that might have a technical glitch. However, it does look like bans to me.
Sometimes I'm honestly amazed that I'm even in the same business as some of you guys.
Recip links are not ILLEGAL according to Google. Using Findwhat as a PPC search engine is not ILLEGAL according to Google. Since 1/2004, the PPC engine receives around 15K page views per day. All told, the domain earned around $1,000 per month and NOT from Adsense.
Lucky for me $1,000 a month is a very small portion of my total earnings. I can imagine many guys landing in hot water over this. Bottom line, if you're banning people for a reason: TELL THEM WHY.
It seems Google affected an entire "class" of website owners. They did so with malice and forethought and 0 accountability. Google has made the whole internet marketing business unpleasant with these violent upheavals. I just recovered from the $1,500 per month they've taken away by turning off 4 content sub-domains that were squeaky clean and now this.
|"The weird thing here is that my banned site saw more googlebot traffic this month than any month in it's history... why would they re-index almost my entire site and then boot it?" |
Same thing happened to me, well, actually not the same but very similar... This month, Google sent more traffic to my site than ever.
Same here. Actually two days before my directory was banned, it hits its top traffic day since it started.
|There is a technical issue/problem on Google's side which caused some domains to drop temporarily. If this is the case however, the ban should be visible on some data centers only, not on all simultaneously. Did people check if their site disappeared on all Google DC's? |
That's a primary thing to do. And yes, banned sites were dropped from all Googles DCs.
|"then"? you mean "than" I recon. And no, English is not my 1st nor 2nd language, and still I can write/speak it. What about you, do you know Arabic? |
Let's not turn this thread into a childish flame war please.
I seriously was not trying to (did you read the sentence after that). I remember GG looking at someone's site and bringing them to the carpet on several spelling errors. And BTW "than" is the correct usage and I know that…hehe
|I dont have a definition for a scraper site actually, and I dont care to have one |
You don't have to have one as in this case we are discussing what Google's definition may be. How often do you see dogpile's results ranking in the serps? I never have. Again, a directory and a search engine are two different things.
The above is not off topic in any way, shape, or form.
JuniorOptimizer msg #:73
|Recip links are not ILLEGAL according to Google. Using Findwhat as a PPC search engine is not ILLEGAL according to Google. |
If you believe reciprocal linking (high % of your backlinks) and also using someone else's content exclusively for your site content is a good/long term solution to run your site I believe most SE's wouldn't agree with that (only my opinion of course). If people are waiting for Google or any other SE to come right out and post 1001 things not to do they will be waiting a long time.
I'm not sure why you keep posting your inane commentary in this thread. 0 % of my content on my network of sites is scraped. I have actually paid content writers to write over 5,000 unique pieces of content. My directory was hand-edited. The main reason I asked for recips or payments is because if you offer free submissions you get spammed to death. You get it?
You must be a very dirty webmaster to assume that everyone is as guilty as you. Many, many innocent webmasters are affected by Google all the time.
Googlebot is still visiting my site, I get many requests from: crawl-66-249-65-171.googlebot.com.
|Googlebot is still visiting my site, I get many requests from: crawl-66-249-65-171.googlebot.com. |
Weird! Maybe you are not banned really and you are very stupid that you thought you got banned when you havnt been (according to google automated reply)? :P
Now, seriously, this might give you a chance to clear any trace to the term "directory" in title and meta tags, and waiting for Gbot to reindex you after making your prayers.
I for one, according to awstats, never hit from gbot since the ban exactly 34 hours ago. Yet, I took my advice and did so, but out of being disperate.
The Contractor has just given the guidelines of running a successful directory
|Maybe i am being thick but i don't see the guidelines - merely questions? |
Ofcourse they are not guidelines if you read them by the letter. But those questions guide you along the lines that one should be thinking about while building a directory.
Added: Inspite doing their best, I sincerely feel few may have faced the axe, which might be nothing more than a collateral damage, and I entirely empathize with them. My directory site may still has not been axed, which I would like to think due to all the right things I have done with the site. I still may face an axe, as MB says by one of those eval.google.com knuckle heads.
[edited by: McMohan at 11:32 am (utc) on July 29, 2005]
|I'm not sure why you keep posting your inane commentary in this thread. 0 % of my content on my network of sites is scraped…..The main reason I asked for recips or payments is because if you offer free submissions you get spammed to death. You get it?… You must be a very dirty webmaster to assume that everyone is as guilty as you. |
What's your problem? I am not accusing anyone of anything. I never said anything about paid inclusion – have I? If you can't look at your site and tell what you have done wrong, how am I going to not look at your site and tell you? If you would rather whine, attack, and come up with conspiracy theories then to try to find the reasons why your site may have been booted so be it. Dirty webmaster…nah, ask anyone that knows me. Keep whining, attacking, and coming up with conspiracy theories …
Hmm, I received a referral from google search 30 minutes ago. Of course when I seached the term myself seconds ago, I saw no trace to my poor directory.
Please, let us encourage suggestions and analysis from senior members. The Contractor has only been trying to give his sincere advices/suggestions, if you read them carefully.
In the hour of gloom, one may tend to become self-indulgent and see advices as criticism, but this is not the end of world. Debate out our differences.
Whining? Yet another insult. I'm not whining. I don't have to keep looking at my site to figure out what "I'm guilty of". Google needs to look at their policy and see what they're guilty of.
Typing in 400 category pages by hand is a lot of work. I also wrote the software script. I'm still on pace for $180,000 this year, so I'm not really crying all that bad.
Just so you know: Scrapers actually "scrape" content and then monetize the traffic through AdSense. That's what Googleguy said they went after on July 28th. Plus why exactly did they nuke the entire domain? Two sub-domains on two seperate IP addresses. And I'm supposed to "keep looking" for what may have possible violate their incredibly vague guidelines?
BTW, there are still many scrapers out there.
"Bottom line, if you're banning people for a reason: TELL THEM WHY."
Exactly. I am not going to question Google intenetions, I don't care about WHY they do things - although MONOPOLY in anything does meansresponsibility. the only thing I want is to hear that YES, WE DON'T WANT DIRECTORIES TO STAY IN. Then I wouldn't ask why Alexa is still in, or Yahoo, or DMOZ - I'd let big guys hug & kiss each other, and I would immediately remove directory - the result of hard work for years, even though its human-edited. Just to please Google, because they are monopoly and we have to obey :(
BUT COME AND SAY THIS, Google(Co.)!
p.s. Receive a message for potential cutomer today, telling that our competiros say that "they are not even listed in Google". What this actually means is that this ban directly leads to financial losses to us. Since we have nothing against Google webmaster guadilines or TOS, I am cosidering filing legal complain, this monopoly and no dialog starm making us mad. And G is totally wrong if they think that they're too big to bother.
|I am cosidering filing legal complain |
Good luck. The Google search engine is a free service and judges have ruled in the past that their SERPs must be interpreted as "Google's opinion on the web". Like humans they may change their opinion every time, and deleting sites from the index is just one way to express their opinion.
"Now, seriously, this might give you a chance to clear any trace to the term "directory" in title and meta tags, and waiting for Gbot to reindex you after making your prayers."
Are you sure the problem is witb term: "directory"?
Sorry, I was wrong... actually, it is not googlebot, it is Mediapartners-Google.
|Just so you know: Scrapers actually "scrape" content and then monetize the traffic through AdSense. That's what Googleguy said they went after on July 28th. |
Look, you can agree or disagree with what I am saying, doesn't matter to me. GoogleGuy never said anything about going after sites that scrape for AdSense money so let's keep things factual instead of more theories. Here is the thread [webmasterworld.com...] and you will not see the word AdSense in his post at all (msg#4). I have seen many sites/scripts that have FindWhat/Searchfeed fill and they are usually nothing more than an endless supply of search terms in one column and backfill PPC listings in another. If you call that a directory that is fine, but to me it is not (and by definition it is not). I am not saying that is what you have done, but simply your site may include some of the same qualities or scripting that make it appear this way.
|Plus why exactly did they nuke the entire domain? |
That is what this thread started out to be. What qualities do those nuked sites have in common or what qualities do they have to make them look like scraped, generated, or low quality sites to Google. It seems no-one really wants to figure that out….
Mmmmmm. I have a Yahoo-like directory with a few thousand links for a particular subject and I have seen no eveidence of any type of ban. In fact, for a couple of my money phrases, this site has gone to the #1 and #3 spot. I don't think it is the directory that is the problem. Nor do I think the term "directory" is the problem.
Ok - The Contractor lets give it a stab from sites that I have seen. Sites with a large number of outbound links in a list format (eg like Directories and Scrapers) - I would have thought that this includes links going through a redirect/cgi bin - G must be smart enough to work that out. Sites with content virtually identical to another site - eg Datafeed sites with virtually no unique content, or Newsgroups with no unique content (very very thin pages) ODP clones
Ok - some of the side effects of the above - normal directories will get hit (even ones with unique user submitted listing - the user submitting probably does not vary the text to much between directories), aswell as sites which have a large number of seemingly outbound links as page content.
By the way. I do not run adsense on this site. It is strictly non-profit and there as a free service.