homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 588 message thread spans 20 pages: 588 ( [1] 2 3 4 5 6 7 8 9 ... 20 > >     
Does Google Ban or Filter Web Directories?

 1:06 pm on Jul 28, 2005 (gmt 0)

I think the subject worth a thread itself. It's a suspision so far. Yet I don't see dmoz, yahoo nor any major web directory were banned/filter nor PRed zero as my web directory did. I tried to check it in Alexa (powered by google) and I see some results from my site. Appearently, Alexa brings old results from Google but something weird is that Alexa itself has PR0 now. But that's another story!

If you run a web directory, feel free to post your experience here.



 3:49 pm on Jul 28, 2005 (gmt 0)

Got several directories .. but none of them are Dmoz or SEO purposed. No problems so far .. infact been seeing a decent increase in traffic since Bourbon.

I suspect you've been caught by some dupe filters.

Or .. [dmoz.org...]

As far as I can tell its been PR0ed for a while.


 4:31 pm on Jul 28, 2005 (gmt 0)

I run several directories that do pretty well in Google, but as of this morning I see that one of them has been de-indexed by them.

The site is about 2 years old and is squeakey clean as far as SEO goes, but today it is gone from Googles SERPs and is now PR0.

I'm not concerned for the loss in traffic, because it pretty much gets its own traffic from regular users, but I would like to know if I did something wrong. I sure hope that it doesn't happen to any of my other directory sites, but without knowing why one site was de-indexed, I don't think it would be wise to make any changes.

Are there any other directory site owners who have had their directories de-indexed lately?


 4:33 pm on Jul 28, 2005 (gmt 0)

Same thing happened with one of my business to business related sites... Almost 3 year old site, PR5 (till now), thousands of indexed pages so far.
As of today, site completely vanished from google...


 4:57 pm on Jul 28, 2005 (gmt 0)

The only thing I can come up with is that about 10% of the directory pages on the site that was removed overlap with two of my other directories, so they have close to the same content.

Could my site be confused as a scraper site?

zoltan, could you site be in a similar situation?


 5:05 pm on Jul 28, 2005 (gmt 0)

>>>Could my site be confused as a scraper site?

Possible if some knucklehead reported it as such, and someone at Google gave it a cursory look and flipped the switch on it.

I have a hand edited business directory that takes free submissions, it charges no fees, that I filled up using my own knowledge of local businesses. Nothing black hat about it. It went gray bar in April.

My suspicion is that it's a hand ban. I can't know whether the root of it is a false spam report or a blacklisting from an eval.google knucklehead. I suspect it's an eval.google related action.

Not sure if it's related, but at the exact same time, the site in my profile (created for my profile, never promoted), also got a gray bar. Now, if a non-promoted website created just to stick in a WebmasterWorld profile gets graybarred, isn't that just a little weird? Was it an eval.google action gone bad?


 5:49 pm on Jul 28, 2005 (gmt 0)

Got several directories .. but none of them are Dmoz or SEO purposed. No problems so far .. infact been seeing a decent increase in traffic since Bourbon.
I suspect you've been caught by some dupe filters.

In fact my directory has listings from dmoz and yahoo directory as well as its own unique listings. Assuming it's been filterised, as when I search for "www.mydomain.com" I see "showing results 1-4 from 330.000"), the 330.000 are my directory cached pages that were dropped from google index, assuming so why all site subdomains got PRed zero?


As far as I can tell its been PR0ed for a while.

My directory is not listed there. All directories listed there arent penalized.

Also I dont know why I suspect alexa being PR0 is related somehow to the topic. But at least only their www subdomain was PRed zero, not all subdomains as it's the case for me.


 5:58 pm on Jul 28, 2005 (gmt 0)

At first I thought an "underhanded" competitor somehow managed to get my niche industry directory deleted from Google, but after seeing each of your comments and speculations, I'm beginning to wonder if indeed Google is filtering out directories now.

If they are, I think that they better differentiate between "scraper" directories and human edited sites like my directory is. I have been compiling and publishing my industry links since early 1997 before there was a Google and before I had competitors, because I helped create my niche market sector by being one of the first in the industry, and definitely the first to promote the entire industry without bias by providing FREE links to all of my direct competitors.

Since that time (moving forward as the buzzwords go), I have been publishing updated versions of my original information as an "independent enthusiast" web site. By far I have the most comprehensive directory on this IMPORTANT topic (good thing the United Nations had the site bookmarked - no political comments please), and it's relevancy always helped it to be among the top in Google's SERPs, until early this morning. Most everyone worldwide has need some time or another to reference this information I provide, yet Google seems to be deliberately choosing to CENSOR the information in preference to some of THE WORSE scarper sites I have ever witnessed, along with a few, good industry web sites.

I have written to Google for explanation, and waiting for reply. Apparently it's not because of Google AdSense TOS violation (if indeed Google itself did the deleting), as AdSense ads still appear on the pages and clicks are being tallied from visitors finding the pages from other search engines. However, I only now obtain 10% of the traffic I had before being completely deleted from Google's data base.

martinibuster - I have similar suspicions that you have: I am now pondering if the "nightmares" that some were speculating about has finally began to crop up... jealous/unscrupulous web masters turning in competing sites as TOS violators or Spam scraper sites, and Google employees not familiar enough with the topic and its historic development to make an accurate relevancy determination in the few seconds that they take to judge a web site, deletes good factual content because of the disparaging words of a competitor or someone else with an axe to grind.

Nobody is safe from worry now (believe me I had one of THE most RELEVANT, UNBIASED web sites in a highly competitive sector)

Guess I should start making scraper web sites? They seem to be still in Google's SERPS in my sector.... That's the lesson I am learning from this example.

What say you Google reps?


 5:58 pm on Jul 28, 2005 (gmt 0)

As I already posted in other threads, our site with a directory has also been banned.

On the other hand we have a few other sites with directories, but they are not banned. What I think is the following:

1. Google gone mad with "scrapers"
2. They decided to ban sites where directory is the majority of site content pages, assuming that such site would be a scraper
3. They don't care much about casualties, because they are big, kind Google that brings traffic in your home for free and you have to be happy.

On the other hand of the story:

1. Lots of real scrapers are still in, and its totally irrelevant to try to determine them automatically or basing on reports of possible competitors.
2. Simple case (ours actually) - site offers 'widgets' and has normal 10-20 pages devoded to widgets, plus forum, and secured area. Additionally the site has a small directory - 5-10K sites - also devoded to 'widgetry, widget components and supplementary services'. The directory would have MUCH more pages than the main site.
3. Once the site get booted with no way back, no clue, no chance and NO PRIOR WINK that anything it does is against any TOS, lots of people may suffer dramatically, and Google IS responsible, just as when you're happened to become a dominant internet power, you should care about the internet and people who are part of it.

Ian Cunningham

 6:10 pm on Jul 28, 2005 (gmt 0)

This is interesting... try searching for these sites in google (DMOZ data sites):


Many don't even appear!


 6:18 pm on Jul 28, 2005 (gmt 0)

Well I am totally off the index... I had ONE google referral about an hour ago, but it must have been some kind of a fluke. Compared with yesterday when I had several thousand by this time of day. I feel like the last four years of my life have just drained away.


 6:22 pm on Jul 28, 2005 (gmt 0)

Alexa.com still comes up with PR7 to me. Touch wood, none of my directories got hurt one bit (hope it's not a matter of "yet").

Can someone please define what a scraper site is?


 6:42 pm on Jul 28, 2005 (gmt 0)

Hi Martinibuster!

>the site in my profile (created for my profile, never promoted), also got a gray bar.

From here (Italy) your site shows a Page Rank of 3!


 7:11 pm on Jul 28, 2005 (gmt 0)

My main site contains about 180 pages of content and then has a directory. As of some point this morning the complete site has vanished from Google ( even the site:www.domain.com command shows no pages).

Yesterday I was a PR6 today nothing. Needless to say the traffic has slowed to a trickle - so much for the past 8 years :-(


 7:16 pm on Jul 28, 2005 (gmt 0)

I run a review site which has been dropped to PR 0 and kicked out of the SERPS as of today. I am not a directory, but I do have many pages with google search api links below my article content.

These scraped linked are less than 10% of my total pages. I have another 150,000 pages of quality newsgroup discussions.

"Scraped" content does not make up a large part of my site... what else could be causing this?


 7:55 pm on Jul 28, 2005 (gmt 0)

I just received a friendly email from the Google AdSense Team saying they are forwarding my inquiry to Google's User Support team. They also attempted to reassure me that there is almost nothing a competitor can do to harm our web sites in Google's search engine data base... I like their frankness but I don't like their "almost nothing"...

Well at least hopefully I will get a definative answer to what happened, and hopefully they can/I can fix it... otherwise looks like I'm almost back at square one as far as AdSense earnings go, after over a year's dedication to publishing the most relevant information about my niche as possible.

I hope it's just something stupid and easily fixed!

"OHHHHHHHHH the pain & agony of it all....." Who was saying earning Google AdSense was "so easy!"


 7:59 pm on Jul 28, 2005 (gmt 0)

alwaysthinking, make sure you post here the response you get from google.


 8:12 pm on Jul 28, 2005 (gmt 0)

Since I operate several directory sites and only one of them got hit. This, along with the fact that there are numerous directory sites being reported that got hit all at once, I suspect that G is tightening up their filters against scraper sites, or maybe trying a new filter altogether.

Google isn't stupid, and I'm sure that they verify sites that they are asked about, so I immediately filed a re-inclusion request and hopefully they will agree that my site should not have been dropped.


 8:16 pm on Jul 28, 2005 (gmt 0)

dataguy, the same situation with me. Only one site dropped, they did not touch the other sites.
Please share google's response. BTW, where did you file a re-inclusion request?


 8:27 pm on Jul 28, 2005 (gmt 0)

zoltan, this is where I started from:

I'll let you know what kind of response I get. Please, anyone else that's going through this same situation, post your experiences here as well.


 8:33 pm on Jul 28, 2005 (gmt 0)

>>>From here (Italy) your site shows a Page Rank of 3!

Yep, it was fixed a week after it happened. So-called "canonical url" issues...

[edited by: martinibuster at 8:42 pm (utc) on July 28, 2005]


 8:34 pm on Jul 28, 2005 (gmt 0)

I also operate several of the review sites I mentioned above, with many many pages of discussions, and only my largest site was dropped.

I filed a request for review through the same avenue as dataguy, and received the default "your site might not have actually been dropped, and here's our webmaster information" email. I responded to the email with my complaint (begging slobbering crying request for reinclusion!) and I'll let you know when (if?) they respond.


 9:08 pm on Jul 28, 2005 (gmt 0)

It looks like my directory was banned this morning.

No pages, no cache, no links and no PR.

Big Hands came in and killed the entire domain.


 9:41 pm on Jul 28, 2005 (gmt 0)

This is the ONLY "major" change that I made to my Directory site that was deleted... added a 301 redirect using coding someone here recommended:

RewriteEngine ON
RewriteCond %{HTTP_HOST} ^example.com
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

It works great, and forced the "www" on to the few links in the search engines that had dropped the "www" I do NOT think this could be the problem...

For further clarification, when I say "my directory," I am speaking about the "best content" part of the web site. There are other sections of the site that has other types of information, including artcles.

However, along with my "directory site," another client's web site that I linked to was also completely deleted from Google's search data, and that particular site could NOT be considered a directory in any manner of thinking, yet it was also deleted at the same time.

The information on that site is complimentary to the front page content of the "primary" deleted site, and there was only one link in the navigational side bar to the other site - nothing I would consider excessive "interlinking." Also, they were not "hidden" links, just straight forward text description navigational links between the web sites with complimentary content. Different hosting servers too...

Finally, on some pages of the deleted "primary" web sites, there were links to my "secondary" web site that has relatively good web presense in Yahoo & MSN, while only minimal presense in Google's SERPs (it wasn't designed for Google). Yet, the secondary web site remains in Google (and the others). If there was a problem between the three sites... why didn't they all become deleted?

Anyway, my point is that the recent complete deletion that some of our sites experienced last night - last few days, might not necessarily be because of filters on Directory web sites. However, that just leaves it to be an even greater mystery.


 9:54 pm on Jul 28, 2005 (gmt 0)

This is the ONLY "major" change that I made to my Directory site that was deleted... added a 301 redirect using coding someone here recommended:
RewriteEngine ON
RewriteCond %{HTTP_HOST} ^example.com
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

I also made this exact change recently and it was the first thing I changed back this morning when my site (a directory) disappeared from Google. I admit it may be something else entirely but if a few more of those dropped added this code to their .htaccess we may have a pattern. I hope so, because at least that was easy to fix.

Ian Cunningham

 10:00 pm on Jul 28, 2005 (gmt 0)

I also have a re-direct from my old forum pages to the new location - although I have used this for a long time to keep legacy links alive.


 10:03 pm on Jul 28, 2005 (gmt 0)

andrea99 - yes, that is why I mentioned the addition of the 301 redirect... I was/am a little leary that that MIGHT have caused the problem...

However, the other client's complimentary content web site that I was linking to did NOT have the 301 redirect... so that's why I doubt if that is what cause both sites to be deleted from Google SERPs last night.


 10:10 pm on Jul 28, 2005 (gmt 0)

alwaysthinking - at this point I am hanging on to any shred of hope that this can be reversed. :)


 10:12 pm on Jul 28, 2005 (gmt 0)

I don't do any redirecting, and since it's on a Windows server (actually it's spread across 2 Windows servers, 1 with a sub-domain) it couldn't use htaccess even if I wanted it to. This site is 99.5% static pages, so just about the only scripting the site contains is the javascript for AdSense.

This site did have about 85,000 pages indexed by Google, so it is a pretty large site. Is this common in the sites that have been de-listed?


 10:13 pm on Jul 28, 2005 (gmt 0)

I have this redirect (non www to www) since the beginning of my website. Don't think this is the problem.

This 588 message thread spans 20 pages: 588 ( [1] 2 3 4 5 6 7 8 9 ... 20 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved