Welcome to WebmasterWorld Guest from 54.211.62.139

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Does Google Ban or Filter Web Directories?

     
1:06 pm on Jul 28, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 21, 2003
posts:427
votes: 0


I think the subject worth a thread itself. It's a suspision so far. Yet I don't see dmoz, yahoo nor any major web directory were banned/filter nor PRed zero as my web directory did. I tried to check it in Alexa (powered by google) and I see some results from my site. Appearently, Alexa brings old results from Google but something weird is that Alexa itself has PR0 now. But that's another story!

If you run a web directory, feel free to post your experience here.

3:49 pm on July 28, 2005 (gmt 0)

Senior Member from HK 

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 14, 2002
posts:2291
votes: 16


Got several directories .. but none of them are Dmoz or SEO purposed. No problems so far .. infact been seeing a decent increase in traffic since Bourbon.

I suspect you've been caught by some dupe filters.

Or .. [dmoz.org...]

As far as I can tell its been PR0ed for a while.

4:31 pm on July 28, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 12, 2004
posts:1355
votes: 0


I run several directories that do pretty well in Google, but as of this morning I see that one of them has been de-indexed by them.

The site is about 2 years old and is squeakey clean as far as SEO goes, but today it is gone from Googles SERPs and is now PR0.

I'm not concerned for the loss in traffic, because it pretty much gets its own traffic from regular users, but I would like to know if I did something wrong. I sure hope that it doesn't happen to any of my other directory sites, but without knowing why one site was de-indexed, I don't think it would be wise to make any changes.

Are there any other directory site owners who have had their directories de-indexed lately?

4:33 pm on July 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Sept 6, 2003
posts:332
votes: 0


Same thing happened with one of my business to business related sites... Almost 3 year old site, PR5 (till now), thousands of indexed pages so far.
As of today, site completely vanished from google...
4:57 pm on July 28, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 12, 2004
posts:1355
votes: 0


The only thing I can come up with is that about 10% of the directory pages on the site that was removed overlap with two of my other directories, so they have close to the same content.

Could my site be confused as a scraper site?

zoltan, could you site be in a similar situation?

5:05 pm on July 28, 2005 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14314
votes: 268


>>>Could my site be confused as a scraper site?

Possible if some knucklehead reported it as such, and someone at Google gave it a cursory look and flipped the switch on it.

I have a hand edited business directory that takes free submissions, it charges no fees, that I filled up using my own knowledge of local businesses. Nothing black hat about it. It went gray bar in April.

My suspicion is that it's a hand ban. I can't know whether the root of it is a false spam report or a blacklisting from an eval.google knucklehead. I suspect it's an eval.google related action.

Not sure if it's related, but at the exact same time, the site in my profile (created for my profile, never promoted), also got a gray bar. Now, if a non-promoted website created just to stick in a WebmasterWorld profile gets graybarred, isn't that just a little weird? Was it an eval.google action gone bad?

5:49 pm on July 28, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 21, 2003
posts:427
votes: 0


Got several directories .. but none of them are Dmoz or SEO purposed. No problems so far .. infact been seeing a decent increase in traffic since Bourbon.
I suspect you've been caught by some dupe filters.

In fact my directory has listings from dmoz and yahoo directory as well as its own unique listings. Assuming it's been filterised, as when I search for "www.mydomain.com" I see "showing results 1-4 from 330.000"), the 330.000 are my directory cached pages that were dropped from google index, assuming so why all site subdomains got PRed zero?

http://dmoz.org/Computers/Internet/Searching/Directories/

As far as I can tell its been PR0ed for a while.

My directory is not listed there. All directories listed there arent penalized.

Also I dont know why I suspect alexa being PR0 is related somehow to the topic. But at least only their www subdomain was PRed zero, not all subdomains as it's the case for me.

5:58 pm on July 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Feb 4, 2005
posts:231
votes: 0


At first I thought an "underhanded" competitor somehow managed to get my niche industry directory deleted from Google, but after seeing each of your comments and speculations, I'm beginning to wonder if indeed Google is filtering out directories now.

If they are, I think that they better differentiate between "scraper" directories and human edited sites like my directory is. I have been compiling and publishing my industry links since early 1997 before there was a Google and before I had competitors, because I helped create my niche market sector by being one of the first in the industry, and definitely the first to promote the entire industry without bias by providing FREE links to all of my direct competitors.

Since that time (moving forward as the buzzwords go), I have been publishing updated versions of my original information as an "independent enthusiast" web site. By far I have the most comprehensive directory on this IMPORTANT topic (good thing the United Nations had the site bookmarked - no political comments please), and it's relevancy always helped it to be among the top in Google's SERPs, until early this morning. Most everyone worldwide has need some time or another to reference this information I provide, yet Google seems to be deliberately choosing to CENSOR the information in preference to some of THE WORSE scarper sites I have ever witnessed, along with a few, good industry web sites.

I have written to Google for explanation, and waiting for reply. Apparently it's not because of Google AdSense TOS violation (if indeed Google itself did the deleting), as AdSense ads still appear on the pages and clicks are being tallied from visitors finding the pages from other search engines. However, I only now obtain 10% of the traffic I had before being completely deleted from Google's data base.

martinibuster - I have similar suspicions that you have: I am now pondering if the "nightmares" that some were speculating about has finally began to crop up... jealous/unscrupulous web masters turning in competing sites as TOS violators or Spam scraper sites, and Google employees not familiar enough with the topic and its historic development to make an accurate relevancy determination in the few seconds that they take to judge a web site, deletes good factual content because of the disparaging words of a competitor or someone else with an axe to grind.

Nobody is safe from worry now (believe me I had one of THE most RELEVANT, UNBIASED web sites in a highly competitive sector)

Guess I should start making scraper web sites? They seem to be still in Google's SERPS in my sector.... That's the lesson I am learning from this example.

What say you Google reps?

5:58 pm on July 28, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 22, 2003
posts:148
votes: 0


As I already posted in other threads, our site with a directory has also been banned.

On the other hand we have a few other sites with directories, but they are not banned. What I think is the following:

1. Google gone mad with "scrapers"
2. They decided to ban sites where directory is the majority of site content pages, assuming that such site would be a scraper
3. They don't care much about casualties, because they are big, kind Google that brings traffic in your home for free and you have to be happy.

On the other hand of the story:

1. Lots of real scrapers are still in, and its totally irrelevant to try to determine them automatically or basing on reports of possible competitors.
2. Simple case (ours actually) - site offers 'widgets' and has normal 10-20 pages devoded to widgets, plus forum, and secured area. Additionally the site has a small directory - 5-10K sites - also devoded to 'widgetry, widget components and supplementary services'. The directory would have MUCH more pages than the main site.
3. Once the site get booted with no way back, no clue, no chance and NO PRIOR WINK that anything it does is against any TOS, lots of people may suffer dramatically, and Google IS responsible, just as when you're happened to become a dominant internet power, you should care about the internet and people who are part of it.

6:10 pm on July 28, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:July 30, 2003
posts:81
votes: 0


This is interesting... try searching for these sites in google (DMOZ data sites):

[dmoz.org...]

Many don't even appear!

6:18 pm on July 28, 2005 (gmt 0)

Preferred Member

joined:July 8, 2002
posts:584
votes: 0


Well I am totally off the index... I had ONE google referral about an hour ago, but it must have been some kind of a fluke. Compared with yesterday when I had several thousand by this time of day. I feel like the last four years of my life have just drained away.
6:22 pm on July 28, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 20, 2004
posts:58
votes: 0


Alexa.com still comes up with PR7 to me. Touch wood, none of my directories got hurt one bit (hope it's not a matter of "yet").

Can someone please define what a scraper site is?

6:42 pm on July 28, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 23, 2000
posts:1186
votes: 0


Hi Martinibuster!

>the site in my profile (created for my profile, never promoted), also got a gray bar.

From here (Italy) your site shows a Page Rank of 3!

7:11 pm on July 28, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 22, 2005
posts:164
votes: 0


My main site contains about 180 pages of content and then has a directory. As of some point this morning the complete site has vanished from Google ( even the site:www.domain.com command shows no pages).

Yesterday I was a PR6 today nothing. Needless to say the traffic has slowed to a trickle - so much for the past 8 years :-(

7:16 pm on July 28, 2005 (gmt 0)

New User

10+ Year Member

joined:May 21, 2005
posts:21
votes: 0


I run a review site which has been dropped to PR 0 and kicked out of the SERPS as of today. I am not a directory, but I do have many pages with google search api links below my article content.

These scraped linked are less than 10% of my total pages. I have another 150,000 pages of quality newsgroup discussions.

"Scraped" content does not make up a large part of my site... what else could be causing this?

7:55 pm on July 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Feb 4, 2005
posts:231
votes: 0


I just received a friendly email from the Google AdSense Team saying they are forwarding my inquiry to Google's User Support team. They also attempted to reassure me that there is almost nothing a competitor can do to harm our web sites in Google's search engine data base... I like their frankness but I don't like their "almost nothing"...

Well at least hopefully I will get a definative answer to what happened, and hopefully they can/I can fix it... otherwise looks like I'm almost back at square one as far as AdSense earnings go, after over a year's dedication to publishing the most relevant information about my niche as possible.

I hope it's just something stupid and easily fixed!

"OHHHHHHHHH the pain & agony of it all....." Who was saying earning Google AdSense was "so easy!"

7:59 pm on July 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Sept 6, 2003
posts:332
votes: 0


alwaysthinking, make sure you post here the response you get from google.
8:12 pm on July 28, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 12, 2004
posts:1355
votes: 0


Since I operate several directory sites and only one of them got hit. This, along with the fact that there are numerous directory sites being reported that got hit all at once, I suspect that G is tightening up their filters against scraper sites, or maybe trying a new filter altogether.

Google isn't stupid, and I'm sure that they verify sites that they are asked about, so I immediately filed a re-inclusion request and hopefully they will agree that my site should not have been dropped.

8:16 pm on July 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Sept 6, 2003
posts:332
votes: 0


dataguy, the same situation with me. Only one site dropped, they did not touch the other sites.
Please share google's response. BTW, where did you file a re-inclusion request?
8:27 pm on July 28, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 12, 2004
posts:1355
votes: 0


zoltan, this is where I started from:
[google.com...]

I'll let you know what kind of response I get. Please, anyone else that's going through this same situation, post your experiences here as well.

8:33 pm on July 28, 2005 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14314
votes: 268


>>>From here (Italy) your site shows a Page Rank of 3!

Yep, it was fixed a week after it happened. So-called "canonical url" issues...

[edited by: martinibuster at 8:42 pm (utc) on July 28, 2005]

8:34 pm on July 28, 2005 (gmt 0)

New User

10+ Year Member

joined:May 21, 2005
posts:21
votes: 0


I also operate several of the review sites I mentioned above, with many many pages of discussions, and only my largest site was dropped.

I filed a request for review through the same avenue as dataguy, and received the default "your site might not have actually been dropped, and here's our webmaster information" email. I responded to the email with my complaint (begging slobbering crying request for reinclusion!) and I'll let you know when (if?) they respond.

9:08 pm on July 28, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 9, 2003
posts:735
votes: 0


It looks like my directory was banned this morning.

No pages, no cache, no links and no PR.

Big Hands came in and killed the entire domain.

9:41 pm on July 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Feb 4, 2005
posts:231
votes: 0


This is the ONLY "major" change that I made to my Directory site that was deleted... added a 301 redirect using coding someone here recommended:

RewriteEngine ON
RewriteCond %{HTTP_HOST} ^example.com
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

It works great, and forced the "www" on to the few links in the search engines that had dropped the "www" I do NOT think this could be the problem...

For further clarification, when I say "my directory," I am speaking about the "best content" part of the web site. There are other sections of the site that has other types of information, including artcles.

However, along with my "directory site," another client's web site that I linked to was also completely deleted from Google's search data, and that particular site could NOT be considered a directory in any manner of thinking, yet it was also deleted at the same time.

The information on that site is complimentary to the front page content of the "primary" deleted site, and there was only one link in the navigational side bar to the other site - nothing I would consider excessive "interlinking." Also, they were not "hidden" links, just straight forward text description navigational links between the web sites with complimentary content. Different hosting servers too...

Finally, on some pages of the deleted "primary" web sites, there were links to my "secondary" web site that has relatively good web presense in Yahoo & MSN, while only minimal presense in Google's SERPs (it wasn't designed for Google). Yet, the secondary web site remains in Google (and the others). If there was a problem between the three sites... why didn't they all become deleted?

Anyway, my point is that the recent complete deletion that some of our sites experienced last night - last few days, might not necessarily be because of filters on Directory web sites. However, that just leaves it to be an even greater mystery.

9:54 pm on July 28, 2005 (gmt 0)

Preferred Member

joined:July 8, 2002
posts:584
votes: 0


This is the ONLY "major" change that I made to my Directory site that was deleted... added a 301 redirect using coding someone here recommended:
RewriteEngine ON
RewriteCond %{HTTP_HOST} ^example.com
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

I also made this exact change recently and it was the first thing I changed back this morning when my site (a directory) disappeared from Google. I admit it may be something else entirely but if a few more of those dropped added this code to their .htaccess we may have a pattern. I hope so, because at least that was easy to fix.
10:00 pm on July 28, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:July 30, 2003
posts:81
votes: 0


I also have a re-direct from my old forum pages to the new location - although I have used this for a long time to keep legacy links alive.
10:03 pm on July 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Feb 4, 2005
posts:231
votes: 0


andrea99 - yes, that is why I mentioned the addition of the 301 redirect... I was/am a little leary that that MIGHT have caused the problem...

However, the other client's complimentary content web site that I was linking to did NOT have the 301 redirect... so that's why I doubt if that is what cause both sites to be deleted from Google SERPs last night.

10:10 pm on July 28, 2005 (gmt 0)

Preferred Member

joined:July 8, 2002
posts:584
votes: 0


alwaysthinking - at this point I am hanging on to any shred of hope that this can be reversed. :)
10:12 pm on July 28, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 12, 2004
posts:1355
votes: 0


I don't do any redirecting, and since it's on a Windows server (actually it's spread across 2 Windows servers, 1 with a sub-domain) it couldn't use htaccess even if I wanted it to. This site is 99.5% static pages, so just about the only scripting the site contains is the javascript for AdSense.

This site did have about 85,000 pages indexed by Google, so it is a pretty large site. Is this common in the sites that have been de-listed?

10:13 pm on July 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Sept 6, 2003
posts:332
votes: 0


I have this redirect (non www to www) since the beginning of my website. Don't think this is the problem.
This 588 message thread spans 20 pages: 588