Does Google Ban or Filter Web Directories?

[moftary]

I think the subject worth a thread itself. It's a suspision so far. Yet I don't see dmoz, yahoo nor any major web directory were banned/filter nor PRed zero as my web directory did. I tried to check it in Alexa (powered by google) and I see some results from my site. Appearently, Alexa brings old results from Google but something weird is that Alexa itself has PR0 now. But that's another story!

If you run a web directory, feel free to post your experience here.

[voices]

I expected to see the eventual decline of directories. Look how many adsense spam directory sites were hitting the top of the serps. I assumed that many of the good directories would also take a hit when the adsense spam sites were filtered out. So far my directories are still doing okay. Hoping the spammers don't cause my evenutal demise.

[The Contractor]

I expected to see the eventual decline of directories.

Nope, I think it hits more than directories from what I've seen and been stickied. You may want to read this thread also [webmasterworld.com...]

[moftary]

Now take a look at [dmoz.org...] and you would note that many (not all!) listed directory there got banned (PR0, cached but filtered from serps, etc..). Some but not all. I would pay thousands to know why some but not all, what are the factors that triger a ban/filter. BTW, my directory is not listed there as I dont have dmoz contribution because my directory contents arent totally based on dmoz (complicated story). Briefly, neither the dmoz contribtuion, nor being listed on this dmoz category triger the ban/filter.

[edited by: moftary at 1:55 pm (utc) on July 31, 2005]

[moftary]

Nope, I think it hits more than directories from what I've seen and been stickied.

No, I think these are different incidents. Scrapers were hit on 22nd of july, while directories were hit on the 28th.

[The Contractor]

regarding Sites Using ODP Data. This has always been the case. Many/most never have any PR and that has been that way since I can ever remember.

[moftary]

The Contractor, can you explain why most but not all? Big names are using ODP with no problems including google itself.

[Andem]

>>>The Contractor, can you explain why most but not all? Big names are using ODP with no problems including google itself.

There's still absolutely tons of small sites still using ODP which are still indexed and fairing quite well I would imagine.

[moftary]

I have another theory..

My two banned directories, one had a computers directory and the other was dedicated to the computer directory. My three unbanned directories has no computers directory, they are focusing on different niche market. Someone here also (forgive me for not remembering names) mentioned that he has a computing directory and was banned.

Anyone has a computers directory and wasnt banned? Anyone has no computer directory and was banned?
Confirmations?

[The Contractor]

The Contractor, can you explain why most but not all?

No, I cannot. My guess would be that the ones that don't get hit for it have a higher % of other content Google deems valuable.

[Ian Cunningham]

>>Anyone has a computers directory and wasnt banned? Anyone has no computer directory and was banned

Had on the computer directory listed on my site, and I happened to be listed in one of the categories too.

[zoltan]

My directory (if I can call it directory, actually it was a portal with very few outbond links, only DMOZ had outbond links) was not a computer directory. As I told on other posts, we do have 90% original content and only 10% DMOZ.
I have another theory. Most of the reported cases were talking about having multiple sites with similar or the same topic. As it is my case... The big problem is that the site with the best traffic and PR gets penalized.
Can anyone confirm this?

I have another theory but I keep it for myself for now...

[The Contractor]

If you go to [google.com...] you will see the vast majority of sites (except the large portals) have low/no PR. The vast majority have PR3 or lower. Sure doesn't take much to get a PR 3 so that should tell you what Google thinks about it in the vast majority of cases. I can put up a single page site with one link to it and achieve what most people using ODP data use an awful lot of database/storage space to achieve.

Like I stated in my other post. For those trying to rank by using feeds, dmoz clones, pseudo directories, multiple sites with very similar content – I think those days are coming to an end.

If you want to put up a dmoz clone or other feeds/duplicate content for your users benefit then you should block it from being crawled.

[The Contractor]

zoltan in your case of having multiple sites all using the same content. I am sure the one which was getting all the traffic etc was your first/original site. You then went and built several others at a later date. The problem being is Google "may" be telling you – "hey, one site is fine, but don't build 7 more using the same content" and they hit you where it hurts.

What you did was not real smart (no disrespect meant). You do not build 7 or 8 duplicate sites all sharing the same IP and try to rank for them. People keep saying it's for the users etc. Then block it from being crawled if you are not trying to rank with it. If you are trying to rank with it then sometimes you suffer the consequences.

[zoltan]

And one more thing. Are we talking about scraper sites? Or duplicate content? Or something else?
From what I see, most of the sites that got banned were / are online well before Adsense was introduced in 2003. I saw posts about being online since 1998 or 1999. My banned site is online since 2002, at that time AdSense was nowhere.
As we know scraper sites are a direct result of AdSense, they appeared well AFTER AdSense. I am sure google does have statistics about the age of the site, a simple if age < or > 2003 or something like this would have been resolved the problem (at least a part of the problem).
Any comments on this?

[Andem]

>>>I have another theory. Most of the reported cases were talking about having multiple sites with similar or the same topic. As it is my case... The big problem is that the site with the best traffic and PR gets penalized.

That theory can be dismissed. I run one large site and do not duplicate it in anyway on any other site or on the actual site it's self. When I said 'my largest site', I meant I have one other site which is totally unrelated and is not content-oriented either.. we just sell domain services. No sites are interlinked either.

Edit: Addon -,

Another myth I can dispell is the idea that only low-pr sites are dropped.

My site is divided into 6 main sections: Front page, News, Forums, Weather, Mobility, Technology. Each main section is atleast a PR6 home page and pages deeper can range from PR2-3 to PR6. Since I promote each section of the site separately, there is a lot of PR coming into the site to spread around.

So it can't be that this is limited to low-pr sites, as I've also seen by other examples through sticky.

[edited by: Andem at 2:54 pm (utc) on July 31, 2005]

[zoltan]

The Contractor, our business model is based on Network marketing. We provide content, scripting and hosting to our partners and they earn a % of the revenue generated by their site. We do have nothing to do with our partner sites other than providing them the above elements. They market their sites as they want and as they can.
Sure, many of the sites share similar info with different design (packing) but why did they ban the highest PR and the highest traffic site?

[The Contractor]

Zoltan, you don't seem to understand.

We provide content, scripting and hosting to our partners and they earn a % of the revenue generated by their site.

You provided a complete copy of your site to several people. It doesn't matter who owns the domains they are on. You willingly duplicated your content. You also have them hosted on the same server and IP of the site you are complaining is gone. Who is responsible for that if not you? You have built a network of duplicate content sites all sharing the same database, content, info, and IP address.

I seriously do not understand why you can't understand this....If that is your business model, it is a failed model.

[edited by: The_Contractor at 3:13 pm (utc) on July 31, 2005]

[moftary]

Had on the computer directory listed on my site, and I happened to be listed in one of the categories too.

Ian, I didnt get you, was your site banned or not?

[zoltan]

"I seriously do not understand why you can't understand this....If that is your business model, it is a failed model."
It is not a failed model, it just need some fine tuning. Anyway, I appreciate your comments.

But, we still don't know what is the exact reason of this July 28 mass banning. My business model is not unique but I highly doubt that too many webmasters use it and the main reason of banning was just the duplicate content.

[Ian Cunningham]

>>Ian, I didnt get you, was your site banned or not?

Yes, sorry it is banned. No idea what happened to my grammar during that post! ;)

My site is unique content (based on reviews and articles), with forums as a large part of the site. I had the DMOZ directory as an afterthought for people looking for handy links - what a bad idea that was!

[Andem]

What I don't understand is if they really wanted to get rid of the duplicate content, why don't they just dump the supplementary index? From a technical standpoint, that's the content that means very little. It all doesn't make sense.

[The Contractor]

I had the DMOZ directory as an afterthought for people looking for handy links - what a bad idea that was!

Why not block it via robots.txt and leave it for your users? You may have to use the removal tool to get that section out of the index. If only a portion of your site is based on duplicate content/dmoz and you block/remove it, I would bet it returns to the serps.

[Ian Cunningham]

>>Why not block it via robots.txt and leave it for your users? You may have to use the removal tool to get that section out of the index. If only a portion of your site is based on duplicate content/dmoz and you block/remove it, I would bet it returns to the serps.

As the site has been removed completely from the serps, I've just deleted the DMOZ section as it was a small part of the site that I don't even want to risk future problems with. Its especially annoying as it wasn't even 1% of the site :(

[mahoogle]

I'm starting to see a few results on other data centers that match 66.102.9.104/99. The data centers that seem to be getting these results are 64.233.167.104 and 64.233.189.104. The interesting thing is that only a subset of serps match across these dcs. I'm watching to see if they mostly sync and spread. My site that was hit hard on July 22, that first started to recieve traffic with Bourban has continued to rank well on 66.102.9.104 for the last week, although these results appear to be contained to this dc.

I'm still not sure if the July 22 update was a rollback or the application of some dupe content filter/having multiple related sites filter. I'm hoping it was a rollback and sites that escaped the sandbox with bourbon will now start to reappear.

[iw5edi]

I did a check, and looks like 66.102.9.104/99 has not been updated, or filter is ignored, while title (this sounds interesting to me) link is updated as well as description and cache is dated 29 Jul.

While on the dc you wrote I don't see any changes since 22 July and my domain is still filtered (but not banned)

[mahoogle]

iw5edi
my site is still mostly filtered on all the other IPs, but this morning I noticed a few result sets on the other ips that matched the positions of 66.102.9.104/99 I'm not sure if this is the beginning of a rollout of what we have been seeing on 66.102.9.104 or if these dcs are out of whack. I will say that this is the first time in a week that I have seen any parity between 66.102.9.104 and any of the other dcs.

[dataguy]

I seriously do not understand why you can't understand this....If that is your business model, it is a failed model.

Give me a break. It might not be a the best marketing technique if your only marketing was though Google's free SERPs, but that doesn't necessarily have anything to do with a business model. Many good business models don't rely on free traffic.

Giving affiliates their own site, whether duplicate or not, can be part of a great business model. I know people making thousands per day doing this... hardly a "failed model".

[The Contractor]

Give me a break. It might not be a the best marketing technique if your only marketing was though Google's free SERPs, but that doesn't necessarily have anything to do with a business model. Many good business models don't rely on free traffic.

You are talking apples and oranges with your comment above. I'm not disagreeing....just don't complain when you get caught in a SE's filters believing Google or any other SE should work around your business model.

[AlexMiles]

Carlosnx said,

>Webmasters did place Google where it is and webmasters can remove when we want.

You know, we are gonna have to do this. In fact I think it would be good practice to do it to any engine having more than 25% of the market.

Remember how Google got where they are? We put them there. They paid 1c a search in their affiliate program, users liked them and stuck with them.

These days, as a search engine, they suck. Dogpile is better. There is no reason for the searcher to use Google. I'm thinking meta-engines are the way to go anyhow. Drawing data from multiple engines has to be more stable and I vote Dogpile.

I can't live like this. A month ago Adwords put me out of business far as they were concerned, without so much as a word, Adsense won't let me filter out enough sites to get targeted ads, and Google dumps my perfectly legitimate sites whenever they feel like killing off a few mom and pops and sucking up to their corporate buddies.

Rather like living with a violent, dictatorial partner (with a charming public face) it saps the will. You can't imagine an alternative so you keep trying to please them, desperately hoping they will change.

What? You like scrabbling for hope through multiple datacenters like a bum through a trashcan?

My self respect can't take any more. My 'search the web' links are going to be Dogpile's, I would suggest you all do the same, if your businesses survive the Directory Disaster. If you don't have any search the web links, make some and write off the slight traffic leak to insurance.

Now I know not everyone will, because to get webmasters to take collective action is like herding cats, and thats what Google is relying on. Inertia. But it is our way out of Google Hell. We hit the road and take up with Dogpile, or whoever we can agree on.

.

[excell]

zoltan - your business model may work - that is not the question...but duplication of information is an issue with search engines, so when you allow duplication of websites and pages throughout your network - it is the same as any other business model that allows or instigates same.