Welcome to WebmasterWorld Guest from

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Did my site drop out of Google index?

6:35 am on Nov 5, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Oct 28, 2005
votes: 0

When I enter 'site: mysite.com', shouldn't I essentially be seeing my site at the top of the results? I see 118,000 results, but my site is not in the results.

It has been around for four years and has 550 pages apprx. Yesterday my traffic from Google seemed to drop by over 50%, from 2200 uniques/day, to 850. There are no server problems/downtimes/issues.

How can I tell if my site has been penalized by G?

3:02 pm on Nov 5, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
votes: 0

Are you by any chance placing a space between the colon and your domain name?

There should be no space -- in other words site:example.com -- and what you describe would be explained if there were a space. In that case, instead of calling the special "site:" operator you would be doing an ordinary 2-word search.

[edited by: tedster at 4:12 pm (utc) on Nov. 5, 2006]

1:42 am on Nov 6, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Oct 28, 2005
votes: 0

Thks for the reply.

I did use it with no space and still, there are no pages indexed at all. I went to the diagnostics tab of the Google Sitemaps page and see the message that currently there are no pages of mine in the index.

What happened is, on Sat Nov 4.06, Google stripped every single page of mine out of its index at once. There is absolutely nothing on my site that violates any of its policies/guidelines at all. I can sticky you the URL if you like to show you that the site is 100% clean.

The only thing showing on the Google Sitemaps diagnostic tab, is an http header error for every directory on my site. When I looked at my large htaccess file, I removed all of the 404/403 directives in case that this was the problem. In fact, I removed all most of the stuff in the htaccess file, except for the basic necessary stuff. Some things in my htaccess that could have been the cause, like for eg, I blocked any user agent with an empty dash or string:

RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule!\.403\.php$ - [F]

I did a reinclusion request yesterday, and can imagine that this thing could take months to resolve, based on dozens of threads here on WebmasterWorld I read where people faced similar things. I wonder if I will ever find the cause of this.

Does Google usually kill every page of a site for no reason? Even when people did "black hat" stuff, it seems they just were dropped to "supplemental results" or dropped serps.

5:55 am on Nov 6, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 19, 2004
votes: 0

Post the original code so that others can determine whether you indvertently caused Google not not index any of your pages.

If this is the case, you may be able to explain this to them and get in quicker.

7:38 am on Nov 6, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Oct 28, 2005
votes: 0

Sure, thanks. Here is the htaccess code:

Options -Indexes
DirectoryIndex index.php
ErrorDocument 404 /404/404.php
ErrorDocument 403 "403 Forbidden"

SetEnvIf Referer ^$ empty_referer
Deny from env=3Dempty_referer

<Limit GET>
order allow,deny
allow from all

<Files .htaccess>
order allow,deny
deny from all

<Files 403.shtml>
order allow,deny
allow from all

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^example\.net [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]

RewriteEngine on
# RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://subdomain.example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://subdomain.example.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.subdomain.example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.subdomain.example.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.example.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://example.com$ [NC]
RewriteRule .*\.(jpg夸peg夙if如ng在mp.*)$ http://www.example.com/$1 [R,NC]

RewriteCond %{HTTP_REFERER} (Field\sblocked同roxyWay吏utpost再gnitum地nonym存teganos吉orton再ddSubtract) [NC]
RewriteRule!\.403\.php$ - [F]

# An empty string or a dash is not a valid UA string.
# Beware : you may not want to use this rule. Check your logs to see whether you
# need it or not.
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule!\.403\.php$ - [F]
# -----------------------------------------------------
# A user-agent (or a referer) which string is :
# #*$!X:+++++++++++++++++++++++++++++++++++++++++++++
# and another which is a series of dashes
RewriteCond %{HTTP_USER_AGENT} ^xx+:\++$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^--+$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^xx+:\++$ [NC]
RewriteRule!\.403\.php$ - [F]

# Forbid if blank (or "-") Referer *and* UA
RewriteCond %{HTTP_REFERER} ^-?$
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule .* - [F]

# Forbid requests for exploits & annoyances
# Bad requests
# CodeRed
RewriteCond %{REQUEST_URI} ^/default\.(ida夷dq) [NC,OR]
RewriteCond %{REQUEST_URI} ^/.*\.printer$ [NC,OR]
# Email
RewriteCond %{REQUEST_URI} (mail.?form圩orm圩orm.?mail妃ail妃ailto)\.(cgi圯xe如l)$ [NC,OR]
# MSOffice
RewriteCond %{REQUEST_URI} ^/(MSOffice囝vti) [NC,OR]
# Nimda
RewriteCond %{REQUEST_URI} /(admin圭md多ttpodbc好siislog字oot存hell)\.(dll圯xe) [NC,OR]
# Various
RewriteCond %{REQUEST_URI} ^/(bin/圭gi/圭gi\-local/存umthin) [NC,OR]
RewriteCond %{THE_REQUEST} ^GET\ http [NC,OR]
RewriteCond %{REQUEST_URI} /sensepost\.exe [NC]
RewriteRule .* - [F]

RewriteCond %{HTTP_USER_AGENT} Anonym [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CherryPicker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Clariabot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Crescent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DA\s7\.0 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DataCha0s/2\.0 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Diamond(/1\.0在ot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DIIbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Dual.Proxy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EmailCollector [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EmailWolf [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ExtractorPro [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Fasterfox [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ia_archiver [NC,OR]
RewriteCond %{HTTP_USER_AGENT} internetseer\.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Kontent_Smarts [NC,OR]
RewriteCond %{HTTP_USER_AGENT} larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} LinkWalker [NC,OR]
# See [webmasterworld.com...]
RewriteCond %{HTTP_USER_AGENT} Microsoft\.URL [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Mozilla.*NEWT [NC,OR]
RewriteCond %{HTTP_USER_AGENT} nicebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} psbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Steganos [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Teleport [NC,OR]
RewriteCond %{HTTP_USER_AGENT} UtilMind.HTTPGet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} webbandit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebEMailExtrac [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Wget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Xenu\sLink\sSleuth [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Zeus.*Webster
RewriteRule!\.403\.php$ - [F]

# Mozilla/4.0 is not a valid Moz-like UA string.
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0$ [NC]
RewriteRule!\.403\.php$ - [F]

[edited by: tedster at 8:30 pm (utc) on Nov. 6, 2006]
[edit reason] use example.com [/edit]