Forum Moderators: open
Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.
At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.
The index-page was always in the top-3 position for our main topics, aka keywords.
I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index
As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.
My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.
Lets call them:
www.foo1.net and www.foo2.net
Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)
The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window
Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).
Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.
So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.
Fetched the page, but again got a code 200 with no redirects at all.
Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"
Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.
Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html
Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.
None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo
I have send an email to Google about this and am still waiting for a responds.
Here is something that might be of help (thanks to Vin DSL). If you can edit your .htaccess file, you can add the following code (only for unix based servers).
RewriteEngine on
php_flag display_errors off
php_flag register_globals off
RewriteCond %{HTTP_REFERER} ^http://(www\.)?offendingSite.com/.*$
RewriteCond %{REQUEST_URI} ^/.*$
RewriteRule ^.* - [F]
Replace "offendingSite.com" with the correct domain.
What this does is, any clicks from the site itself should get an error (403 forbidden) but someone searching on google will still be able to use the offending link to find your site since the referrer is google.
The idea is next time google crawls the offending site it'll get a 403 error and won't be able to cache your page.
I'm not sure if this will actually work on googlebot but it should if it checks like a browser. Perhaps some of the senior members who have experience with crawler bots can confirm this.
Googlebot doesn't deliver referers ...
What's happening with this code write now is: following the link from the directory is resulting in this standard error:
Forbidden
You don't have permission to access / on this server.
--------------------------------------------------------------------------------
Apache/1.3.31 Server at www.mysite.com Port 80
but through google (if you do a search) the link works.
In addition, robots do not follow links directly as users do, they record them and visit them later, therefore no referrer data could sensibly exist. Whilst it is possible that some redirects are followed directly (unlike regular links) there would still be no referrer data.
This does raise an interesting point. Previously I have been almost certain that the fault lies in the indexing service, however, if googlebot is following redirects as browsers do (rather than treating them as links) then this might explain a great deal - in fact, it might explain everything.
I really hate being wrong, but I think my analysis earlier in this thread may have been.
Kaled.
An update, earlier my index page started appearing once again in google's cache (dated oct1). But the old hijacked link is still cached and ranks as #1 for my keyphrase where as the real link isn't even in the top 100.
Going to wait and see if things improve but if not I think I'll write to them to fix this.
The problem is obviously a 302 redirect from a lesser site. His site is now rated #5 in a category I have always rated #1.
I have a change. That freaking other URL has finally disappeared when I do a search on my domain name. However, Google now gives my [my-site].com URL instead of the www.[my-site].com URL.
I don't really understand that, and I have 301 redirects from the non-www pages to the www-pages.
However, the other directory that hijacked me is still in there. So, who knows if they will take over where the other place used to be when searching for my domain name, soon.
I still wonder if I should put a link to that page from mine, but nobody has answered my question, and I don't want to make things worse.
Anyway, that's where I'm at now.
When www.mydomain.com re-appeared in google yesterday only then it had a cache (2 days old) of the page and could be searched.
I haven't asked the offending site to remove their link. I'm curious to see if google will fix itself (which it slowly is from the looks of it). I don't think you should put a link to that site though, I don't think it's helpful.
I think the fastest way to get the offending page removed would be to have them point it to a blank page on your or any page on their site. So next time google crawls it'll replace the old cache and the offending page would drop very fast in rankings. I think this is the best solution.
I think the fastest way to get the offending page removed would be to have them point it to a blank page on your or any page on their site. So next time google crawls it'll replace the old cache and the offending page would drop very fast in rankings. I think this is the best solution.
Ashdar, that site used to have a meta refresh and a 302 redirect to my site. They have dropped the meta refresh and changed the link to a 301 redirect, which should be fine, but Google hasn't crawled that page of its directory, yet.
My site is cached, and my index page with the www is in Google, I was only referring to what appears when you search wwww.[my-site].com. That situation had to do with a different directory. My site was hijacked by two different sites simultaneously.
I switched to redirect 301 and got rid of metarefresh last week. In addition to that I don't allow to crawl any php files, but I see new cache of offended pages every day, although links don't go to those pages - they go to my internal pages. What is happening? Any help would be appreciated.
Thanks
