Forum Moderators: open
Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.
At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.
The index-page was always in the top-3 position for our main topics, aka keywords.
I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index
As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.
My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.
Lets call them:
www.foo1.net and www.foo2.net
Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)
The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window
Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).
Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.
So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.
Fetched the page, but again got a code 200 with no redirects at all.
Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"
Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.
Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html
Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.
None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo
I have send an email to Google about this and am still waiting for a responds.
Here is what I got...
The link now ranks #5 for a good phrase. The link is...
[widget.com...]
This link belongs to a directory that allows you to login and edit your listing. When I found this link, I immediately logged in and deleted it from the offending directory (3 weeks ago). My thinking was that I would sit back and wait for the link to disappear and see if my site came back.
Today, not only did it not disappear, but it actually gained a few spots in the SERPs! When I run...
cache:http://www.widget.com/link.php?id=5932 it shows MY homepage. So what is the deal with that? And the date on the cached file is yesterday and the date on the cached page is the day before! (I date all my pages)
So now I am totally ticked. The original meta refresh is gone. It has now been replaced by a 302 redirect. It ranks where my site should be. The 302 goes to their home page but the cache shows my page. It seems that deleting my link from thier directory did not work.
So what the heck do I do? I suppose I could add my link back into their directory, that way I could at least get some google referrals again (all referrals from the G have now disappeared - gone - nada - zilch).
And to make matters even more irritating, I found another site doing the EXACT SAME THING! I also deleted that link which led to a meta refresh, which is now a 302, which now shows my home page as being cached, yet goes to their home page.
Meanwhile, I have 2 subpages that rank in the 500s. That's it. All other ranking for my site is gone except for these redirects. Two months ago. My site was ranking anywhere from 50 to 10 depending on the phrase. That's all gone but these other links still remain.
Still no response from google.
Would be interesting to see if in that they case they would still favor algorithmic handling (which won't help for a long time) or nuke the site manually
Thanks
The SERPS haven't changed, but I got hit by googlebot about 100 times today. Not a crawl, just the robots and home page.
Maybe this is the beginning of the end. I'll keep you informed.
By the way, no googlebot in weeks until today... we'll see.
Did you try the googlebot@google.com email address, yet? I read the article from Pandia, and I really don't think Google will tell anybody anything via email after that. They have to be aware of that article. If you search "Spammers hijack web site listings" the article is all over the place, now.
Your situation is almost identical to mine. Except that I couldn't manually remove my link, because it kept giving me an error-so I emailed the webmaster of that site to remove the link and they did. They also made a snide comment about banning my IP from submitting to their directory so I wouldn't have this problem again. I know it isn't the same directory, because they use a different linking structure than the example you posted.
After they removed the link, which actually meant the link now goes to a page on their site which then goes to a 404 error page...they did move up in the SERPS for the search terms I found them in.
Eventually, it did get dropped from those search results, altogether. It took more that a few days, though.
I didn't lose any other pages in Google, just the index page they had redirected to, which at the time was showing my page in the cache. My index page was completely out of Google for a time, but I can find it in there now. Also, during this whole thing Googlebot was only hitting my index page and not crawling further, but today I noticed they did crawl some other pages.
As I stated before, after all this, with the link removed from their directory--if you search on just my domain name, that link they were using now shows up instead of my domain name. Which it didn't do before!
Like you, I am wondering if this is because the link still goes to their site before going to the 404 error page.
>> If this problem's been around so long it's a shame someone didn't press Google on it pre-IPO (Macro #78)
Not exactly pressure, but at least mentioning the problem: What about those redirects, copies and mirrors? [webmasterworld.com] (May 3, 2004)
>> And after searching I found many threads, all with different variations of the same problem. (Webdude #7)
In the thread above, msg #8, you'll find a nice collection of 24 threads dating back to... june 16, 2003
[webmasterworld.com...]
It's simply a Google bug, albeit a very annoying one.
This resonates, because when I spoke with a Google search engineer about the meta refresh redirect problem at SES San Jose in August, 2003, he used the exact same words. "Hmmm, it sounds like a Google bug," he said. I know it's a big system, but that was over a year ago.
I found something in your collections of threads that I find interesting. In a thread titled:
"Not sure but I think it is Page Jacking" October 9th 2003 ( [webmasterworld.com...] )
In message 1 mille reports a page jacking
In message 2 you reply to Brett...
Obviously, at least one post was removed. You must have made a point to cause Brett to remove his own post!
... traveshamockery I say!
... screw them at their own game...what this entire post comes down to is greed
I beg to differ. A few members here had their pages screwed up (in at least one case, a hobby site) apparently by other sites exploiting what possibly turns out to be a "Google bug", and they wanted to find out why. Screwing people at their own game would be, in this case, like becoming a pickpocket just because one had one's own pocket picked.
I hope the members affected get their sites back to where they should be when the game is played fairly. In my case I'm pleased to say my php redirects are now returning a 301 (thanks to charlier and BigJay).
What this entire post comes down to is greed
I see "Search Engine Optimizer" is what you listed as your occupation in your profile. Bye!
quotations, I feel for you but I thought Google was serious about DMCA issues.
Apparently not serious enough.
They told me that they do not care what the content of the pages they index is and that they would do nothing about it.
