Forum Moderators: open
Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.
At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.
The index-page was always in the top-3 position for our main topics, aka keywords.
I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index
As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.
My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.
Lets call them:
www.foo1.net and www.foo2.net
Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)
The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window
Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).
Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.
So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.
Fetched the page, but again got a code 200 with no redirects at all.
Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"
Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.
Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html
Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.
None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo
I have send an email to Google about this and am still waiting for a responds.
The search engines need to fix this real soon ..
Webmaster@google.com has already responded to many of us and told us that it is working exactly as they want it to work and that we should shut up and leave them alone.
Yahoo responded to my note by banning my site from their index.
Have Dvorak, Cringely, Sullivan, or Bricklin written about this yet?
Apparently, you overlooked this plumsauce.This testing was all done within one domain.
guess i did, it's been a long thread to follow.
Of course, "reason" and "Google" are not synonymous so anything is worth trying I guess.
Kaled.
Why would changing the <base> tag have an effect?
The event that led to this idea is described in: [webmasterworld.com...]
To summarize quickly, Googlebot (possibly in a one-off glitch) clearly followed a 301 Location: redirect and left the base URI set to the domain that issued the redirect rather than the domain spelled out in the Location: header. A correct interpretation of the HTTP, HTML and RFC1808 standards? I'm skeptical, but of course, it's more interesting to know what's going to happen then to waste time trying to get Google to agree with my interpretation of what they should be doing.
IMO, it's entirely possible that Google believes these standards require it to assume that the cross-domain target of a redirect has the base URI of the source domain, hence leading it to "give" that content to the hijacker.
Why wonder whether Google might have decided to pay attention to the <base> element of the destination URL's HTML? Mainly because of 12.4.1 in the HTML 4.01 spec, which says the <base> element must be given priority over:
meta data discovered during a protocol interaction, such as an HTTP header
setting that <base> element is at least a handle that the hijackee can twist, so it seems worth a try.
I can't argue with that - if I were a victim, I'd try it.
Kaled.
Sorry, I have been out of the office. It seems some things are starting to happen.
webdude, are you seeing the same results I'm seeing? After you confirm my results, I'll try substituting a few pages at a different domain for these test pages.
Sorry DaveAtIFG. I am not at my usual spot with tools for testing, etc. I am looking at the cache according to the URLs from the search you sticky'd me and it shows the cache page as being the pages being redirected to. Is this what you are seeing?
Also, an update on my site.
Yippy! I think GoogleGuy or somebody finally noticed! The offending links are all gone and My site now is listed normally, home page and all. All links to the offending site are completely gone, zilch, nada, zero. My site is not ranked where it was, but I think giving it time will get it back to where it was.
Here is what changed for those interested...
keyword1 keyword2
Offending site was ranked 50
My site now 156
keyword1 keyword2 keyword3
Offending site was ranked 1
My site now 50 and 51
keyword1 keyword2 keyword4
Offending site was ranked 5
My site now 73 and 74
So it looks like maybe the squeeky wheel DOES get the grease. I'll be back in the office tomorrow and test some stuff out and get back. I haven't been able to check for any crawls and stuff. I don't have access to my log files here.
anywayz
I am so happy for you!
Hopefully they will get to me some day soon, too. I am not seeing any change for my site at this time.
Oddly, regarding the original site that hijacked me:
I've been watching another site that I noticed was hijacked by them. Yesterday...still showing up in Google SERPS as hijacked. Today, all signs of hijacking for that site gone from the SERPS, even though the offending site still has the 302 and meta redirect to that site in place.
Yet, my site is still showing up when searching for my domain name, even though they no longer actively link to me.
What the hell?
I have been in contact with the owner of the second site that hijacked me, and he has removed all meta refresh links from his site. (He told me it was built into the phplinks program he uses). However, he has changed them to 302 redirects.
Webdude, I am real happy you got yours sorted.
How about you, Marcello?! Any good news to report?
However, he has changed them to 302 redirects.
Maia, please note that I had a few problems with 302 redirects. I wrote earlier that sometimes my script page (not built to be viewed) was ranked where the target URL should have been. I've changed them to 301s so hopefully the problem will clear itself.
I'm very pleased to hear of apparent progress on the general issue of this thread, but hope these aren't just one-offs that are only being resolved by one-off human intervention.
To DaveAtIFG - Power to your elbow!
Good stuff, DaveAtIFG. I've been following this thread closely, in fear of my own site getting hijacked eventually... perhaps this thing is now in the process of getting shut-down.
GoogleGuy, if you were in on this, many thanks man. It's good to see that you're still onboard.
Maia, please note that I had a few problems with 302 redirects.
Oh, I am aware. I was trying to say he removed the meta refreshes, but then switched to another method that still causes problems.
I asked all the directories which used 302 redirected links to remove my listings. Halting link-buiding campaign to complete de-linking campaign. Yay!
I wrote earlier that sometimes my script page (not built to be viewed) was ranked where the target URL should have been.
He told me Google wasn't supposed to be indexing any of the results that were supposed to be just for click-tracking. I'm unclear if you meant the same thing with your statement.
I've changed them to 301s so hopefully the problem will clear itself.
You have integrity. Cheers, Patrick!
