Forum Moderators: open
Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.
At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.
The index-page was always in the top-3 position for our main topics, aka keywords.
I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index
As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.
My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.
Lets call them:
www.foo1.net and www.foo2.net
Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)
The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window
Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).
Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.
So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.
Fetched the page, but again got a code 200 with no redirects at all.
Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"
Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.
Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html
Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.
None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo
I have send an email to Google about this and am still waiting for a responds.
What got my attention was the attitude and integrity you displayed in #96 and #97.
I am not going to give the URL of the offending sites.
Sure, I could experiment and try to get competitors "kicked out of google." but I heven't, nor will I.I suspect it made an impression on GoogleGuy as well.
It remains to be seen whether "the algo is fixed." But I'm very pleased to hear that you're making some progress!
From #216:
Are you sick of me yet?
Can you confirm/verify this webdude?
My conclusion is that, effectively, the 302 and the meta refreshed pages are "page jacked."
To me, the next question is, "What SHOULD Google index when they encounter a 302 or a meta refresh?" Clearly, what they are indexing now facilitates page jacking, whether page jacking is intended or not. But what are the implications to the entire web by handling these differently? Perhaps they should all be handled just as a 301 is handled?
Are you getting sick of ME yet? :)
[webmasterworld.com...]
I find it interesting that googlebot is crawling differently according to some posts. Especially msg #104 and 106.
Could this be a fix in the algo? Funny how this is all happening at once. Funny also that php seems to be involved in a lot of this.
The 301 test lists the target page URL, title, snippet, and content in the cache.
The 302 test and the meta refresh test list the target page title, snippet and content in the cache, but links to and displays the redirecting pages' URLs.
Can you confirm/verify this webdude?
Yes, I can confirm this. It is showing the page in the cache but the URLs are incorrect for the 302 and meta refresh. Very interesting.
If you keep the pages there long enough for another crawl of the test pages, maybe we can find if the algo has truly changed. Will this change in the next week or 2? I don't know. I am showing the pages as being cached on Sep 24.
This is quite fascinating.
Re: [webmasterworld.com...]
In regards to Webdude's last post. Mr. Dave, if you want to experiment further you can put some redirects to a subdomain I don't use anymore and see what Google does with it. It's only partially indexed, though, so I don't know if that would make a difference.
What the heck would happen if I put a 302 from my subdomain to that link from my friends at StealYourIndexPagedot com?
Googlebot has been doing some major crawling of my site over the past few days, too. More than any other month in the short history of my site.
Another thing I noticed in that other thread was that people have mentioned their cached pages reverting back to September 12, or so. I swear when I checked the cache on my index page the other day it said September 26 at 2 AM, and now it is September 27 at 1 AM. I wonder if that means anything?
Having followed this thread and not being from a technical background, I have a question. From what I can see, there is (or was) a tendency for pages to be wrongly indexed in Google's SERPS when one site linked to another site's page using a meta-refresh or a scripted redirect (or a combination of both). Assuming this is (or was) a search engine bug, and also assuming the bug will be (or has been) fixed, will sites that continue to practice those redirects - perhaps quite innocently - receive some kind of Google penalty or will the solution obviate such a step?
I am seeing that for many of them no-more text snippet appears and that the cache has been deleted. Some other ones that are using a re-direct to a page within their same site are staying the same.
What to make out of this...... Only Google knows.
Future will tell
Why should you be penalized for a legitimate redirect? After all, this forum uses it too.
