homepage Welcome to WebmasterWorld Guest from 54.205.189.156
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 5 6 7 8 [9] 10 11 12 13 > >     
Dupe content checker - 302's - Page Jacking - Meta Refreshes
You make the call.
Marcello




msg:59535
 11:35 am on Sep 7, 2004 (gmt 0)

My site, lets call it: www.widget.com, has been in Google for over 5-years, steadily growing year by year to about 85,000 pages including forums and articles achieved, with a PageRank of 6 and 8287 backlinks in Google, No spam, No funny stuff, No special SEO techniques nothing.

Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.

At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.

The index-page was always in the top-3 position for our main topics, aka keywords.

I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index

As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.

My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.

Lets call them:
www.foo1.net and www.foo2.net

Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)

The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window

Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).

Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.

So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.

Fetched the page, but again got a code 200 with no redirects at all.

Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"

Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.

Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html

Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.

None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo

I have send an email to Google about this and am still waiting for a responds.

 

quotations




msg:59775
 4:03 pm on Sep 26, 2004 (gmt 0)

The search engines need to fix this real soon ..

Webmaster@google.com has already responded to many of us and told us that it is working exactly as they want it to work and that we should shut up and leave them alone.

Yahoo responded to my note by banning my site from their index.

Have Dvorak, Cringely, Sullivan, or Bricklin written about this yet?

DaveAtIFG




msg:59776
 4:53 pm on Sep 26, 2004 (gmt 0)

GoogleGuy has asked to review my test data and I have provided it to him. The promptness of his reply suggests to me that he is concerned about about this issue.

Maia




msg:59777
 5:27 pm on Sep 26, 2004 (gmt 0)

If he needs more data, I sent him some via stickymail a few weeks ago.

gemini




msg:59778
 3:01 am on Sep 27, 2004 (gmt 0)

I noticed that Google rolled back on Sept. 5th cache for the offenders sites and reindexed the offended ones as of yesterday and today again (Sept 26). Also, some cached pages with the 302 redirects has been removed and the original pages too. Something is happening... is anyone seeing the same?

Maia




msg:59779
 4:03 am on Sep 27, 2004 (gmt 0)

Nope. Their URL, my cache dated September 21.

plumsauce




msg:59780
 7:10 am on Sep 27, 2004 (gmt 0)

DaveAtIFG,

Apparently, you overlooked this plumsauce.
This testing was all done within one domain.

guess i did, it's been a long thread to follow.

ronburk




msg:59781
 7:43 am on Sep 27, 2004 (gmt 0)

Is there anybody out there with hijacked pages who can a) add an HTML-compliant <base> element to a hijacked page and change a bit of text, b) wait for the next Googlebot visit to see it and for the tweaked text to appear in the Google index, and c) report back whether that happened to "turn off" the hijacking of that page?

kaled




msg:59782
 9:11 am on Sep 27, 2004 (gmt 0)

Why would changing the <base> tag have an effect? Robots will read this tag for the purpose of following links, but there is no reason for this tag to play any further part in search engine algos.

Of course, "reason" and "Google" are not synonymous so anything is worth trying I guess.

Kaled.

ronburk




msg:59783
 5:18 pm on Sep 27, 2004 (gmt 0)

Why would changing the <base> tag have an effect?

The event that led to this idea is described in: [webmasterworld.com...]

To summarize quickly, Googlebot (possibly in a one-off glitch) clearly followed a 301 Location: redirect and left the base URI set to the domain that issued the redirect rather than the domain spelled out in the Location: header. A correct interpretation of the HTTP, HTML and RFC1808 standards? I'm skeptical, but of course, it's more interesting to know what's going to happen then to waste time trying to get Google to agree with my interpretation of what they should be doing.

IMO, it's entirely possible that Google believes these standards require it to assume that the cross-domain target of a redirect has the base URI of the source domain, hence leading it to "give" that content to the hijacker.

Why wonder whether Google might have decided to pay attention to the <base> element of the destination URL's HTML? Mainly because of 12.4.1 in the HTML 4.01 spec, which says the <base> element must be given priority over:
meta data discovered during a protocol interaction, such as an HTTP header

Now, my reading of the specs would not necessarily lead me to think that setting the <base> element in the target URL's HTML should affect this -- but hey, I already disagree with how Google's interpreting the specs, and setting that <base> element is at least a handle that the hijackee can twist, so it seems worth a try. Also, I suspect that a vanishingly small number of HTML pages contain a <base> element, so it's possible that no one has tested this possibility (at least consciously).

kaled




msg:59784
 5:49 pm on Sep 27, 2004 (gmt 0)

setting that <base> element is at least a handle that the hijackee can twist, so it seems worth a try.

I can't argue with that - if I were a victim, I'd try it.

Kaled.

webdude




msg:59785
 12:07 am on Sep 28, 2004 (gmt 0)

An Update.

Sorry, I have been out of the office. It seems some things are starting to happen.

webdude, are you seeing the same results I'm seeing? After you confirm my results, I'll try substituting a few pages at a different domain for these test pages.

Sorry DaveAtIFG. I am not at my usual spot with tools for testing, etc. I am looking at the cache according to the URLs from the search you sticky'd me and it shows the cache page as being the pages being redirected to. Is this what you are seeing?

Also, an update on my site.

Yippy! I think GoogleGuy or somebody finally noticed! The offending links are all gone and My site now is listed normally, home page and all. All links to the offending site are completely gone, zilch, nada, zero. My site is not ranked where it was, but I think giving it time will get it back to where it was.

Here is what changed for those interested...

keyword1 keyword2
Offending site was ranked 50
My site now 156

keyword1 keyword2 keyword3
Offending site was ranked 1
My site now 50 and 51

keyword1 keyword2 keyword4
Offending site was ranked 5
My site now 73 and 74

So it looks like maybe the squeeky wheel DOES get the grease. I'll be back in the office tomorrow and test some stuff out and get back. I haven't been able to check for any crawls and stuff. I don't have access to my log files here.

anywayz

Maia




msg:59786
 12:37 am on Sep 28, 2004 (gmt 0)

Webdude!

I am so happy for you!

Hopefully they will get to me some day soon, too. I am not seeing any change for my site at this time.

Oddly, regarding the original site that hijacked me:

I've been watching another site that I noticed was hijacked by them. Yesterday...still showing up in Google SERPS as hijacked. Today, all signs of hijacking for that site gone from the SERPS, even though the offending site still has the 302 and meta redirect to that site in place.

Yet, my site is still showing up when searching for my domain name, even though they no longer actively link to me.

What the hell?

I have been in contact with the owner of the second site that hijacked me, and he has removed all meta refresh links from his site. (He told me it was built into the phplinks program he uses). However, he has changed them to 302 redirects.

Webdude, I am real happy you got yours sorted.

How about you, Marcello?! Any good news to report?

Patrick Taylor




msg:59787
 1:06 am on Sep 28, 2004 (gmt 0)

However, he has changed them to 302 redirects.

Maia, please note that I had a few problems with 302 redirects. I wrote earlier that sometimes my script page (not built to be viewed) was ranked where the target URL should have been. I've changed them to 301s so hopefully the problem will clear itself.

I'm very pleased to hear of apparent progress on the general issue of this thread, but hope these aren't just one-offs that are only being resolved by one-off human intervention.

To DaveAtIFG - Power to your elbow!

Stefan




msg:59788
 1:38 am on Sep 28, 2004 (gmt 0)

Congrats Webdude. I hope it sticks.

Good stuff, DaveAtIFG. I've been following this thread closely, in fear of my own site getting hijacked eventually... perhaps this thing is now in the process of getting shut-down.

GoogleGuy, if you were in on this, many thanks man. It's good to see that you're still onboard.

Maia




msg:59789
 1:47 am on Sep 28, 2004 (gmt 0)

Maia, please note that I had a few problems with 302 redirects.

Oh, I am aware. I was trying to say he removed the meta refreshes, but then switched to another method that still causes problems.

I asked all the directories which used 302 redirected links to remove my listings. Halting link-buiding campaign to complete de-linking campaign. Yay!

I wrote earlier that sometimes my script page (not built to be viewed) was ranked where the target URL should have been.

He told me Google wasn't supposed to be indexing any of the results that were supposed to be just for click-tracking. I'm unclear if you meant the same thing with your statement.

I've changed them to 301s so hopefully the problem will clear itself.

You have integrity. Cheers, Patrick!

Marcello




msg:59790
 2:26 am on Sep 28, 2004 (gmt 0)

Yes.....Yes
My Index-Page is back!
Position #1 for my main search-term
(showing a cache of 25 Sept.)

"foo.com/www-widget-com.html" is nowhere to be found in the first 500 results (didnt look further) for the above mentioned search-term.

link:www.widget.com is still showing:
"Your search - link:www.widget.com - did not match any documents"

but I think this has to do with the fact that google has many different databases for different kind of searches (thats what I hope)

"foo.com/www-widget-com.html" is still there (without a cache) but has gone completely to the bottom of the SERPS somewhere and as foo's page has been deleted, it will eventualy disappear when it gets the visit of GoogleBot.

I have also noticed that some other sites which I was watching and also had been hijacked by a 302 (not my sites) by other foo's, have all had their Cache deleted and that the original pages are back in the SERPS.

A big thanks to all people that have contributed and helped in this topic.

5x54u




msg:59791
 2:57 am on Sep 28, 2004 (gmt 0)

Persistence paid off for you. Nice work!

M

dirkz




msg:59792
 6:54 am on Sep 28, 2004 (gmt 0)

I'd like to know whether this is now fixed in the algo, or some sites/pages were nuked.

Macro




msg:59793
 8:18 am on Sep 28, 2004 (gmt 0)

Marcello, WELL DONE!

webdude




msg:59794
 3:37 pm on Sep 28, 2004 (gmt 0)

More details for those who have stuck with this thread so far.

DaveAtIFG -

I checked the cache of your 3 test pages you set up for the test. All 3 pages show the following for the cache --

302 Redirect Test Page
"The Pricing.html...

Meta Refresh Test Page
"Order.html now...

301 Redirect Test Page
"The License.html ...

Hope this helps!

Now, for some news on my site.

I checked my log files and I am finding some very interesting stuff. First of all, on Saturday, I got some hits from WebMasterWorld checking cache and header info on my home page. This happened Saturday evening. And this is what I find interesting, about an hour later, I started to get crawled by googlebot. In fact, it is now Tuesday and googlebot is still crawling. Not a very fast crawl, but a very deep crawl. The most extensive crawl I have had to date. It is even feasting on all my forum pages of which I have many. Coincidence? I am not sure. But this is what is happening.

Another thing that I find interesting is that I still can find the link that has my cache when I search google for [widget.com...] but I cannot find it in the results for any of my keywords. And what is even more interesting is when I search for this link, the link displayed in google now goes directly to my site. No third part refresh or redirects. So the link is there, it shows my cache, and it now goes directly to my site but it cannot be found in the results unless you search for the link specifically --- ummm ---- interesting.

I went to the offending site and randomly checked the cache of some of it's links. The cache for these links STILL show the actual website just the way mine did. So I decided to search the actual links in google and guess what! They all now go directly to the websites. So either something radical has happened in the algo on how these redirects are being handled, or the information that I gave google caused them to hand edit some of this stuff. The latter, I highly doubt.

So my conclusion, and I STILL may be wrong about this, is that it was a googlebug. It now appears to be fixed from my point of view, but we won't know if it is an algo change unless we start getting more posts from some of the others that have had this problem.

Marcello - Glad you got your site back.
Maia - Hang in there.

Now for some gushing!

I would like to thank a lot of people on the site that helped me with this. Especially Maia, plumsauce, dirkz Brett and a really big thank you to DaveAtIFG! DaveAtIFG devoted his valuable time and set up some tests on one of his sites to try to get to the bottom of this. He also got the ear of GoogleGuy and I think that ultimately helped quite a bit. My hat's off to you DaveAtIFG!

And of course a thank you to WebMasterWorld and Brett, with his chainsaw, who let this thread get out of control, but let it hang so we could get to the bottom of this. I believe that WebMasterWorld is the most valuable tool for anyone who is into SEO. Granted, it's huge and you have to take the good with the bad. But there are some very knowledgable people here who are willing to help if you find yourself in a pickle. I just found that out first hand and IMHO, the majority of the users on these forums are some of the "Best OF The Web!"

WebMasterWorld truly does have the eyes of google.

WebDude

DaveAtIFG




msg:59795
 4:06 pm on Sep 28, 2004 (gmt 0)

I sure hope you're done gushing webdude! :)

What got my attention was the attitude and integrity you displayed in #96 and #97.
I am not going to give the URL of the offending sites.
Sure, I could experiment and try to get competitors "kicked out of google." but I heven't, nor will I.
I suspect it made an impression on GoogleGuy as well.

It remains to be seen whether "the algo is fixed." But I'm very pleased to hear that you're making some progress!

From #216:
Are you sick of me yet?

I'm getting there!;)

DaveAtIFG




msg:59796
 5:39 pm on Sep 28, 2004 (gmt 0)

Seriously, to summarize/clarify, what I see when examining Google's listing of my test pages:
The 301 test lists the target page URL, title, snippet, and content in the cache.
The 302 test and the meta refresh test list the target page title, snippet and content in the cache, but links to and displays the redirecting pages' URLs.

Can you confirm/verify this webdude?

My conclusion is that, effectively, the 302 and the meta refreshed pages are "page jacked."

To me, the next question is, "What SHOULD Google index when they encounter a 302 or a meta refresh?" Clearly, what they are indexing now facilitates page jacking, whether page jacking is intended or not. But what are the implications to the entire web by handling these differently? Perhaps they should all be handled just as a 301 is handled?

Are you getting sick of ME yet? :)

webdude




msg:59797
 6:45 pm on Sep 28, 2004 (gmt 0)

Hey, are you guys keeping up with the Gbot running hard thread located at...

[webmasterworld.com...]

I find it interesting that googlebot is crawling differently according to some posts. Especially msg #104 and 106.

Could this be a fix in the algo? Funny how this is all happening at once. Funny also that php seems to be involved in a lot of this.

idoc




msg:59798
 7:25 pm on Sep 28, 2004 (gmt 0)

I frequently find alot of these posts relate together from different perspectives. Also, I agree, and am hopeful that the "fix is in". I had an idea google was on it.

webdude




msg:59799
 8:06 pm on Sep 28, 2004 (gmt 0)

DaveAtIFG

The 301 test lists the target page URL, title, snippet, and content in the cache.
The 302 test and the meta refresh test list the target page title, snippet and content in the cache, but links to and displays the redirecting pages' URLs.
Can you confirm/verify this webdude?

Yes, I can confirm this. It is showing the page in the cache but the URLs are incorrect for the 302 and meta refresh. Very interesting.

If you keep the pages there long enough for another crawl of the test pages, maybe we can find if the algo has truly changed. Will this change in the next week or 2? I don't know. I am showing the pages as being cached on Sep 24.

This is quite fascinating.

DaveAtIFG




msg:59800
 8:35 pm on Sep 28, 2004 (gmt 0)

#104 and #106 are very interesting...

Re: [webmasterworld.com...]

Maia




msg:59801
 9:24 pm on Sep 28, 2004 (gmt 0)

It is very interesting. I don't think I understand what exactly Googlebot is trying to do there, since two different sites can have the same file names-such as the index file.

In regards to Webdude's last post. Mr. Dave, if you want to experiment further you can put some redirects to a subdomain I don't use anymore and see what Google does with it. It's only partially indexed, though, so I don't know if that would make a difference.

What the heck would happen if I put a 302 from my subdomain to that link from my friends at StealYourIndexPagedot com?

Googlebot has been doing some major crawling of my site over the past few days, too. More than any other month in the short history of my site.

Another thing I noticed in that other thread was that people have mentioned their cached pages reverting back to September 12, or so. I swear when I checked the cache on my index page the other day it said September 26 at 2 AM, and now it is September 27 at 1 AM. I wonder if that means anything?

Patrick Taylor




msg:59802
 1:56 am on Sep 29, 2004 (gmt 0)

Confirms WebmasterWorld Google News as an excellent and influential forum, as well as the fact that persistence sometimes pays off.

Having followed this thread and not being from a technical background, I have a question. From what I can see, there is (or was) a tendency for pages to be wrongly indexed in Google's SERPS when one site linked to another site's page using a meta-refresh or a scripted redirect (or a combination of both). Assuming this is (or was) a search engine bug, and also assuming the bug will be (or has been) fixed, will sites that continue to practice those redirects - perhaps quite innocently - receive some kind of Google penalty or will the solution obviate such a step?

Marcello




msg:59803
 8:04 am on Sep 29, 2004 (gmt 0)

I am still following some sites that are using the 301 and 302 technique to redirect to other sites, some directories and some "our links" styles. Those are not hijackers.

I am seeing that for many of them no-more text snippet appears and that the cache has been deleted. Some other ones that are using a re-direct to a page within their same site are staying the same.

What to make out of this...... Only Google knows.
Future will tell

dirkz




msg:59804
 9:37 am on Sep 29, 2004 (gmt 0)

> Assuming this is (or was) a search engine bug, and also assuming the bug will be (or has been) fixed, will sites that continue to practice those redirects - perhaps quite innocently - receive some kind of Google penalty or will the solution obviate such a step?

Why should you be penalized for a legitimate redirect? After all, this forum uses it too.

kaled




msg:59805
 10:26 am on Sep 29, 2004 (gmt 0)

Fingers crossed, the problem is being resolved. However, before everyone congratulates Google, bear in mind that no admissions, apologies or statements have been forthcoming. This is a very bad sign - a glimpse of the future perhaps.

Kaled.

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 5 6 7 8 [9] 10 11 12 13 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved