homepage Welcome to WebmasterWorld Guest from 54.196.168.78
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 5 6 7 8 9 [10] 11 12 13 > >     
Dupe content checker - 302's - Page Jacking - Meta Refreshes
You make the call.
Marcello




msg:59535
 11:35 am on Sep 7, 2004 (gmt 0)

My site, lets call it: www.widget.com, has been in Google for over 5-years, steadily growing year by year to about 85,000 pages including forums and articles achieved, with a PageRank of 6 and 8287 backlinks in Google, No spam, No funny stuff, No special SEO techniques nothing.

Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.

At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.

The index-page was always in the top-3 position for our main topics, aka keywords.

I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index

As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.

My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.

Lets call them:
www.foo1.net and www.foo2.net

Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)

The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window

Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).

Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.

So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.

Fetched the page, but again got a code 200 with no redirects at all.

Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"

Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.

Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html

Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.

None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo

I have send an email to Google about this and am still waiting for a responds.

 

kaled




msg:59805
 10:26 am on Sep 29, 2004 (gmt 0)

Fingers crossed, the problem is being resolved. However, before everyone congratulates Google, bear in mind that no admissions, apologies or statements have been forthcoming. This is a very bad sign - a glimpse of the future perhaps.

Kaled.

webdude




msg:59806
 2:23 pm on Sep 29, 2004 (gmt 0)


Fingers crossed, the problem is being resolved. However, before everyone congratulates Google, bear in mind that no admissions, apologies or statements have been forthcoming. This is a very bad sign - a glimpse of the future perhaps.

kaled,

You are absolutely correct. I sent several emails of this problem to several of the goole emails and I didn't even get a response. Don'r get me wrong/ I am glad that whatever was wrong seems to be getting fixed. But...

To date. No response. Not even an acknowledgement.

Patrick Taylor




msg:59807
 3:01 pm on Sep 29, 2004 (gmt 0)

a glimpse of the future perhaps

Hopefully not. Google may be a fine search engine but is currently in a commanding position as far as the web goes. Anyone whose future relies on the continuing success and progress of the web should hope that the circumstances that led to this thread become few and far between.

For average webmasters who face the prospect of investing their limited resources into a good honest website - whether it be an online business or a site whose presence simply enriches the web experience - the prospect of an unexplained and/or unrecoverable disaster (one of those commonplace and sudden disappearances from Google's SERPS for no apparent reason, for example) is currently a disincentive - and ultimately bad for everyone. The disincentive would be much reduced if Google and the other major search engines would make themselves more openly accessible to webmasters, even if it means paying a review fee or some such arrangement instead of having to resort to a forum like this. I don't mean moans about sites that drop a few places here and there, but sites that are hit by a bug or whatever one wants to call it. A situation in which one can get no intelligible response (to a genuine problem) from the "world's greatest search engine" simply isn't good enough and will become even less good enough if it holds back the web maturing itself out of the "Wild West" ethos it often seems to characterise. Or is it a former eastern block kind of thing, where the controlling forces are depressingly unaccountable and impenetrable?

Of course it makes interesting and informative reading, and in some cases a fair end result ensues, but steadfastedly pursuing such an issue in a forum thread shouldn't be the method people have to resort to - where sometimes a case is taken up (great!) and other times not.

DaveAtIFG




msg:59808
 4:30 pm on Sep 29, 2004 (gmt 0)

no admissions, apologies or statements have been forthcoming
I doubt there will be any. If Google acknowledges any shortcoming, someone, somewhere, will file a lawsuit. "I lost thousands of dollars because of your flawed search engine!"

"Why are you suing for millions?"

"Punitive damages! They will never do this again, to anyone!" (Meaning: I didn't do this to get rich but to help my fellow webmasters and teach Google a lesson. Now my conscience won't bother me at all as I enjoy my undeserved windfall. ;) )

[google.com...]

a glimpse of the future perhaps
A glimpse of the present I think. :)

kaled




msg:59809
 4:36 pm on Sep 29, 2004 (gmt 0)

About ten years ago, I had to have two wisdom teeth removed. During this, my dentist slipped and cut the side of my mouth. Her reaction was simple
1) She apologised.
2) She put right her mistake (with a couple of stitches).
3) She got on with the job.

Call me idealistic, but if everyone behaved the same way, including big corporations, the world would be a better place.

Kaled.

g1smd




msg:59810
 8:55 pm on Sep 29, 2004 (gmt 0)

>> I swear when I checked the cache on my index page the other day it said September 26 at 2 AM, and now it is September 27 at 1 AM. I wonder if that means anything? <<

One of my sites has been crawled at 01:05 to 01:08 GMT every day since September 8th, but moved an hour earlier, now crawled at 00:03 to 00:08 for the last 3 or 4 days.

Bluepixel




msg:59811
 7:31 pm on Sep 30, 2004 (gmt 0)

What's strange is that I see that phenomen on MSN, Yahoo and google for my site. Makes me think if they all use the same code... :-)

gemini




msg:59812
 8:12 pm on Sep 30, 2004 (gmt 0)

Doesn't look like it ended to me...

I changed my links to static pages on Sept. 9th - everything got reindexed since than. A few days ago cached 302 redirects were rolled back to Sept.5th cache (I belive). I added a line in my robots.txt Disallow: *.php and changed 302 redirect to 301 Permanent even though I didn't have the actual links on my pages anymore - only direct static ones. Today I see in my logs someone was looking at cached by google redirect pages - I checked and YES they are recrawled on Sept.27 and 28th :( What the #()@k!?

I didn't write the code for the directory. Here is the redirection code:
[PHP]
{
header("HTTP/1.1 301 Moved Permanently");
header("Location: $url");
//die('<meta http-equiv="refresh" content="0;url='. $url .'">');
}
[/PHP]

as you can see I changed it from 302 to 301 and than I went back and removed the metarefresh line since the pages were re-cached. I hope it will solve the problem...

Suggestions are appreciated.

gemini

ashdar




msg:59813
 6:09 am on Oct 2, 2004 (gmt 0)

I'm having the same 'page jacking' 302 re-direction from a directory listing.

Our site was listed on the directory on sept 27-28. Right after I noticed our main domain page dissapear from google. The page is now cached under their link dated sept-28.

Here is the server header check results:

Server Response: [directorySite.com...]
Status: HTTP/1.1 302 Moved Temporarily
Server: Zeus/4.1
Date: Sat, 02 Oct 2004 05:52:16 GMT
Connection: close
Set-Cookie: hits=++2887+; expires=Sat, 02-Oct-04 05:53:17 GMT
Location: [mysite.com...]
P3p: CP="CAO DSP COR CURa ADMa DEVa OUR IND PHY ONL UNI COM NAV INT DEM PRE" policyref="www.somesite.com/w3c/p3p.xml"
Content-Type: text/html
X-Powered-By: PHP/4.3.1

Out of the 10 newest listing on this directory 2/10 had this problem, even though it seems all ~2000 links in the directory (I randomly checked two dozen) are setup in this method of 302 re-directions.

Any suggestions what I should do? Should I contact this site and ask to be removed or wait it out?

Maia




msg:59814
 8:22 am on Oct 2, 2004 (gmt 0)

Well, it looks like we can rule out:

One-The problem was fixed.
Two-Apache 1.3x.

Ashdar, if your site is the one I think it is, the site linking to you is also using a meta refresh.

I had two separate directories hijack me recently. I had the first site remove my listing from the directory on September 2.

Both my index page and the page the site had the link on have been recrawled and have newly cached pages, however Google still returns that link the site used to have to me when searching for my domain name instead of returning my own URL.

The other directory removed the meta refreshes and ultimately changed his directory to 301s instead of 302s, but it doesn't look like Google has recrawled that page of his directory yet, so I still have hijacked results all over in Google.

I don't know what advice to give you.

I'll tell you that if this happens to me again, I'll try filing a DMCA complaint. Some people have had success with that.

Good luck.

PS-Can I put a link from my site to the page containing the Second Directory's link to me to try to send Googlebot over there faster? Or would that be a bad move in Google's eyes?

ashdar




msg:59815
 6:30 pm on Oct 2, 2004 (gmt 0)

I'm not rushing to do anything yet but considering all options. I have found some things and learned a lot.

Here is something that might be of help (thanks to Vin DSL). If you can edit your .htaccess file, you can add the following code (only for unix based servers).

RewriteEngine on
php_flag display_errors off
php_flag register_globals off
RewriteCond %{HTTP_REFERER} ^http://(www\.)?offendingSite.com/.*$
RewriteCond %{REQUEST_URI} ^/.*$
RewriteRule ^.* - [F]

Replace "offendingSite.com" with the correct domain.

What this does is, any clicks from the site itself should get an error (403 forbidden) but someone searching on google will still be able to use the offending link to find your site since the referrer is google.

The idea is next time google crawls the offending site it'll get a 403 error and won't be able to cache your page.

I'm not sure if this will actually work on googlebot but it should if it checks like a browser. Perhaps some of the senior members who have experience with crawler bots can confirm this.

dirkz




msg:59816
 7:08 pm on Oct 2, 2004 (gmt 0)

> The idea is next time google crawls the offending site it'll get a 403 error and won't be able to cache your page.

Googlebot doesn't deliver referers ...

ashdar




msg:59817
 8:13 pm on Oct 2, 2004 (gmt 0)

I'm not sure what you mean dirkz (sorry I'm confused). Are you saying it should work?

What's happening with this code write now is: following the link from the directory is resulting in this standard error:

Forbidden
You don't have permission to access / on this server.
--------------------------------------------------------------------------------
Apache/1.3.31 Server at www.mysite.com Port 80

but through google (if you do a search) the link works.

kaled




msg:59818
 11:58 pm on Oct 2, 2004 (gmt 0)

I think the point being made by dirkz is that googlebot does not deliver referrer data, so that the HTTP_REFERER var will always be null when the request is from Googlebot.

In addition, robots do not follow links directly as users do, they record them and visit them later, therefore no referrer data could sensibly exist. Whilst it is possible that some redirects are followed directly (unlike regular links) there would still be no referrer data.

This does raise an interesting point. Previously I have been almost certain that the fault lies in the indexing service, however, if googlebot is following redirects as browsers do (rather than treating them as links) then this might explain a great deal - in fact, it might explain everything.

I really hate being wrong, but I think my analysis earlier in this thread may have been.

Kaled.

ashdar




msg:59819
 7:11 am on Oct 3, 2004 (gmt 0)

Thank you for the explanation kaled. I see what he meant now.

An update, earlier my index page started appearing once again in google's cache (dated oct1). But the old hijacked link is still cached and ranks as #1 for my keyphrase where as the real link isn't even in the top 100.

Going to wait and see if things improve but if not I think I'll write to them to fix this.

Ambient




msg:59820
 1:26 pm on Oct 3, 2004 (gmt 0)

The same problem happened to me. A competitor decided to abandon his site and give up his business, and redirected to my site using a 302. Sure enough, my index page was gone from all Google rankings. Still waiting (Google says it can take 6 weeks, relax). The subpages all rank fine.

The problem is obviously a 302 redirect from a lesser site. His site is now rated #5 in a category I have always rated #1.

Maia




msg:59821
 1:59 pm on Oct 3, 2004 (gmt 0)

Sure, Google said relax. When I got my index page back, I had lost my Page Rank. It's still PR0.

I have a change. That freaking other URL has finally disappeared when I do a search on my domain name. However, Google now gives my [my-site].com URL instead of the www.[my-site].com URL.

I don't really understand that, and I have 301 redirects from the non-www pages to the www-pages.

However, the other directory that hijacked me is still in there. So, who knows if they will take over where the other place used to be when searching for my domain name, soon.

I still wonder if I should put a link to that page from mine, but nobody has answered my question, and I don't want to make things worse.

Anyway, that's where I'm at now.

ashdar




msg:59822
 4:36 pm on Oct 3, 2004 (gmt 0)

Maia Is your domain cached by google? I had the same thing initially google only had my domain.com without the www. But this wasn't cached and couldn't be searched with keywords.

When www.mydomain.com re-appeared in google yesterday only then it had a cache (2 days old) of the page and could be searched.

I haven't asked the offending site to remove their link. I'm curious to see if google will fix itself (which it slowly is from the looks of it). I don't think you should put a link to that site though, I don't think it's helpful.

I think the fastest way to get the offending page removed would be to have them point it to a blank page on your or any page on their site. So next time google crawls it'll replace the old cache and the offending page would drop very fast in rankings. I think this is the best solution.

Maia




msg:59823
 9:10 pm on Oct 3, 2004 (gmt 0)

I think the fastest way to get the offending page removed would be to have them point it to a blank page on your or any page on their site. So next time google crawls it'll replace the old cache and the offending page would drop very fast in rankings. I think this is the best solution.

Ashdar, that site used to have a meta refresh and a 302 redirect to my site. They have dropped the meta refresh and changed the link to a 301 redirect, which should be fine, but Google hasn't crawled that page of its directory, yet.

My site is cached, and my index page with the www is in Google, I was only referring to what appears when you search wwww.[my-site].com. That situation had to do with a different directory. My site was hijacked by two different sites simultaneously.

gemini




msg:59824
 1:44 pm on Oct 5, 2004 (gmt 0)

Okay, I'm kinda getting lost here..

I switched to redirect 301 and got rid of metarefresh last week. In addition to that I don't allow to crawl any php files, but I see new cache of offended pages every day, although links don't go to those pages - they go to my internal pages. What is happening? Any help would be appreciated.

Thanks

ashdar




msg:59825
 8:06 pm on Oct 5, 2004 (gmt 0)

Maybe your attempt to dis-allow php is blocking the robot from seeing the 301 re-direct and it's caching whatever it's seeing instead.

I'm not sure why you are trying to dis-allow php pages.

gemini




msg:59826
 8:38 pm on Oct 5, 2004 (gmt 0)

well, it may be not logically, but I was trying to get rid of php pages in the serps.

ashdar




msg:59827
 7:54 am on Oct 6, 2004 (gmt 0)

I think it would take longer for 'broken' links to get removed from google's cache than re-directing it with a 301 to a page or real destination.

But if you want to remove it I think the fastest way is described here: [google.ca...]

gemini




msg:59828
 1:02 pm on Oct 6, 2004 (gmt 0)

I think I figured it out and I guess I was false alarming. Actually there are no snipets and no cache for the links in the serps when I do [site:mysite.com inurl:link.php?id=], but when I search for the actual mysite.com/link.php?id=123 it gives me snipet of the url I redirect to with the correct url and it's cache. So, I guess as long as there is no actual cache for the link in the serps and other site's snipet we are fine.

tosho




msg:59829
 11:41 pm on Oct 6, 2004 (gmt 0)

In addition, robots do not follow links directly as users do, they record them and visit them later, therefore no referrer data could sensibly exist. Whilst it is possible that some redirects are followed directly (unlike regular links) there would still be no referrer data.

This does raise an interesting point. Previously I have been almost certain that the fault lies in the indexing service, however, if googlebot is following redirects as browsers do (rather than treating them as links) then this might explain a great deal - in fact, it might explain everything.

Kaled, looks like they do follow redirects right away. this is from my access log:

66.249.64.170 - - [06/Oct/2004:00:02:33 -0400] "GET / HTTP/1.0" 301 0 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66.249.64.170 - - [06/Oct/2004:00:02:34 -0400] "GET /index.php3 HTTP/1.0" 200 2142 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

Works as intended in my code, if / is requested, a 301 is pointed to /index.php3, but look at the times of the consecutive requests. same IP, a second later, looks like redirect following to me. No referers though.

g1smd




msg:59830
 11:49 pm on Oct 6, 2004 (gmt 0)

They do also follow new links right away sometimes too.

The cache date for one page of a site is a few seconds before the cache date for another page on a different site that has been linked to from the first mentioned page.

That time difference was noticed the very first time that Google followed that new link, about a week ago. Both pages had been online for many months, and both hadn't been re-cached for many weeks.

tosho




msg:59831
 12:21 am on Oct 7, 2004 (gmt 0)

sorry for the off topic, but, yeah, they are fast these days. the toolbar is a new content source too. today, I tested a brand new page on my site with ie and toolbar on. No links to the page, no one knows about it. 5 seconds later the page got a googlebot visit. big brother on steroids :)

kaled




msg:59832
 9:39 am on Oct 8, 2004 (gmt 0)

Now all we need is logs proving that Googlebot behaves the same way with redirects from one domain to another.

Here's a logic suggestion for Google.
1) Treat all redirects from one domain to another as permanent.
2) Always index the target of a permanent redirect.

There are legitimate reasons why temporary redirects should be allowed across domains, and a means of enabling this (at the temporary home) should be provided. This would probably mean introducing a new HTTP header response along the lines of

Temporary-Home-Of: permanent-domain.com
or introducing a new standard file called redirects.txt to hold such information.

Kaled.

Marcello




msg:59833
 9:27 am on Oct 10, 2004 (gmt 0)

I thought the 302 problem was being solved...
BUT I AM WRONG!

As the redirecting page at "foo.com" had been deleted and my Index-Page at my site started to appear again, I thought that Google had solved the problem of the "Duplicate Content" and "Page Jacking" created by 302 redirects and MetaRefresh tags.

But today 4 new pages appeared in the Serps, each using the MetaTag:
<meta http-equiv="refresh" content="0; url=http://www.widget.com/">

The Cache of the above 4 sites is simply my Index-Page.
They have not taken-over my position in the SERPS,
and can currently only be seen by clicking:
"repeat the search with the omitted results included"

My site has not recuperated 100% yet from the first 302-hijack,
I still dont get any data for "link:widget.com" and for "Similar Pages"

Even I have, when searching in Yahoo, 89,000 backlinks of which many PR4 up to PR6

The new foo's this time are Directory-style sites that use some commercial link-portal script to build their directory
( I do not think that the hijacking is done intentionaly by those sites)

The New Hijacking pages are in the format of:
www.foo1.com/links.php?action=link_id=111
www.foo2.com/redir.asp?link=222
www.foo3.info/get_url.asp?SiteID=333
www.foo4.com/links/click.php?id=444

The 4 above pages show my Index-Page in the SERPS with a Cache-Date of 6-October

So everything is starting over again......
Hopeless....Hopeless...Hopeless!

div01




msg:59834
 8:20 pm on Nov 7, 2004 (gmt 0)

I am also seeing a 'via.php' used in a similar similar fashion as the 'tracker2.php' script.

Spine




msg:59835
 7:16 pm on Nov 8, 2004 (gmt 0)

This might be only semi related.

I had my .htaccess set up so that in case of 404, 403 errors etc the user would be directed to my index page instead of the default 'file not found' page.

I noticed recently when doing a site: check that pages I had removed were showing up, but with the title and snippet from my index page.

It seems google was taking the content on the index page as belonging to the non-existant file.

I changed my .htaccess so that a custom 404 page comes up.

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 5 6 7 8 9 [10] 11 12 13 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved