Welcome to WebmasterWorld Guest from 54.198.108.19

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Lost in Google

     
6:39 pm on Feb 27, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:June 17, 2003
posts:43
votes: 0


My site has been in 1st page serps for many years. Just a few weeks ago, my listing went from a title display and decription to just this:

www.mydomain.com/
Similar pages

In addition, my Yahoo listing disappeared as well. I then did a Yahoo search for pages with my domain included and found most of my interior pages indexed but not my home page.

What happened? Is it possible my site was not ready to be crawled when Googlebot and Slurp robots visied my site - simultaneously?

This is a very "white hat" site - no tricks at all, just good content...

I went and manually requested my site be spidered on both Google and Yahoo, and sent an email to Yahoo requesting any explanation as well.

Is there anything else I can do? Any ideas of why this happened?

7:01 pm on Mar 8, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 27, 2005
posts:93
votes: 0


incrediBILL

Let us see how much you will laugh by placing the address of your main website here, in this forum.

This time your pain will be associated with the grief of your site, not from laughter.

You fail to see the wood from the trees. No contribution and zero input into this problem other than a tumultuous burst of vilification.

Please place your site URL here, in this forum. I will place a few links pointing to your site, it should give you an increase in pagerank, right?

Let us see how much you will stick to your belief regarding the comments you made.

7:36 pm on Mar 8, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 20, 2003
posts:197
votes: 0


As for sending referrers when spidering a page, that doesn't really make sense. If you have 5000 inbound links, do you really want Google to request the same page 5000 times with different referrers? Robots aren't browsers and they generally don't follow links. They just add the linked-to URL to their database (if it's not already there) and it'll get spidered along with all the rest at some future date. The problem is that Google is associating the content returned for the target URL with that URL and all URLs that link to it, rather than simply recording the fact that the redirect URLs are redirects and ignoring them when generating search results.

Perhaps the Request URI can be checked on your site, if it does not match the page you serve a blank page, this could work, I'm going to go check if google actually sends the wrong request URI with hijacked pages.

7:43 pm on Mar 8, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 27, 2005
posts:93
votes: 0


gmiller,

Are you suggesting that a "referrer" only when googlebot has detected your site via a 302 redirect, is a bad idea?

Please explain again why you think it is a bad way forward.

I don't think a site has the right to pump 302 directives to a site that does not want them. Who wrote the scripts? Was it a computer whiz kid, is the script googlebot friendly? what else does the script do? is a meta refresh generated during this process?. I beg you to answer that one especially. Where you aware of that? And are you aware of the residue effect……I won’t explain that one, Please let us know your version.

So, really, you are saying that anybody can create a manipulative script, say a variant of NukeModule GO-PHP that has been modified to hijack your site through an intricate deceptive process that many people do not understand, and I doubt you do, as being OK to live with and we just sit back and watch our sites disappear into oblivion in results? Until google sorts it out, right?

We are trying to help find a solution, action, not speculative gestures, is what is needed.

Go2

8:04 pm on Mar 8, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 17, 2002
posts:95
votes: 0


Google must also allow the target site to remove the offending redirect. It can easily be done, very easily indeed and this may help thousands of people who have lost their hard earned cash and websites.

I don't know if this has been suggested before but perhaps the robots.txt directive could be extended to prevent crawling from a redirect. For example:


User-agent: *
Disallow: /do
Disallow-redirect: /

would disallow the do directory for crawlers following regular links and disallow the entire site if the crawler comes from a redirect.

A robots.txt based redirection filter would probably be easier to use for many webmasters since not everyone is using Apache or php on the server.

8:17 pm on Mar 8, 2005 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14664
votes: 99


No contribution and zero input into this problem other than a tumultuous burst of vilification.

No contribution?

I explained to you that many large legitimate sites like search engines and alexa redirect to monitor actual click-thrus, other than Google all the rest seem to redirect.

I'm sorry if that blows your mind.

8:19 pm on Mar 8, 2005 (gmt 0)

New User

10+ Year Member

joined:Mar 8, 2005
posts:5
votes: 0


I just added the domains of URLs redirecting to my site in the .htaccess file. Will have to see if it fixes my problems.
8:39 pm on Mar 8, 2005 (gmt 0)

New User

10+ Year Member

joined:Nov 23, 2003
posts:35
votes: 0


Question about Google removal tool. subdomain example anyone.ourdomain.com

Because of an error on my part with a subdomain that I set up, google indexed 366 files under the subdomain. A few weeks ago I used the removal tool and submitted a request to de-index items at anyone.ourdomain.com/ Got this reply today.

"Thank you for your reply. This message is to notify you that anyone.ourdomain.com/ has been removed from the Google index. You do not need to take any further action. Please let us know if you have additional questions or concerns."

Thing is, as of 1 hr ago all 366 entries are still showing in the index when doing site:anyone.ourdomain.com

How long does it usally take before the files are washed out of the index and will not show up under site:anyone.ourdomain.com
And, will all entires be removed or just the index or default page for anyone.ourdomain.com/

The domain is now inactive in the DNS

If anyone knows please share.

Trawler

8:51 pm on Mar 8, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Dec 13, 2002
posts:246
votes: 0


Go2 that's an interesting idea, but it seems to me the problem is that Google is being fooled into not know that it is being redirected so I don't think it would work.

In any event the robots.txt should look more like this I believe:

User-agent: Googlebot
Disallow: /referrals

9:05 pm on Mar 8, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
posts:3444
votes: 1


I think Bobby is right here, I tried in the robots.txt with adding the last of the urls that where redirecting to me and activated the removal tool, a day late it said it was complete, but nothing has been done in the serps.

They rely should ban all redirecting sites and go from a clean www.domain.com as a index site and not ww.domain.comgo-php/blabla.., alwas look for ww.domain.com and go from there.

Well as said I dont think can do anything more, then wait and hope the best or we should start doing this our selfs if it is realy that kind of sites google wants, I dont wait 4 month more thats for sure, then we will see what a real webmaster can do with bad tricks, but which obvious loves.

sad sad sad

9:06 pm on Mar 8, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts:1830
votes: 3


BOBBY,

How do you make a copy of it with Google cache?
What kind of problems are you dealing with?

Use your Browser and do a "save as" it will save background colors, images, text, links and etc. You'll have to open in your browser to see it and anyone you send it to tell them to open in a browser.

The problems I was dealing with in that one instance was a client's competitor taking text right off my client's home page and put it on his own page or he would use that text in ads he was posting around the internet. Having documented proof with a date on it via a 3rd impartial party, proving who had the text up first convinced the sites/hosts that controled those pages/websites to take them down.

IncrediBILL


Your anxiety about Alex redirects is misguided as many search engines (like Yahoo, Lycos, Excite and others) and directory sites redirect to your site in order to count the traffic as people click on individual websites.

If they use a tracking script OK, but not a 302 redirect.

Gmiller:

Complaining about Alexa legitimately using 302 redirects is just silly.

Since when is posting a 302 redirect to my site silly?

9:21 pm on Mar 8, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 27, 2005
posts:93
votes: 0


incrediBill,

You think that these directories and many search engines plainly and cleanly count clicks through their systems.

I think you must be living on another planet or in a utopian society where there is a total absence or crime inhabited by sinless altruists of divine proportions. And alexa and the likes revered as the proclaimed deities of proficiency.

Good job we are not talking about reciprocal link exchange, you would advocate that no link pages exist in robots denied pages, or java routed pages that bots cannot access etc. There are even more tricks out there than you may think.

The internet is a dirty playing field with no referees with many casualties that have nowhere to look other than to seek advice from us guys. We should help muster up something to read not muster up a mud fight like I am doing, sorry about that.

OK, Yes many engines like alexa redirect. So what do we have.

Let me tell you. A wild and uncontrolled mixture of deceptive redirects. And many of them cause googlebot problems because these scripts are deployed by anybody and everybody. Hell, I set up my own CGI pearl based that causes a very nasty 302. Done deliberately to retaliate if needed. Working on it I am sure I can get it to dynamically generate a ZERO second meta refresh as well to the targeted page.

Is the guy implementing the scripts at alexa in uniform to a particular directive regarding the serverside redirects?, and is his script compliant to INKTOMI, GOOGLEBOT, MSNBOT etc, etc. I certainly did not conform and I did it to cause a definitive 302 and I expect it to damage the target site I point to. I have never used it.

Do you know the person in alexa that oversees the scripting language that has been incorporated serverside to work the script directives. Do alexa describe anywhere that their 302 redirects are approved by google.

Why is alexa allowed to place a LOCATION PROTOCOL to your site in their server.

Google specifically indicates without a shadow of a doubt that they are against sneaky redirects. There are so many methods to create a 302.

The only good script is a script that bears the hallmark of google’s approval. Alexa must disclose whether their redirecting method is indeed been approved by google.

Look at it this way. It is googlebot that will be given the instruction on the server level. You simply cannot be sure that googlebot is not being manipulated at this critical point, whether intentionally or unintentionally. Yahoo had already defined an overview so are we to assume that the 2 bots are identical, why is there a difference then between them.

9:35 pm on Mar 8, 2005 (gmt 0)

New User

10+ Year Member

joined:Feb 22, 2005
posts:28
votes: 0


Strange that there is this debate on the possible evils of 302 redirects, when there is a thread on this forum :

How to handle outbound links in a directory?

- thats discussing the merits of creating them

Is it me?

9:47 pm on Mar 8, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 5, 2002
posts:413
votes: 0


Thinking about some non-technical remedy, or prevention of this problem...
The motivation$ for these sites seems to be either..
----------
1) adsense (notify google that hi-jacking sites are running adsense)...or contact adsense content advertisers and let them know where their ads are running!)
2) affiliate ads (contact merchants and let them know what their affiliate is doing)
3) other (in my case, a malicious network attempts drive-by install of spyware/virus)notified G,... hope they care?!
4) ignorance (contact offending site -unlikely-, or host)
5) malicious revenge (no remedy?)

just throwing out ideas-add your own!

10:15 pm on Mar 8, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
posts:3444
votes: 1


milehigh - I have begone to tell adsense, but until now no changes, but they did reply, so Im not sure if they care.
11:21 pm on Mar 8, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 10, 2003
posts:929
votes: 13


Getting back to the original topic of this thread...
Does anyone know if G has a problem indexing Doublely 301 redirected pages? Our problem may have started 2 weeks ago when we added code to our .htaccess to 'rewrite' 301 redirect xyz.com to www.xyz.com per G's recommendation (we suddenly started getting double indexing under xyz.com and www.xyz.com a few weeks ago on one domain after another change by G, which was hurting our rankings so we added the code to all domains as a pre-emptive measure). Well we also happen to already have a number of pages (including the home page) redirected to our new domain wxy.com. So someone trying to access [xyz.com...] is going to get 301 redirected to www.xyz.com then 301'd to www.wxy.com. Is that a no-no? What else could we do? And why would G make it affect the entire domain?
The code added looks sorta like this:
RewriteEngine On
RewriteCond %{HTTP_HOST}!www.xyz.com
RewriteRule (.*) [xyz.com...] [R=301,L]

Redirect 301 /index.htm [wxy.com...]
Redirect 301 /abcdefg.htm [wxy.com...]

P.S. is the way that about.com links using the following format:
zzzzzz.about.com/gi/dynamic/offsite.htm?site=http://www.xyz.com%2Fxyz.htm

Considered a BAD 302?

[edited by: MikeNoLastName at 11:46 pm (utc) on Mar. 8, 2005]

This 206 message thread spans 14 pages: 206
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members