Forum Moderators: Robert Charlton & goodroi
www.mydomain.com/
Similar pages
In addition, my Yahoo listing disappeared as well. I then did a Yahoo search for pages with my domain included and found most of my interior pages indexed but not my home page.
What happened? Is it possible my site was not ready to be crawled when Googlebot and Slurp robots visied my site - simultaneously?
This is a very "white hat" site - no tricks at all, just good content...
I went and manually requested my site be spidered on both Google and Yahoo, and sent an email to Yahoo requesting any explanation as well.
Is there anything else I can do? Any ideas of why this happened?
Let us see how much you will laugh by placing the address of your main website here, in this forum.
This time your pain will be associated with the grief of your site, not from laughter.
You fail to see the wood from the trees. No contribution and zero input into this problem other than a tumultuous burst of vilification.
Please place your site URL here, in this forum. I will place a few links pointing to your site, it should give you an increase in pagerank, right?
Let us see how much you will stick to your belief regarding the comments you made.
As for sending referrers when spidering a page, that doesn't really make sense. If you have 5000 inbound links, do you really want Google to request the same page 5000 times with different referrers? Robots aren't browsers and they generally don't follow links. They just add the linked-to URL to their database (if it's not already there) and it'll get spidered along with all the rest at some future date. The problem is that Google is associating the content returned for the target URL with that URL and all URLs that link to it, rather than simply recording the fact that the redirect URLs are redirects and ignoring them when generating search results.
Perhaps the Request URI can be checked on your site, if it does not match the page you serve a blank page, this could work, I'm going to go check if google actually sends the wrong request URI with hijacked pages.
Are you suggesting that a "referrer" only when googlebot has detected your site via a 302 redirect, is a bad idea?
Please explain again why you think it is a bad way forward.
I don't think a site has the right to pump 302 directives to a site that does not want them. Who wrote the scripts? Was it a computer whiz kid, is the script googlebot friendly? what else does the script do? is a meta refresh generated during this process?. I beg you to answer that one especially. Where you aware of that? And are you aware of the residue effect……I won’t explain that one, Please let us know your version.
So, really, you are saying that anybody can create a manipulative script, say a variant of NukeModule GO-PHP that has been modified to hijack your site through an intricate deceptive process that many people do not understand, and I doubt you do, as being OK to live with and we just sit back and watch our sites disappear into oblivion in results? Until google sorts it out, right?
We are trying to help find a solution, action, not speculative gestures, is what is needed.
Google must also allow the target site to remove the offending redirect. It can easily be done, very easily indeed and this may help thousands of people who have lost their hard earned cash and websites.
I don't know if this has been suggested before but perhaps the robots.txt directive could be extended to prevent crawling from a redirect. For example:
User-agent: *
Disallow: /do
Disallow-redirect: /
would disallow the do directory for crawlers following regular links and disallow the entire site if the crawler comes from a redirect.
A robots.txt based redirection filter would probably be easier to use for many webmasters since not everyone is using Apache or php on the server.
No contribution and zero input into this problem other than a tumultuous burst of vilification.
No contribution?
I explained to you that many large legitimate sites like search engines and alexa redirect to monitor actual click-thrus, other than Google all the rest seem to redirect.
I'm sorry if that blows your mind.
Because of an error on my part with a subdomain that I set up, google indexed 366 files under the subdomain. A few weeks ago I used the removal tool and submitted a request to de-index items at anyone.ourdomain.com/ Got this reply today.
"Thank you for your reply. This message is to notify you that anyone.ourdomain.com/ has been removed from the Google index. You do not need to take any further action. Please let us know if you have additional questions or concerns."
Thing is, as of 1 hr ago all 366 entries are still showing in the index when doing site:anyone.ourdomain.com
How long does it usally take before the files are washed out of the index and will not show up under site:anyone.ourdomain.com
And, will all entires be removed or just the index or default page for anyone.ourdomain.com/
The domain is now inactive in the DNS
If anyone knows please share.
Trawler
In any event the robots.txt should look more like this I believe:
User-agent: Googlebot
Disallow: /referrals
They rely should ban all redirecting sites and go from a clean www.domain.com as a index site and not ww.domain.comgo-php/blabla.., alwas look for ww.domain.com and go from there.
Well as said I dont think can do anything more, then wait and hope the best or we should start doing this our selfs if it is realy that kind of sites google wants, I dont wait 4 month more thats for sure, then we will see what a real webmaster can do with bad tricks, but which obvious loves.
sad sad sad
How do you make a copy of it with Google cache?
What kind of problems are you dealing with?
Use your Browser and do a "save as" it will save background colors, images, text, links and etc. You'll have to open in your browser to see it and anyone you send it to tell them to open in a browser.
The problems I was dealing with in that one instance was a client's competitor taking text right off my client's home page and put it on his own page or he would use that text in ads he was posting around the internet. Having documented proof with a date on it via a 3rd impartial party, proving who had the text up first convinced the sites/hosts that controled those pages/websites to take them down.
IncrediBILL
Your anxiety about Alex redirects is misguided as many search engines (like Yahoo, Lycos, Excite and others) and directory sites redirect to your site in order to count the traffic as people click on individual websites.
If they use a tracking script OK, but not a 302 redirect.
Gmiller:
Complaining about Alexa legitimately using 302 redirects is just silly.
Since when is posting a 302 redirect to my site silly?
You think that these directories and many search engines plainly and cleanly count clicks through their systems.
I think you must be living on another planet or in a utopian society where there is a total absence or crime inhabited by sinless altruists of divine proportions. And alexa and the likes revered as the proclaimed deities of proficiency.
Good job we are not talking about reciprocal link exchange, you would advocate that no link pages exist in robots denied pages, or java routed pages that bots cannot access etc. There are even more tricks out there than you may think.
The internet is a dirty playing field with no referees with many casualties that have nowhere to look other than to seek advice from us guys. We should help muster up something to read not muster up a mud fight like I am doing, sorry about that.
OK, Yes many engines like alexa redirect. So what do we have.
Let me tell you. A wild and uncontrolled mixture of deceptive redirects. And many of them cause googlebot problems because these scripts are deployed by anybody and everybody. Hell, I set up my own CGI pearl based that causes a very nasty 302. Done deliberately to retaliate if needed. Working on it I am sure I can get it to dynamically generate a ZERO second meta refresh as well to the targeted page.
Is the guy implementing the scripts at alexa in uniform to a particular directive regarding the serverside redirects?, and is his script compliant to INKTOMI, GOOGLEBOT, MSNBOT etc, etc. I certainly did not conform and I did it to cause a definitive 302 and I expect it to damage the target site I point to. I have never used it.
Do you know the person in alexa that oversees the scripting language that has been incorporated serverside to work the script directives. Do alexa describe anywhere that their 302 redirects are approved by google.
Why is alexa allowed to place a LOCATION PROTOCOL to your site in their server.
Google specifically indicates without a shadow of a doubt that they are against sneaky redirects. There are so many methods to create a 302.
The only good script is a script that bears the hallmark of google’s approval. Alexa must disclose whether their redirecting method is indeed been approved by google.
Look at it this way. It is googlebot that will be given the instruction on the server level. You simply cannot be sure that googlebot is not being manipulated at this critical point, whether intentionally or unintentionally. Yahoo had already defined an overview so are we to assume that the 2 bots are identical, why is there a difference then between them.
How to handle outbound links in a directory?
- thats discussing the merits of creating them
Is it me?
just throwing out ideas-add your own!
Redirect 301 /index.htm [wxy.com...]
Redirect 301 /abcdefg.htm [wxy.com...]
P.S. is the way that about.com links using the following format:
zzzzzz.about.com/gi/dynamic/offsite.htm?site=http://www.xyz.com%2Fxyz.htm
Considered a BAD 302?
[edited by: MikeNoLastName at 11:46 pm (utc) on Mar. 8, 2005]
