Forum Moderators: mack
I see you are back in business, and I have a question which I asked long time before, but I guess the old Dude was gone or something, so I did not get a good answer. Here is the problem:
I have a few links out of my site which I don't want any bot to follow, so I have a folder which I have excluded from crawling in the robots.txt and there I use meta tags to redirect to those pages - about 15 of them.
However, for over 6 months now, MSN bot has been hitting those pages. When I do a site: search on MSN, those are the first pages that come up in the results (no text, just URLs and my domain name as a title), almost always with a recent date of visit behind them, and then our regular pages follow.
This is very disturbing, as I blame it for our bad position on MSN - our website is full of 100% unique content, fresh news, 1000+ links, NO SPAM (I know it's relative), good Google position, and good Yahoo position (as good as it will get without spamming), and targeting a keyword which is not competitive at all. And yet, we are in the back 40s (literally) for that keyword.
Ok, I will give it the benefit of the doubt, that I rank badly because of something else, but what can I do about those pages, and what is MSN doing to correct this, if anything?
I have two other website - one has the same problem (pages in the robots.txt show as first returned results when site: search) and the other one - simillar pages show up in the top 10 for site: search.
This really bugs me, and it should do the same to you, as it seems that MSN is giving more weight to pages with 0 content whatsoever. Please, let us know about this, as I am sure other people have the same problem.
I am waiting for MSNDude to get back to about my problems but haven't heard anything and I can't see any movement from MSN's robot in my server stats so I am not sure what is going on. Maybe I just too eager to get things right again since almost all my site, if not all of them, has been "penalized" by MSN.
Like you, my sites are ranked high in Google and Yahoo! but MSN? Where is MSN? If they are ever going to compete with Google then they need to do something fast. I used to have MSN/Live.com as my start page but have changed to Google, not that they have better results but my sympathy for MSN is kinda' low right now.
Anyhow - it's not about redirect, but about MSN bot not honoring the robots.txt and why are those pages listed as first returned results for the "site:" command. It's MSN bot's problem to fix, and not I. I was wondering if MSN knew about this and if they are doing something about it.
Yes, but I keep them in a folder which is outed in the robots.txt - so nomatter the content of the folder - MSN bot has no business going there - so it's more of robots.txt problem, than with redirects.
"I use meta tags to redirect to those pages - about 15 of them"
You're using meta-refresh to redirect them? And there are links to those pages? Are they fully indexed and cached or just the URL?
[edited by: Marcia at 10:46 pm (utc) on Jan. 14, 2007]
It's the weekend, he might not work 7 days a week (or from home on weekends). :)
use noindex meta tags, they usually work very well, an where the url has already being indexed, it will fall out with time
It's the weekend, he might not work 7 days a week (or from home on weekends). :)
He sure should :)
I dont think the problem is with my website - I do link to those pages from almost erevy page of my website - they are affiliate program links. It's actually a little confusing - I dont thing MSN is "caching" them, but it regularly visits them and my problem is that it ranks them before any other page of my website with the "site:" command, which I am 99% sure is what causes our low ranking.
