Forum Moderators: mack
woop01, thanks for posting your story about robots.txt
In 2 years, Yahoo, Google, Teoma (Ask), and Mama never excluded a site if it did not have a robots.txt file. The assumption seemed to be no robots.txt indicated permission to crawl everything. As I have posted on here many times, several pages of our site have been the #1 results for 2 years on all those SEs on 2 -4 keyword searches, with no robots.txt file. Not a single one of our pages shows up anywhere in MSN beta. So, it sounds like you MAY be describing our problem.I just added a robots.txt to our root to see if THAT might be MSN beta's problem with listing our site. The file contains some comments and:
User-agent: *
Disallow:If someone reading this is not familiar with robots.txt syntax, there are examples and a robots.txt file checker here:
Make sure you use an editor that breaks lines Unix style (line feed w/o carriage return). The Notepad editor, like most Windows platform tools, adds a carriage return with the line feed which won't work.
Sorry for the delay on responding to this part of the thread. I just want to quickly clarify this. MSNBot will support both and either. That is, you can use a carriage return and line feed or just one of those.
Additionally, like other crawlers we will crawl and index a site that has no robots.txt.
- msndude (msd)
OK, who DID say that? Personally I would benefit in the SERPs.
I might LIKE it to be true, but I really doubt the sites I see would all validate.
Without digging thru the archives, is there any corroboration? -Larry
That 100% validation statement was taken out of context from one of my posts. We rank #1 in Yahoo, Google, and every other SE we've checked for a 2 keyword search, but we do not show up in any MSN beta SERP. Here is what I originally posted that is being taken out of context, like "whisper down the lane":
...I am not going for 100% W3C validation however. I use some IE6 only attributes very selectively because I like the results so I am keeping them.
Wouldn't it be a kick in the pants if after Microsoft has given us DHTML and other non-standard goodies, now MSN requires 100% W3C valdation to be included in the results, and the other SEs don't care about that. lol
In future, try to read posts more carefully before disagreeing otherwise people may conclude you are stupid and/or argumentative.
Pretty strong from someone that has trouble following Webmaster World TOS.
Kaled. said:
If you read my second post, you would see that my settings specifically stated that I wanted no more than 2 results from the same site and they should be grouped together.
MSN considers sub-domains to be different sites which is not unreasonable.
In fact on Google; trying to not display results on your example by either the main domain or sub domain does not work at all.
I just ran on MSN beta that 2 keyword search where we are #1 on all the other SEs. We are still nowhere to be seen on the MSN beta SERPs, so I ran the top 2 results through the W3C validator.
#1 Link - 58 errors
#2 Link - 167 errors
So we can add W3C validation to the list of criteria that MSN beta does not care about.
-- now MSN requires 100% W3C valdation to be included in the results --
Just to clarify -- this is not true. MSN Search does not require 100% W3C validation in order to be included in our Web Search results.
- msndude (msd)
My sites are created in Dreamweaver.
Dreamweaver, is great to use but produces some dodgy code.
My sites are top in MSN (beta), Google and Yahoo.
A couple of years ago I started validating my code with various checkers. The results were shocking, so many errors I thought all was lost.
That was then. Now I know not to worry.
No search engine can do this, that would mean that only like probably less than 0.01% of all webpages in the world could get indexed, probably a lot less, since almost all xhtml pages don't validate.
requiring validation would be retarded. neither yahoo nor google validate and last time i checked neither did microsoft.com. until just recently msn itself didn't validate either (btw kudos to them for making it right, though it doesn't seem to feed a valid document to the validator i can see it on my end). amazing what some people will believe when told though..
Also, the odd tobacco company ad link site that uses the actress name is not listed in any of the top 200 results.
When I remove the quotes and search MSN beta again, that odd tobacco company ad site appears on the page 1 SERP and our site is not listed in any of the top 200 results.
Most of the results in either search are for people with the same name as the actress (both keywords adjacent and in the same order), but the links offered in the SERPs are quite different.
Maybe one MSN department is in charge of the quoted keywords algorithm and another MSN department is in charge of the unquoted keywords algorithm?
That's a question folks, please don't quote it and run with it like the 100% W3C validation remark. lol
One other thing. A search with and without quotes produces an identical page 1 SERP on Yahoo, and only slightly different ordered results on Teoma and Google.
MSN considers sub-domains to be different sites which is not unreasonable.
That's nothing less than ambrosia to spammers. With a little SEO (and maybe some luck) one site could easily push all others off the front page of the SERPS.
Kaled.
MSN considers sub-domains to be different sites which is not unreasonable.
Yes it is. They are part of the same domain. Nobody is saying ignore them but they should be treated just like inner pages of the same site. Now if you say its easier to pick up relevant information if they are treated as different domains because they have a harder time dealing with deep content then you may be onto something. Google did this for a while and even extedned it to networks and results were the better for it, theyve eased off from that these days though.
i typed my own site name in (nobody else in the world has the same name) and i get every site with my site name on (links and so forth) but not a sniff of my site in the first 30 results.
i do get my .co.uk site postion 26 which does not have a single link to it (all to mysite.com)
anyone else have this problem?
---
Also...
the site is excluded in the UK only search yet is based on a UK registered IP Address which google, yahoo and co agree makes it a UK site.
I see the same>
MSN search seems to be very search location/msn.tld oriented.
That is fine if you specify it like in Google, but not if you just want the most authoritative information source..
Someone too enthousiastically turned the geo.location knob.
Time to reset it.
Here's something new. When I enter the two word actress name in quotes to force an exact match search, our site is the #1 result on MSN beta, the same top ranking it has been given with an unquoted 2 keyword search by all the other SEs.
This has been the case from early days with the MSN Beta. I've typed in a lot of 2 word rock band names - put them in quotes and the results are good - leave the quotes off and up to 70% of the pages are unrelated.
Damn! *grumble* I always thought Joe Camel was a guy.
Can anyone tell me if the msn.co.uk is live in the UK?
Only theory for now is that if the site can't be specifically tagged to a geo (either no geo info, or too many conflicting options) then it defaults to USA/North America.
CF
Lets hope the MSN boys n girls pick this little issue up.
Thanks. Dixon.
[ads.msn.com...]
That goes against all this talk about searchers wanting paid results and normal search results clearly defined. The fact that MSN has done (what seems) a pretty good job of defining the PPC ads makes one assume that all the other links in the SERPs area are natural. Not the case.
This blows and pushes the natural serps down yet another spot. Soon we'll be below the fold...awesome.
They are indeed, I hope it's not intentional.
And the serps are basically educated guesses of what the pages are about. Which are mostly wrong.
I think the pay per click model is a good thing in a way. And for those of us that can't afford the PPC there are the free serps which are a hit or miss depending on your SEO experience.
Without us SEO's the engines wouldn't have very good Serp's at all in my opinion. The engines would have better results if they would listen to our pages and what we're telling them in our text.
My site, #3 out of 6 million+ dropped down to 3rd page.
I was crowded down by some of the crappiest spammiest stuff I ever saw for my arcane search word.
One ranking site sells 'bling' and tee-shirts.
No mention of the search word, conceptually a parsec away.
Looks like somebody figured out the MSN Beta ranking algorith.
They made short work of what was (briefly) some pretty good results.
I hope this is just a glitch, but fear otherwise. -Larry
That goes against all this talk about searchers wanting paid results and normal search results clearly defined. The fact that MSN has done (what seems) a pretty good job of defining the PPC ads makes one assume that all the other links in the SERPs area are natural. Not the case.
That reminds me on Google - MSN must have learned this from them... Adwords with high PPC and the traffic pattern changes like a mistery producing more tageted traffic...
I run a online hotel booking site made obviously only from affiliate links.
In one year I received over 70.000 targeted hits and only 300 of them generated sales.
Last month I had about 5700 targeted hits and made only $ 600 affiliate sales.
Being frustrated with the low volume of sales, at the beginning of January I’ve added Adsense to all my pages.
The result?
Affiliate sales over $ 700
Adsense revenue over $ 350
From my own experience affiliate sites are not playing a fair game. I don’t worry to much
about their performance and revenue as is nothing I can do to control them and monitories them. I’m at their pitty.
With adsense is different. The more exposure you get the more revenue you have.
Getting a click is not the same with making a sale. Adsense really works!
I’d like to hear more from other webmasters unhappy with their affiliate revenue.
Kellyandsummer said “We currently receive all visitors via PPC, and push the traffic to the sites we are affiliated with.”
Well, if I was to do the same I’d lose lots of cash.
From 5700 hits I’ve made about $ 600.
One PPC would have cost me $1.60
So you do the maths.
Moreover, other websites affiliated with the same travel company as myself are using PPC on Google and Overture and they’ve been there for months.
Beats me. I don’t know how they do it.
My website can be found on few major search engines at the top of the first page
using relevant targeted keywords and still the results are miserable.
Thanks.
