Forum Moderators: mack
woop01, thanks for posting your story about robots.txt
In 2 years, Yahoo, Google, Teoma (Ask), and Mama never excluded a site if it did not have a robots.txt file. The assumption seemed to be no robots.txt indicated permission to crawl everything. As I have posted on here many times, several pages of our site have been the #1 results for 2 years on all those SEs on 2 -4 keyword searches, with no robots.txt file. Not a single one of our pages shows up anywhere in MSN beta. So, it sounds like you MAY be describing our problem.I just added a robots.txt to our root to see if THAT might be MSN beta's problem with listing our site. The file contains some comments and:
User-agent: *
Disallow:If someone reading this is not familiar with robots.txt syntax, there are examples and a robots.txt file checker here:
Make sure you use an editor that breaks lines Unix style (line feed w/o carriage return). The Notepad editor, like most Windows platform tools, adds a carriage return with the line feed which won't work.
Sorry for the delay on responding to this part of the thread. I just want to quickly clarify this. MSNBot will support both and either. That is, you can use a carriage return and line feed or just one of those.
Additionally, like other crawlers we will crawl and index a site that has no robots.txt.
- msndude (msd)
I'm talking in the organic SERP's, either very co incidental or a corrupt and dangerous practise.
I imagine there are very few sites that are no. 1 on the three main se. Anyone that's getting 2 out of 3 is doing pretty well in my opinion.
We have consistent no1 for 100s of very competitive keywords (on 3 of 5 sites) across all search engines.
3 out of 5 for us, however our 2 newer sites all show blank spots on Google only. No1 on the others.
Anybody that thinks Google is performing, and ranking sites in a fair manor needs to open their eyes to what is happening.
Serious damage has already been done... Google you should be very afraid in my opinion.
