Forum Moderators: mack
I have launched 7 new sites early this month. expected msnbot to be the first to visit the new sites and was surprised to see Yahoo was the first.
Until today, msnbot rarely visit the new sites and grabs only the robots.txt
My robots.txt is valid and excludes several directories from being indexed but DOES NOT block any robot.