Robots.Txt

May In fact be hindering website placement

[john_patti]

I was interested in a submission from entireweb.com. I stubled across this statement

" The web page must be accessible to our spider and must not reside on a secure server, use any blocking technologies (such as robots.txt files), or require a username, password, cookie or other authentication."

You would think robot.txt is helpful to SE's, so can anybody give me a reason why they bypass this?

[agerhart]

They may be trying to make webmasters aware of their robots.txt, and insure that they are not disallowing spiders and robots access to their webpages.

[deejay]

I agree with Agerhart.

If they can't handle a site simply because of the existence of a robots.txt file, they're not gonna have much of a web to index.

I'm sure it's just a case of bad phrasing and they intended to say something about robots.txt files which ban their bot.