Proper robots.txt

[theadvocate]

Hello,

I know this has been beaten to death, and I have done many searches, but have not found the specific answer.

I was told that my robots.txt file "needs fixing", but they would not elaborate.

My meta tags have the following:

<META NAME="robots" CONTENT="index, follow">

My robots.txt only has the following:

User-agent: *

I know you use one or the other. I rely on Meta Tags, but the reason I created the robots.txt file is because our error logs were getting filled up with errors that the robot.txt could not be found, so I put one on the server.

The response "Your robots.txt file needs fixing" came from my inquiry as to why MSN spiders might not be crawling our site. Is there anything wrong with our robots.txt file that might prevent a spider from doing anything, or is it just neutral because it does not define anything?

In other words, does it need fixing? And if so, how?

TIA

[fiestagirl]

To allow all robots complete access:

User-agent: *
Disallow:

[theadvocate]

Thank you for your reply.

So putting "Disallow:" with nothing after it lets everything spider our site?

And that is the only thing I need to do to fix the robots.txt file?

TIA

[topr8]

i have sites with no robots txt and they get indexed by msn.

i'm not sure that is your problem.

...

you can always validate your robots txt file if you like, do a google search for robots.txt validator and there are a bunch of them.

[tedster]

Note that we have an entire forum here, devoted to Robot.txt

[webmasterworld.com...]

For that reason, I'm closing this thread and asking that any further discussion go to the dedicated forum - thanks.