How to Validate robots.txt and do I even have one?

[neophitemoneymaker]

Hi, I am very very new to all this. (I don't even know how to spell neophite!) How do I know if I have a robots.txt file on my website? (There is no file named robots.txt anywhere in the directories)

Secondly, when I use the robots.txt VALIDATOR, it returns with 157 errors or something like that and does not validate.Please help.

Thanks,
mm

[Brett_Tabke]

No, you don't need a robots.txt. If you had one, it would be located here:

http://www.domain.com/robots.txt

[mack]

Your robots.txt, if you have one will be located at wwwroot (the same location as your homepage).

Technichaly you do not need to have one, although if you do not have one you will get a lot of "404 file not found" entries in your logs from search engine bots looking for the file. To prevent this you can always just use a blank file.

Mack.

[neophitemoneymaker]

Okay thanks, and what about the robots.txt validator? Why is that NOT validating? Does it have to validate for SE's to pick up the site properly? I ask because Google and Yahoo just have this lame listing for my site (just the site URL) with no description or anything. And I am getting ZERO referals from them, which make me hugely depressed and stressed (I want to quit my job by August.)

[mack]

We cant really offer advice about why it doesnt validate without seeing it.

The fact that it doesn't validate may prevent it from working, or worse still it may prevent bots visiting your pages. (worst case scinario)

Does the validation tool give you any reasons?

Mack.

[justgetit]

Hi Mack, I changed my username, again thanks for your help and concern.

I don't know if this is legal but the site is www.example.com

and the Validator gives me 157 errors and 90 warnings

[edited by: ThomasB at 9:48 pm (utc) on May 1, 2005]
[edit reason] no specifics please [/edit]

[justgetit]

googlebot did spend 66+ min on my site this past week, but the listing on google still doesn't include a description.

[maccas]

Hi justgetit, I couldn't find a robots.txt at yourdomain.com/robots.txt and yes it is illegal :) to post your url

[justgetit]

Hello maccas,

Alright, do you think I should have a robots.txt file?

again why is the validator coming back with so many WARNINGS and ERRORS?

Thanks,
Mike

[mack]

My guess is it is hitting you with every available error because it's not there. Perhaps the app they you are using doesn not stop when it comes across a 404.

Mack.

[maccas]

As brett said earlier, you don't need one unless you want to block specific pages, folders etc from being indexed. Have a look at [searchengineworld.com...]

[Brett_Tabke]

You don't actually have a robots.txt file neophitemoneymaker. So, it is looking at whatever url you are feeding it.

[justgetit]

Thanks Brett and maccas, \