Robot.txt

Using it or not

[le_gber]

Hi I just wanted to know if using robot.txt help in ranking at all.
When a spider hit my server they look for robot.txt, to see if there isn't anything that shouldn't be indexed.
Should I put one even if I don't have anything to tell the spider not to index?

Thanks

Leo

[Sinner_G]

Can't find the thread now, but Googleguy somewhere wrote to have a robots.txt, so I guess it can't hurt to have one.

[hetzeld]

Hi Leo,

The file robots.txt won't help your rankings. Having such a file on your server will prevent getting a 404 error every time it is asked.

If you don't have one, and use custom 404 page(s), make sure you're returning the 404 header so that your custom 404 page won't be interpreted as a robots.txt with a lot of "syntax errors".

If you use the Apache directive "ErrorDocument" in a .htaccess file, avoid using full URLs for the custom 404 page (no http prefix), i.e.:

ErrorDocument 404 /your-good-404-page.html

and not:

ErrorDocument 404 [domain.tld...]
// previous line is wrong as it doesn't return the correct header!

Dan

[le_gber]

Googleguy somewhere wrote to have a robots.txt

ok will do one then.

use custom 404 page & Apache directive "ErrorDocument"

none of these

thank you both for your help

leo