Robots txt?

Do I Need One?

[Gpatt]

Hello All

If I do not have a "robot.txt" file will this adversly affect my search engine rankings?

If so:
What would I need to put to allow all robots to access all of my pages?

Thanks if you can help!

[lorax]

IMHO it's always a good idea to have a robots.txt file. Many of the SEs look for this file first and if they don't find it will keep looking for it each time they come back before crawling your website.

[Matt Probert]

If I do not have a "robot.txt" file will this adversly affect my search engine rankings?

No. The oprional file "robots.txt" is used to PREVENT some or all (well behaved) robots from spidering all or part of a web site.

If you want the whole site picked up by search engines, you have no need of a robots.txt file.

Matt

[ncw164x]

It won't affect your search engine rankings but it's wise to give a spider what's requested than to server a 404 error which is what you will be giving if you don't have a robots.txt file in your sites root directory. The file is only used to disallow well behaved spiders robots, others like email siphons or site rippers will just ignore the file.

To make a robots.txt file use a text editor like notepad, you can have a blank file or add any of the following as required

(This will allow access to all of your site)
User-agent: *
Disallow:

(This will disallow access to your cgi-bin directory)
User-agent: *
Disallow: /cgi-bin/

(This will disallow access to your images directory)
User-agent: *
Disallow: /images/

Save the file as robots.txt and upload to your root directory via FTP in ASCII mode

For more details see the link below
[searchengineworld.com...]

Hope this helps

[Gpatt]

Thanks Gents!