Robots.txt

can anyone explain a way of developing one?

[agerhart]

I am in charge of SEO at the startup company I am working for, and I have just been presented with the project of developing a robots.txt that works. I know very little about this subject except for the fact that it defines what pages a SE robot will look at. If anyone could explain this subject a little furthur it would help. Even a link to some other article that has this information would be of great help. Thanks
-A Gerhart

[Brett_Tabke]

You know the easiest thing to do? Start looking at robots.txt on bigger sites.
They are usually well formed and only require editing.

The robots page [info.webcrawler.com]
The Robots.txt Exclusion Standard [info.webcrawler.com]

Robots.txt file [tardis.ed.ac.uk] Syntax Checker
Robots.txt Examples [searchtools.com] at SearchTools.com

Intomi FAQ with blurbs robots.txt [support.inktomi.com].

[agerhart]

Thank Brett,
After I posted this topic I went around and looked at some of the robots.txt, and I have a good feeling of how to write one now. The only problem I have is not knowing which robots are used by each SE. Is there a site that lists which SE's use which robots?

[agerhart]

Brett,
I have just created my first, very simple Robots.txt for a site that I have been assisting with. I was wondering if you could tell me know if I have written in correctly:

User-agent: *
Disallow: /stats
Disallow: application.asp
Disallow: en_email.asp
Disallow: in_email.asp
Disallow: newscript2.asp
Disallow: newscript3.asp
Disallow: riv_extract.asp
Disallow: riv_extract_app.asp
Disallow: riv_extract2.asp
Disallow: riv_signup.asp
Disallow: signup.asp

-That's it......like I said, pretty simple. Will this work, or do I need a forward slash in front of the ASP files?

-A Gerhart

[Brett_Tabke]

Looks good to me. Run it through the robots.txt checker after you put it on the server.