Welcome to WebmasterWorld Guest from 54.159.246.164

Forum Moderators: goodroi

Message Too Old, No Replies

What in the world is a robot.txt?

I need this to be stated simple for me

   
5:42 pm on Feb 10, 2004 (gmt 0)

10+ Year Member



Hi,

I submitted my site to something called "Scrubbing the web" and it offered an analizer thing on there which said my robot.txt thing wasnt there.

Can anybody explain this to me in a simple manner please? Are there different codes you have to put in the meta tag place for different sites?

I just clicked on some sort of Robots test and it gave hundreds of errors and I dont have a clue what its talking about.

Claudia

6:02 pm on Feb 10, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi Claudia welcome to webmaster world

You don't need a robots.txt file, it is used to disallow spiders from your site if they are programmed to follow the robots.txt guide lines
[robotstxt.org...]

A few examples of what you may put in a robots.txt file
This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/1.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

If you do not want certain directories to be spidered
User-agent: *
Disallow: /cgibin (change this to what you require)

or any directories which are private
User-agent: *
Disallow: /sitestats (change this to what you require)

You would create the file using say notepad and FTP using ASCII mode to your site root directory

hope this helps

ncw164x

6:22 pm on Feb 10, 2004 (gmt 0)

10+ Year Member



ncw164x

Thank you for your reply. So does this mean that if I want the search engine to include all my pages, I really dont have to put any kind of a robot txt line on my page and it will do it automatically?

..and that you only have to put that line in there if you DONT want it to do certain pages?

6:27 pm on Feb 10, 2004 (gmt 0)

10+ Year Member



Yep. That's about the size of it, Claudia :)
6:35 pm on Feb 10, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You can have a blank robot.txt file, this is so when a robot requests the file a 404 error (file not found) does not appear in your site's stats

ncw164x