| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|What in the world is a robot.txt?|
I need this to be stated simple for me
| 5:42 pm on Feb 10, 2004 (gmt 0)|
I submitted my site to something called "Scrubbing the web" and it offered an analizer thing on there which said my robot.txt thing wasnt there.
Can anybody explain this to me in a simple manner please? Are there different codes you have to put in the meta tag place for different sites?
I just clicked on some sort of Robots test and it gave hundreds of errors and I dont have a clue what its talking about.
| 6:02 pm on Feb 10, 2004 (gmt 0)|
Hi Claudia welcome to webmaster world
You don't need a robots.txt file, it is used to disallow spiders from your site if they are programmed to follow the robots.txt guide lines
A few examples of what you may put in a robots.txt file
This indicates that nothing is disallowed and the spider can follow all links
To allow a single robot complete access and exclude all others
This would prevent your entire web site from being indexed
If you do not want certain directories to be spidered
Disallow: /cgibin (change this to what you require)
or any directories which are private
Disallow: /sitestats (change this to what you require)
You would create the file using say notepad and FTP using ASCII mode to your site root directory
hope this helps
| 6:22 pm on Feb 10, 2004 (gmt 0)|
Thank you for your reply. So does this mean that if I want the search engine to include all my pages, I really dont have to put any kind of a robot txt line on my page and it will do it automatically?
..and that you only have to put that line in there if you DONT want it to do certain pages?
| 6:27 pm on Feb 10, 2004 (gmt 0)|
Yep. That's about the size of it, Claudia :)
| 6:35 pm on Feb 10, 2004 (gmt 0)|
You can have a blank robot.txt file, this is so when a robot requests the file a 404 error (file not found) does not appear in your site's stats
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved