homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

What in the world is a robot.txt?
I need this to be stated simple for me

 5:42 pm on Feb 10, 2004 (gmt 0)


I submitted my site to something called "Scrubbing the web" and it offered an analizer thing on there which said my robot.txt thing wasnt there.

Can anybody explain this to me in a simple manner please? Are there different codes you have to put in the meta tag place for different sites?

I just clicked on some sort of Robots test and it gave hundreds of errors and I dont have a clue what its talking about.




 6:02 pm on Feb 10, 2004 (gmt 0)

Hi Claudia welcome to webmaster world

You don't need a robots.txt file, it is used to disallow spiders from your site if they are programmed to follow the robots.txt guide lines

A few examples of what you may put in a robots.txt file
This indicates that nothing is disallowed and the spider can follow all links
User-agent: *

To allow a single robot complete access and exclude all others
User-agent: Googlebot/1.0
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

If you do not want certain directories to be spidered
User-agent: *
Disallow: /cgibin (change this to what you require)

or any directories which are private
User-agent: *
Disallow: /sitestats (change this to what you require)

You would create the file using say notepad and FTP using ASCII mode to your site root directory

hope this helps



 6:22 pm on Feb 10, 2004 (gmt 0)


Thank you for your reply. So does this mean that if I want the search engine to include all my pages, I really dont have to put any kind of a robot txt line on my page and it will do it automatically?

..and that you only have to put that line in there if you DONT want it to do certain pages?


 6:27 pm on Feb 10, 2004 (gmt 0)

Yep. That's about the size of it, Claudia :)


 6:35 pm on Feb 10, 2004 (gmt 0)

You can have a blank robot.txt file, this is so when a robot requests the file a 404 error (file not found) does not appear in your site's stats


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved