homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

How do i do a robot.txt
What does it do?

5+ Year Member

Msg#: 2613 posted 11:48 am on Dec 13, 2005 (gmt 0)

What is robot.txt.
I am confused i have been doing the site for a year now and have a feeling the reason my site is badly performing in the SE's is the robopt.txt has not been done.
Also what is the best way of making you sitemap as i have done an xml sitemap with a third party tool and dont think it is doing the right job.?
Any help much appreciated
Very keen, Youngkeen



WebmasterWorld Senior Member 10+ Year Member

Msg#: 2613 posted 12:45 pm on Dec 13, 2005 (gmt 0)

robots.txt is simply a text file that you place in the root of your web folder that gives instructions to spiders such as Googles Googlebot - [google.com...] or Yahoos Slurp - [help.yahoo.com...] what it can and can't crawl.

By default if you don't have a robots.txt it assumes everything on your site is freely crawable, if it is you don't have the need for a robots file. If however you don't want it crawling a certain page you can indicate this in the file.

This page has more details (hope it is ok to list, sure i have seen it here before)

But to answer your quesion, the lack of a robots.txt file won't be hurting your rankings.


10+ Year Member

Msg#: 2613 posted 12:53 pm on Dec 13, 2005 (gmt 0)

At the top of this page there is a lonk named "Glossary"

Click that and look ofr robots.txt - click it.

There is a description and a link to the full spec.


Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved