Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

How do i do a robot.txt

What does it do?

11:48 am on Dec 13, 2005 (gmt 0)

New User

10+ Year Member

joined:Apr 24, 2005
votes: 0

What is robot.txt.
I am confused i have been doing the site for a year now and have a feeling the reason my site is badly performing in the SE's is the robopt.txt has not been done.
Also what is the best way of making you sitemap as i have done an xml sitemap with a third party tool and dont think it is doing the right job.?
Any help much appreciated
Very keen, Youngkeen
12:45 pm on Dec 13, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 8, 2001
votes: 0

robots.txt is simply a text file that you place in the root of your web folder that gives instructions to spiders such as Googles Googlebot - [google.com...] or Yahoos Slurp - [help.yahoo.com...] what it can and can't crawl.

By default if you don't have a robots.txt it assumes everything on your site is freely crawable, if it is you don't have the need for a robots file. If however you don't want it crawling a certain page you can indicate this in the file.

This page has more details (hope it is ok to list, sure i have seen it here before)

But to answer your quesion, the lack of a robots.txt file won't be hurting your rankings.

12:53 pm on Dec 13, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 22, 2002
votes: 0

At the top of this page there is a lonk named "Glossary"

Click that and look ofr robots.txt - click it.

There is a description and a link to the full spec.