Welcome to WebmasterWorld Guest from

Forum Moderators: brotherhood of lan & mack

Message Too Old, No Replies

How do i do a robot.txt

What does it do?



11:48 am on Dec 13, 2005 (gmt 0)

10+ Year Member

What is robot.txt.
I am confused i have been doing the site for a year now and have a feeling the reason my site is badly performing in the SE's is the robopt.txt has not been done.
Also what is the best way of making you sitemap as i have done an xml sitemap with a third party tool and dont think it is doing the right job.?
Any help much appreciated
Very keen, Youngkeen


12:45 pm on Dec 13, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

robots.txt is simply a text file that you place in the root of your web folder that gives instructions to spiders such as Googles Googlebot - [google.com...] or Yahoos Slurp - [help.yahoo.com...] what it can and can't crawl.

By default if you don't have a robots.txt it assumes everything on your site is freely crawable, if it is you don't have the need for a robots file. If however you don't want it crawling a certain page you can indicate this in the file.

This page has more details (hope it is ok to list, sure i have seen it here before)

But to answer your quesion, the lack of a robots.txt file won't be hurting your rankings.


12:53 pm on Dec 13, 2005 (gmt 0)

10+ Year Member

At the top of this page there is a lonk named "Glossary"

Click that and look ofr robots.txt - click it.

There is a description and a link to the full spec.



Featured Threads

Hot Threads This Week

Hot Threads This Month