homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

General Guide to Robots.txt
is there one here :)

 1:52 pm on Jul 6, 2004 (gmt 0)


I am creating a website and have read about robot.txt etc and after reading alot i have become very confused!

can anyone point me in the direction of a good Guide?

Or is there one here?

I basicly want robots to go through all my site :)

I looked some robot.txt software but, again slightly confused!

Any help at all will be helpfull!

p.s. is it now necessary for you to have robots.txt?



 2:17 am on Jul 7, 2004 (gmt 0)

Try this [searchengineworld.com].

If you want to permit any robot to spider your entire site, you don't need a robots.txt. Robots.txt is exclusionary, if no robots.txt file is presented to spiders, they assume unfettered access. However, each spider that requests the missing robots.txt file will generate a 404 in your logs.

To create an appropriate robots.txt, use a plain text editor (NOT a word processor!) and create a file that contains these two lines:
User-agent: *

The wild card in the first line indicates that subsequent lines apply to ANY spider. The second line "disallows nothing," so your entire site is spidered.

Upload the file using your FTP program's ASCII transfer mode, to your site's root directory.

Not much to it really... :)


 2:20 am on Jul 7, 2004 (gmt 0)

It is not necessary to have one but it would save you some 404 errors if you would just put a blank text file there. You only need a robots.txt file if you have stuff you don't want spidered.


 7:51 pm on Jul 7, 2004 (gmt 0)

Thanks :D

I dont have a log as such, i use a free 1
(stat counter)

Its not a big site just wanting to know when it gets crawled and how often etc....


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved