homepage Welcome to WebmasterWorld Guest from 54.161.202.234
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How do I apply the robots.txt?
Seems important and need to know more!
askjoe




msg:1526961
 6:17 pm on Mar 7, 2003 (gmt 0)

I know that the robots.txt tells the spider where to go what commands do I use to get it to best do its job? Is this a file that I must attach to my website If I use front page how do I apply it there?

What exactly needs to be done in order to fully maximize my listings with this feature? Your advice is greatly appreciated!

Askjoe

 

heini




msg:1526962
 6:23 pm on Mar 7, 2003 (gmt 0)

AskJoe:
[searchengineworld.com...]
That gives you everything that you need.
Basically robots.txt must be in the root folder. It tells bots, what they are allowed to spider, it does not really tell them where to go though.

How to set it up gets explained there very well.
Look under Examples

korkus2000




msg:1526963
 6:24 pm on Mar 7, 2003 (gmt 0)

Try this tutorial
[searchengineworld.com...]

You can also look at these examples

[webmasterworld.com...]
[google.com...]

Make the file and upload it to your root directory. Thats it.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved