Welcome to WebmasterWorld Guest from 54.196.232.162

Forum Moderators: goodroi

Message Too Old, No Replies

How do I apply the robots.txt?

Seems important and need to know more!

     
6:17 pm on Mar 7, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 2, 2003
posts:107
votes: 0


I know that the robots.txt tells the spider where to go what commands do I use to get it to best do its job? Is this a file that I must attach to my website If I use front page how do I apply it there?

What exactly needs to be done in order to fully maximize my listings with this feature? Your advice is greatly appreciated!

Askjoe

6:23 pm on Mar 7, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
posts:4404
votes: 0


AskJoe:
[searchengineworld.com...]
That gives you everything that you need.
Basically robots.txt must be in the root folder. It tells bots, what they are allowed to spider, it does not really tell them where to go though.

How to set it up gets explained there very well.
Look under Examples

6:24 pm on Mar 7, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member korkus2000 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 20, 2002
posts:3732
votes: 0


Try this tutorial
[searchengineworld.com...]

You can also look at these examples

[webmasterworld.com...]
[google.com...]

Make the file and upload it to your root directory. Thats it.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members