| How do I apply the robots.txt?
Seems important and need to know more! |
askjoe
msg:1526961 | 6:17 pm on Mar 7, 2003 (gmt 0) |
I know that the robots.txt tells the spider where to go – what commands do I use to get it to best do its job? Is this a file that I must attach to my website – If I use front page how do I apply it there? What exactly needs to be done in order to fully maximize my listings with this feature? Your advice is greatly appreciated! Askjoe
|
heini
msg:1526962 | 6:23 pm on Mar 7, 2003 (gmt 0) |
AskJoe:
[searchengineworld.com...]
That gives you everything that you need.
Basically robots.txt must be in the root folder. It tells bots, what they are allowed to spider, it does not really tell them where to go though. How to set it up gets explained there very well.
Look under Examples
|
korkus2000
msg:1526963 | 6:24 pm on Mar 7, 2003 (gmt 0) |
Try this tutorial
[searchengineworld.com...] You can also look at these examples [webmasterworld.com...]
[google.com...] Make the file and upload it to your root directory. Thats it.
|
|
|
