| 4:20 pm on Sep 26, 2003 (gmt 0)|
[searchengineworld.com...] - should help you with both Questions :)
| 4:59 pm on Sep 26, 2003 (gmt 0)|
Thanks for the reply Mike!
I had searched WW and read several other posts and also the robots.txt tutorial previous to posting question and could not find anything saying if it blocked home page or if I could block all pages written in ASP.
Any other suggestions completely appreciated! :)
Site is ready to fly and I want to make sure to please the SE God's. Only the finest Homemade Organic Cookies and Almond Milk for Ms. Googlebot! She works very hard and we got to keep her healthy ;)
| 6:14 pm on Sep 26, 2003 (gmt 0)|
If I were to put:
Would spiders still be able to access my home and links.html page?
Thanks again :)
| 8:05 pm on Sep 26, 2003 (gmt 0)|
If those page names don't start with something already in your list, then yes, they'd be spiderable. Robots.txt uses prefix matching. For example, the second of these two lines is redundant:
The first line blocks anything the second line also might block. You may find other opportuinties to make use of this prefix-matching to reduce the size of your robots.txt file.
Note that you should have a space after the ":"
As hinted at several times above, you need to read and fully understand the robots.txt standard [robotstxt.org]. We prefer to keep the discussion here general -- and useful to more than one person. Because of that, we prefer not to have "fix my site" threads.
Once you've got your file sorted out, validate it here [searchengineworld.com]
After you're comfortable with the basics, you might find this thread [webmasterworld.com] interesting.
| 10:34 pm on Sep 26, 2003 (gmt 0)|
You fast assistance and links are highly appreciated.
Out of curiosity if I LEFT OUT the part you noted
***The robots.txt should have a space after the ":" ***
Would ALL the pages on website be spiderable?
Once again..... Sincere Thanks :)
| 10:40 pm on Sep 26, 2003 (gmt 0)|
I don't know. Your robots.txt might be considered invalid by some robots. Then they would decide for themselves whether to spider your site - or not.
| 9:33 pm on Sep 27, 2003 (gmt 0)|
Thanks again Jim, and best of luck with all your endeavors :)