Forum Moderators: phranque
a) Is there any reason I shouldn't divide things up among subdirectories on the server, e.g. images in one place, temporary html files in another, major html pages in the root, etc. This will keep things organized, but I'd like to know if there are drawbacks.
b) What is commonly hidden from the robots using the robots.txt file? Is there any point in hiding a lot of stuff (apart from the important html content pages, of course)? What I have in mind is that it might make it easier for search engine robots to find the important stuff if a lot extraneous subdirectories are hidden.
Thanks.
b) Blocking particular directories *could* make others more likely to be read, but only because of the law of averages, and the intuitiveness of the robot to reading your linking structure. (Open to thoughts to the contrary.)