joined:Dec 28, 2013
I am trying to reconfigure a robots.txt file. I know this approach may be impossible but... I want to exclude everything except certain specified directories (instead of allowing everything except certain paths/files)
Consider this block:
This works except for one fatal flaw. It blocks the use of the default home page referenced by the url domain name alone, such as:
Since the 'index.htm' or whatever default file returned by the web-server is implied and not implicit the rule fails for the domain name by itself. I don't care much for the idea of allowing everything by default and then having to hunt down everything I don't want indexed/crawled. Whoever came up with this idea was creating crawlers
I know you can allow subdirs after a disallow statement but how then can you handle anything in the root? Hell, that's the one place I want to limit. It seems like it would be much simpler to be able to just list areas of a site you want crawled, not the other way around. Am I crazy? Or is this just stupid?
Any workarounds I can't see?