Welcome to WebmasterWorld Guest from 18.104.22.168
Still the question remain,if we not using robot.txt to block a and b sub-folder, will crawlers index all and flag empty sub-folder as 404 page?
, Google and others many well attempt to access higher level folders to see what the response is.
you do NOT then implement SEF URLs like
and feed those into a rewrite. Disaster strikes when
can search engines read that auto-generated index file? And, if so, will they proceed to crawl everything else in the directory?
joined:Dec 29, 2003