Msg#: 4645761 posted 3:21 pm on Feb 16, 2014 (gmt 0)
any requests for index.html should simply be 301 redirected to the directory root url (trailing slash) and at that point the existence or content of the index.html file is irrelevant from a web perspective. next time googlebot crawls the index.html url, it will get the 301 response. on a subsequent request for the directory root url, the default directory index document will be served. eventually the index will reflect this change in your server response.
Msg#: 4645761 posted 4:16 pm on Feb 16, 2014 (gmt 0)
do search engines spider a page that is 301'd
They can't physically crawl it, because a 301 is obligatory. Either accept the redirect, or go away. But search engines never stop asking for old URLs-- especially if it's the root. They would ask for it even if nobody in the universe linked to it. Like asking for nonexistent index pages of known directories, only more so. Even humans do that pretty regularly.
Why do you want the redirect? Can't you quietly rewrite, and let the page live wherever it physically happens to live?