Addressing only the mechanical aspects of the question:
Hunch: You've got an inappropriate use of belt-plus-suspenders. Or belt-plus-braces, depending on dialectal preference. The googlebot is going haywire because it isn't allowed into a directory, so it doesn't know that the subdirectories inside that directory don't exist any more.
Also, I set all user's folders to return a 404 error when accessed.
Normally you don't have to set a 404 explicitly when you're dealing with physical files. Did you really delete the directories? If so, a 410 will make the googlebot go away faster-- but only if you let it ask for the pages. You can remove the whole directory in gwt at the same time.
If this area was formerly visited by humans, make sure you've got a nice custom 410 page. Or at least use your existing 404 page. The Apache default 410 page is scary.
If the files are still there but you've changed them to restricted access, that should yield a 401 without any more work from your end.