Was IE5/Mac the first official release with quirks mode, or are you counting betas and such? Had quirks mode already made its way into Mozilla milestone releases prior to that?
I believe that the first Mozilla milestone to implement Doctype switching was Milestone 14, which was completed on March 1, 2000. That was after IE 5 Mac by a little bit. However, before compatibility mode was tied into the Doctype, one could switch between modes via a menu item in older Mozilla builds.
An archived discussion of the development of Doctype switching in Mozilla can be read at: [bugzilla.mozilla.org...]