Best I can tell from quick internet searches: Old English: wīfmann/menn (“female person/s”). The w rounded the following vowel giving a wo- pronunciation, which for some reason (umlaut?) stuck for the singular but not the plural. The spelling of the plural changed to match that of the singular in spite of the pronunciation.
Everything here carries the caveat “in some dialects, …” because English