The original Spanish Flu would absolutely fuck us up.
The reason it became mild has nothing to do with us building immunity but because a virus killing its host is bad for business (or strictly speaking, reproduction). A lack of viable hosts puts pressure on the virus's gene pool and in the end, the variant that is most successful at spreading and reproducing will win out.
That means not killing your host and only doing mild, repairable damage to other potential hosts so that humans don't take an infection so seriously.
We saw this exact pattern with COVID, with successive iterations being less deadly than the last.