The point of the firewall is not to make your computer an impenetrable fortress. It’s to block any implicit port openings you didn’t explicitly ask for.
Say you install a piece of software that, without your knowledge, decides to spin up an SSH server and start listening on port 22. Now you have that port open as a vector for malware to get in, and you are implicitly relying on that software to fend it off. If you instead have a firewall, and port 22 is not one of your allowed ports, the rogue software will hopefully take the hint and not spin up that server.
Generally you only want to open ports for specific processes that you want to transmit or listen on them. Once a port is bound to a process, it’s taken. Malware can’t just latch on without hijacking the program that already has it bound. And if that’s your fear, then you probably have a lot of way scarier theoretical attack vectors to sweat over in addition to this.
Yes, if you just leave a port wide open with nothing bound to it, either via actually having the port reserved or by linking the process to the port with a firewall rule, and you happened to get a piece of actual malware that scanned every port looking for an opening to sneak through, sure, it could. To my understanding, that’s not typically what you’re trying to stop with a firewall.
In some regards a firewall is like a padlock. It keeps out honest criminals. A determined criminal who really wants in will probably circumvent it. But many opportunistic criminals just looking for stuff not nailed down will probably leave it alone. Is the fact that people who know how to pick locks exist an excuse to stop locking things because “it’s all pointless anyway”?