There are many reasons why it is bad for everyone, but I will cite a few specifics.
To start with, in the absence of good alternatives, vast majority of pedestrians and cyclists would also be regular drivers. The only exceptions would be people who are too poor to avoid a car. If the drivers and the walkers are the same people then the people who regularly walk still suffer from slowed traffic because they still take the car for some trips.
Poor mobility for cars also translates to increased cost of living. Most businesses rely upon vehicle transport in some way or another and the increased cost of doing business gets passed onto the consumer.
In addition, the existence of high throughput streets with higher speed limits tends to concentrate traffic into specific predictable areas. When you reduce the throughput of those areas, traffic gets distributed onto more roads. The result is that cyclists and pedestrians are less able to avoid cars with strategic route planning.
I wouldn’t ride my bike nearly as often if cars could go 45 mph on Canadian streets.
To clarify, this isn’t every street in the USA, it’s only major thoroughfares. Most side streets in US cities have a 25-30 mph speed limit. In a world with well designed bike infrastructure we would have dedicated paths separated from traffic so you don’t have to share the road with fast moving cars.