Honestly, 4 seconds is really slow, especially with static HTML. I built my first companies’ site myself, it includes a video on the front page and jquery, is built by PHP, and on descent Internet connections the front page will load in slightly over a second, other pages dip under that.
There are loads of tweaks you can make to -any- site, and total amount of bytes really isn’t the only speed factor here.