Shrinking it (through established legal channels) is impeachment and removal which has a high bar. Enlarging it is just passing a law, which is only hard because the senate has a policy (not a law) to effectively not pass laws without supermajorities. The latter could be done with a simple majority of politicians with a spine.