Article III only lays out there there will be a supreme court and a Chief justice and makes Congress responsible for establishing them. It does not lay out the makeup or structure of that court. The current body of 9 justices is set by federal statute and could be changed by a simple act of Congress.
Article III also explicitly states that whatever Justices are appointed hold their office as long as they maintain good behavior (I e., as long as they haven’t been impeached) and that Congress cannot reduce their pay.
Term limits are explicitly unconstitutional.
Setting the number of judges is explicitly within Congress’ constitutional powers.
Randomized panels would probably be challenged just because it’s never been tested, but the language in the Constitution re: Congress establishing the Supreme Court is vague. That said, Congress has already established inferior Federal courts that operate in this manner, so there’s precedent.