I’d say they’re post-scarcity anarchist. There’s no central/communal resource dispersal as needed for socialism, nor the central/communal resource allocation/planning needed for communism.
There’s seemingly no authority outside starfleet exerting any power, nor does anyone ever claim a motivation beyond exploration or study (to do something meaningful). The lack of money and unlimited access to replicated resources pending available dilithium also points to a society without exploitative discrepancies.
The humans also never are reported to have any resource hogging, the only tensions/stratification seem to be militarily (and against external parties also diplomatically), meritocratic, and even then the bottleneck seems mostly to be to not fall behind other races.
I don’t see neither capitalism, socialism, communism, despotism, theocracy, nor fascism, but many aspects of anarchism. If you’ve read anything about The Culture, they openly speak about being anarchist, and it’s very similar to Star Trek.