As you said there is really nothing special about 10 and you could easily redefine prefixes to whatever base you want.
In base 12 a day has 20 hours, an hour has 50 minutes and a week has 120 hours. The numbers are as arbitrary as before. Except the 0th index is always 0 in these cases.
While the origin has been explained well, a base change really doesn’t solve the underlying problem with time as we define it. I don’t think this has a good solution and probably doesn’t need one.