The ruling cellphone OS before Android and iOS was Symbian. Java was often used for apps. But when smartphones got more and more popular Symbian couldn’t keep up.
Apart from that there was Blackberry’s OS, Windows Mobile and Palm OS. And probably some others.