The secret chats feature isn’t between anyone I believe, it’s between two people. But I don’t actually know for certain because I’ve not looked into it beyond a cursory googling.
That said, you’d be correct in that just like any service out there, the moment you let random people join there’s no level of encryption that can keep your secrets secret.