A mastodon DM is a private post. Setting a post to private before sending means only mentioned accounts can see it, and all mentioned accounts get notified. In a sense this enables both DMs and group messages too. But it’s not the most elegant interface for it tbf. A good app makes this feel more DM like, such as Ice Cubes on iOS or the equivalent on Android.