The server is used for hole punching, to open up a P2P connection thorugh NATs and Firewalls. If it doesn’t work the server also relays the traffic between the clients.
Getting an end to end connection through todays internet is unfortunately not easy for an average user.