Need help routing Wireguard container traffic through Gluetun container [Solved]

Decronym Bot , 3 days ago

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters	More Letters
DNS	Domain Name Service/System
HTTP	Hypertext Transfer Protocol, the Web
IP	Internet Protocol
NAT	Network Address Translation
VPN	Virtual Private Network
VPS	Virtual Private Server (opposed to shared hosting)

---

[Thread #885 for this sub, first seen 24th Jul 2024, 20:25] [FAQ] [Full list] [Contact] [Source code]

CumBroth , 9 days ago

Gluetun likely doesn’t have the proper firewall rules in place to enable this sort of traffic routing, simply because it’s made for another use case (using the container’s network stack directly with network_mode: “service:gluetun”).

Try to first get this setup working with two vanilla Wireguard containers (instead of Wireguard + gluetun). If it does, you’ll know that your Wireguard “server” container is properly set up. Then replace the second container that’s acting as a VPN client with gluetun and run tcpdump again. You likely need to add a postrouting masquerade rule on the NAT table.

Here’s my own working setup for reference.

Wireguard “server” container:

<span style="color:#323232;">[Interface]
</span><span style="color:#323232;">Address = <address>
</span><span style="color:#323232;">ListenPort = 51820
</span><span style="color:#323232;">PrivateKey = <privateKey>
</span><span style="color:#323232;">PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
</span><span style="color:#323232;">PostUp = wg set wg0 fwmark 51820
</span><span style="color:#323232;">PostUp = ip -4 route add 0.0.0.0/0 via 172.22.0.101 table 51820
</span><span style="color:#323232;">PostUp = ip -4 rule add not fwmark 51820 table 51820
</span><span style="color:#323232;">PostUp = ip -4 rule add table main suppress_prefixlength 0
</span><span style="color:#323232;">PostUp = ip route add 192.168.16.0/24 via 172.22.0.1
</span><span style="color:#323232;">PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip route del 192.168.16.0/24 via 172.22.0.1
</span><span style="color:#323232;">
</span><span style="color:#323232;">#peer configurations (clients) go here
</span>

and the Wireguard VPN client that I route traffic through:

<span style="color:#323232;"># Based on my VPN provider's configuration + additional firewall rules to route traffic correctly
</span><span style="color:#323232;">[Interface]
</span><span style="color:#323232;">PrivateKey = <key>
</span><span style="color:#323232;">Address = <address>
</span><span style="color:#323232;">DNS = 192.168.16.81 # local Adguard
</span><span style="color:#323232;">PostUp = iptables -t nat -A POSTROUTING -o wg+ -j MASQUERADE #Route traffic coming in from outside the container (host/other container)
</span><span style="color:#323232;">PreDown = iptables -t nat -D POSTROUTING -o wg+ -j MASQUERADE
</span><span style="color:#323232;">
</span><span style="color:#323232;">[Peer]
</span><span style="color:#323232;">PublicKey = <key>
</span><span style="color:#323232;">AllowedIPs = 0.0.0.0/0
</span><span style="color:#323232;">Endpoint = <endpoint_IP>:51820
</span>

Note the NAT MASQUERADE rule.

scapeg0at OP , 5 days ago

Thank you for the reply! I’ve been busy the last couple of days so I just got around to looking back at this.

I tested out your advice and setup a wireguard container with the MASQUERADE NAT rule and it worked! However, when I tried it out again with the gluetun container. I’m still running into issues, but there is progress!

With my setup before when I connect my client to the wireguard network I would get a “no network” error. Now when I try access the internet the connection times out. Still not ideal, but at least it’s a different error than before!

With the MASQUERADE NAT rule in place, running tcpdump on the docker network shows that at least the two containers are talking to each other:

<span style="color:#323232;">17:04:29.927415 IP 172.22.0.2 > 172.22.0.100: ICMP echo request, id 4, seq 9823, length 64
</span><span style="color:#323232;">17:04:29.927466 IP 172.22.0.100 > 172.22.0.2: ICMP echo reply, id 4, seq 9823, length 64
</span>

but I still cannot get any internet access through the wireguard tunnel.

When exploring around the gluetun config I confirmed that the MASQUERADE rule was actually set:

<span style="color:#323232;">Chain PREROUTING (policy ACCEPT 2933 packets, 316K bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain INPUT (policy ACCEPT 839 packets, 86643 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain OUTPUT (policy ACCEPT 12235 packets, 741K bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain POSTROUTING (policy ACCEPT 11408 packets, 687K bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;"> 2921  284K MASQUERADE  0    --  *      eth+    0.0.0.0/0            0.0.0.0/0
</span>

I think that the issue may be the default firewall rules of the gluetun block all traffic besides the VPN traffic via the main iptable:

<span style="color:#323232;">Chain INPUT (policy DROP 0 packets, 0 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;"> 2236  164K ACCEPT     0    --  lo     *       0.0.0.0/0            0.0.0.0/0
</span><span style="color:#323232;">11914   12M ACCEPT     0    --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
</span><span style="color:#323232;">   87 15792 ACCEPT     0    --  eth0   *       0.0.0.0/0            172.22.0.0/24
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain FORWARD (policy DROP 381 packets, 22780 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain OUTPUT (policy DROP 76 packets, 5396 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;"> 2236  164K ACCEPT     0    --  *      lo      0.0.0.0/0            0.0.0.0/0
</span><span style="color:#323232;"> 8152  872K ACCEPT     0    --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
</span><span style="color:#323232;">    0     0 ACCEPT     0    --  *      eth0    172.22.0.100         172.22.0.0/24
</span><span style="color:#323232;">    1   176 ACCEPT     17   --  *      eth0    0.0.0.0/0            213.152.187.229      udp dpt:1637
</span><span style="color:#323232;">  212 12843 ACCEPT     0    --  *      tun0    0.0.0.0/0            0.0.0.0/0
</span>

I tried adding simple iptables rules such as iptables -A FORWARD -i tun+ -j ACCEPT (and the same with eth+ as the interface) but with no luck.

If you think you can help I’ll be down to try out other solutions, or if you need more information I can post it when I have time. If you don’t think this will be an easy fix I can revert back to the wireguard-wireguard container setup since that worked. I tried to get this setup working so I could leverage the gluetun kill-switch/restart.

CumBroth , 4 days ago (edited 4 days ago)

I think you already have a kill-switch (of sorts) in place with the two Wireguard container setup, since your clients lose internet access (except to the local network, since there’s a separate route for that on the Wireguard “server” container") if any of the following happens:

• “Client” container is spun down
• The Wireguard interface inside the “client” container is spun down (you can try this out by execing wg-quick down wg0 inside the container)
• or even if the interface is up but the VPN connection is down (try changing the endpoint IP to a random one instead of the correct one provided by your VPN service provider)

I can’t be 100% sure, because I’m not a networking expert, but this seems like enough of a “kill-switch” to me. I’m not sure what you mean by leveraging the restart. One of the things that I found annoying about the Gluetun approach is that I would have to restart every container that depends on its network stack if Gluetun itself got restarted/updated.

But anyway, I went ahead and messed around on a VPS with the Wireguard+Gluetun approach and I got it working. I am using the latest versions of The Linuxserver.io Wireguard container and Gluetun at the time of writing. There are two things missing in the Gluetun firewall configuration you posted:

• A MASQUERADE rule on the tunnel, meaning the tun0 interface.
• Gluetun is configured to drop all FORWARD packets (filter table) by default. You’ll have to change that chain rule to ACCEPT. Again, I’m not a networking expert, so I’m not sure whether or not this compromises the kill-switch in any way, at least in any way that’s relevant to the desired setup/behavior. You could potentially set a more restrictive rule to only allow traffic coming in from <wireguard_container_IP>, but I’ll leave that up to you. You’ll also need to figure out the best way to persist the rules through container restarts.

First, here’s the docker compose setup I used:

<span style="color:#323232;">networks:
</span><span style="color:#323232;">  wghomenet:
</span><span style="color:#323232;">    name: wghomenet
</span><span style="color:#323232;">    ipam:
</span><span style="color:#323232;">      config:
</span><span style="color:#323232;">        - subnet: 172.22.0.0/24
</span><span style="color:#323232;">          gateway: 172.22.0.1
</span><span style="color:#323232;">
</span><span style="color:#323232;">services:
</span><span style="color:#323232;">  gluetun:
</span><span style="color:#323232;">    image: qmcgaw/gluetun
</span><span style="color:#323232;">    container_name: gluetun
</span><span style="color:#323232;">    cap_add:
</span><span style="color:#323232;">      - NET_ADMIN
</span><span style="color:#323232;">    devices:
</span><span style="color:#323232;">      - /dev/net/tun:/dev/net/tun
</span><span style="color:#323232;">    ports:
</span><span style="color:#323232;">      - 8888:8888/tcp # HTTP proxy
</span><span style="color:#323232;">      - 8388:8388/tcp # Shadowsocks
</span><span style="color:#323232;">      - 8388:8388/udp # Shadowsocks
</span><span style="color:#323232;">    volumes:
</span><span style="color:#323232;">      - ./config:/gluetun
</span><span style="color:#323232;">    environment:
</span><span style="color:#323232;">      - VPN_SERVICE_PROVIDER=<your stuff here>
</span><span style="color:#323232;">      - VPN_TYPE=wireguard
</span><span style="color:#323232;">      # - WIREGUARD_PRIVATE_KEY=<your stuff here>
</span><span style="color:#323232;">      # - WIREGUARD_PRESHARED_KEY=<your stuff here>
</span><span style="color:#323232;">      # - WIREGUARD_ADDRESSES=<your stuff here>
</span><span style="color:#323232;">      # - SERVER_COUNTRIES=<your stuff here>
</span><span style="color:#323232;">      # Timezone for accurate log times
</span><span style="color:#323232;">      - TZ= <your stuff here>
</span><span style="color:#323232;">      # Server list updater
</span><span style="color:#323232;">      # See https://github.com/qdm12/gluetun-wiki/blob/main/setup/servers.md#update-the-vpn-servers-list
</span><span style="color:#323232;">      - UPDATER_PERIOD=24h
</span><span style="color:#323232;">    sysctls:
</span><span style="color:#323232;">      - net.ipv4.conf.all.src_valid_mark=1
</span><span style="color:#323232;">    networks:
</span><span style="color:#323232;">      wghomenet:
</span><span style="color:#323232;">        ipv4_address: 172.22.0.101
</span><span style="color:#323232;">
</span><span style="color:#323232;">  wireguard-server:
</span><span style="color:#323232;">    image: lscr.io/linuxserver/wireguard
</span><span style="color:#323232;">    container_name: wireguard-server
</span><span style="color:#323232;">    cap_add:
</span><span style="color:#323232;">      - NET_ADMIN
</span><span style="color:#323232;">    environment:
</span><span style="color:#323232;">      - PUID=1000
</span><span style="color:#323232;">      - PGID=1001
</span><span style="color:#323232;">      - TZ=<your stuff here>
</span><span style="color:#323232;">      - INTERNAL_SUBNET=10.13.13.0
</span><span style="color:#323232;">      - PEERS=chromebook
</span><span style="color:#323232;">    volumes:
</span><span style="color:#323232;">      - ./config/wg-server:/config
</span><span style="color:#323232;">      - /lib/modules:/lib/modules #optional
</span><span style="color:#323232;">    restart: always
</span><span style="color:#323232;">    ports:
</span><span style="color:#323232;">      - 51820:51820/udp
</span><span style="color:#323232;">    networks:
</span><span style="color:#323232;">      wghomenet:
</span><span style="color:#323232;">        ipv4_address: 172.22.0.5
</span><span style="color:#323232;">    sysctls:
</span><span style="color:#323232;">      - net.ipv4.conf.all.src_valid_mark=1
</span>

You already have your “server” container properly configured. Now for Gluetun: I exec into the container docker exec -it gluetun sh. Then I set the MASQUERADE rule on the tunnel: iptables -t nat -A POSTROUTING -o tun+ -j MASQUERADE. And finally, I change the FORWARD chain policy in the filter table to ACCEPT iptables -t filter -P FORWARD ACCEPT.

Note on the last command: In my case I did iptables-legacy because all the rules were defined there already (iptables gives you a warning if that’s the case), but your container’s version may vary. I saw different behavior on the testing container I spun up on the VPS compared to the one I have running on my homelab.

Good luck, and let me know if you run into any issues!

EDIT: The rules look like this afterwards:

Output of iptables-legacy -vL -t filter:

<span style="color:#323232;">Chain INPUT (policy DROP 0 packets, 0 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">10710  788K ACCEPT     all  --  lo     any     anywhere             anywhere
</span><span style="color:#323232;">16698   14M ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
</span><span style="color:#323232;">    1    40 ACCEPT     all  --  eth0   any     anywhere             172.22.0.0/24
</span><span style="color:#323232;">
</span><span style="color:#323232;"># note the ACCEPT policy here
</span><span style="color:#323232;">Chain FORWARD (policy ACCEPT 3593 packets, 1681K bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain OUTPUT (policy DROP 0 packets, 0 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">10710  788K ACCEPT     all  --  any    lo      anywhere             anywhere
</span><span style="color:#323232;">13394 1518K ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
</span><span style="color:#323232;">    0     0 ACCEPT     all  --  any    eth0    dac4b9c06987         172.22.0.0/24
</span><span style="color:#323232;">    1   176 ACCEPT     udp  --  any    eth0    anywhere             connected-by.global-layer.com  udp dpt:1637
</span><span style="color:#323232;">  916 55072 ACCEPT     all  --  any    tun0    anywhere             anywhere
</span>

And the output of iptables -vL -t nat:

<span style="color:#323232;">Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">    0     0 DOCKER_OUTPUT  all  --  any    any     anywhere             127.0.0.11
</span><span style="color:#323232;">
</span><span style="color:#323232;"># note the MASQUERADE rule here
</span><span style="color:#323232;">Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">    0     0 DOCKER_POSTROUTING  all  --  any    any     anywhere             127.0.0.11
</span><span style="color:#323232;">  312 18936 MASQUERADE  all  --  any    tun+    anywhere             anywhere
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain DOCKER_OUTPUT (1 references)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">    0     0 DNAT       tcp  --  any    any     anywhere             127.0.0.11           tcp dpt:domain to:127.0.0.11:39905
</span><span style="color:#323232;">    0     0 DNAT       udp  --  any    any     anywhere             127.0.0.11           udp dpt:domain to:127.0.0.11:56734
</span><span style="color:#323232;">
</span><span style="color:#323232;">Chain DOCKER_POSTROUTING (1 references)
</span><span style="color:#323232;"> pkts bytes target     prot opt in     out     source               destination
</span><span style="color:#323232;">    0     0 SNAT       tcp  --  any    any     127.0.0.11           anywhere             tcp spt:39905 to::53
</span><span style="color:#323232;">    0     0 SNAT       udp  --  any    any     127.0.0.11           anywhere             udp spt:56734 to::53
</span><span style="color:#323232;">
</span>

scapeg0at OP , 3 days ago

I tried out your solution and it worked! I thought it was an iptables / firewall issue on the gluetun end but I didn’t know which table the packets were going through.

There’s also a way to set persistent iptable rules via gluetun, the docs are here

Thank you for your help! I’ll clean up my configs, and post my working configs and setup process!

NegativeLookBehind , 10 days ago

This might be a dumb question, but are the interface names correct in your configurations?