I have setup a headscale server on my rented VPS. It uses a https domain and is working fine, despite the tunneling frequently fails. So I tried to also setup a custom DERP server.

And while I can connect to my DERP server from every node:

r/Tailscale - Custom DERP server running and connected, but can' t work as relay

But here is the problem: when I try to connect two nodes, the DERP isn' t working as a relay. I can tell this because those nodes being able to tunnel and directly connect can still connect to each other, while those previously can connect through tailscale official DERP servers now cannot.

So there seems to be something wrong with my custom DERP server, but I cannot figure it out. I need your help.

Here are some additional information:

I used the derper-docker for simplicity, and both headscale and derper runs behind an Nginx reverse proxy, if they would be a concern.

The DERP server is running fine, I can visit the DERP_DOMAIN and see that page.