Hi, folks! I’ve learned a lot these past 8-9 months in self-hosting, but I’m not sure what’s going on with my reverse proxy setup. At this point, I’m only running Nginx Proxy Manager, Jellyfin, and Komga, separated from my home network via VLANs and firewall rules. My home network is 192.168.1.X and hosted services are on 192.168.10.X. Yet when I look at access logs for Jellyfin or Komga, both are reporting access from 192.168.65.1 when accessed from outside my home network. I don’t have a 65 subnet, so this is strange. The reverse proxy is run on a mini PC at .10.201, and Jellyfin and Komga are hosted on a NAS at .10.202; all of them run via Docker containers.
The instructions for Jellyfin for reverse proxy just say to list the address for the reverse proxy, which I did using IPv4 and the domain, and various configurations of only one or the other, as well as the 192.168.65.1 address that manifested out of nowhere. I haven’t observed any kind of change in IP address reporting after making changes here. I haven’t begun to try to configure Komga for the reverse proxy, because I figure until I get it working for Jellyfin and understand how to do it, it probably doesn’t matter; I only listed it here as evidence that two different services were reporting the same phantom IP address. I did ask for help on this issue on a Discord server a little while ago, and some of those folks suggested looking at the Docker networks. I left network settings on basically default for all three of these applications, and they all seem to generate something like a 172.17.0.X address, not the 192.168.65.1 that I’m seeing in logs.
I know there are supposed to be extra headers on network packets that these services can be aware of if they know it’s going through a reverse proxy first, and I figure this is important for proper monitoring of folks accessing my hosted services, so i want to get this figured out before I start running a few more apps. Any help anyone can offer would be appreciated. A lot of times, the documentation for this stuff is written up with an assumption of more understanding than the person reading it might have. Or maybe I just glanced right by something obvious and stupid.
I can take a look after work, but won’t that just change the proxy’s IP listed in my logs instead of properly forwarding along the source IP address?
Yep, you’ll have to fix that by telling your proxy to put the client IP in a header that the target app can recognize and use:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For
But, also, it’s a good idea to have Docker network ranges that don’t overlap your LAN. That would make me uncomfortable too.
You can also look into using IPv4 link-local addresses for Docker bridges (169.254.). Here’s an explanation:
https://github.com/11notes/RTFM/blob/master/linux/container/docker/daemon.md#networking
That frees you from accidental overlaps with pretty much everything else, but it comes with its own gotcha: link-local isn’t routable. This can come back to bite you if you have an app that relies on announcing a public UDP ip:port to a peer and announces the link-local IP. When a peer tries to connect back, it won’t work. So you have to either tweak the app’s config to make sure you announce a non-169.254. IP, or put the app container on a bridge/macvlan network so it auto-detects and announces a routable IP. There’s all kinds of apps that can be affected by this kind of peer-to-peer behavior (file transfer apps, games, VoIP apps etc.)
But TBH I’ve been using link-local ranges for a while now and only got bitten once (Deluge).
I thought that’s what I was trying to do by adjusting both NPM proxy host settings as well as informing Jellyfin where to expect proxy headers from.
Does it overlap my LAN? Is 192.168.65.X not distinguishable from my defined subnets with a 24-bit mask, now that I know where they’re coming from?
And to be honest, you’ve talked past my understanding with a lot of that paragraph, so I’ll have to comb through it later tonight when I have more time. Keep in mind that all of my Docker network settings are default, so it’s a setting I’ve never dabbled with. From what I can tell from the settings it defined on its own, these containers created a 172.17.0.X address, and somehow the proxy responds with (“announces”?) a 192.168.65.1 address. Are you saying that I need to define for it an unused address that’s on my existing 10 VLAN? Somehow all of the tutorials I’ve found in the past months to set up services behind a reverse proxy never mentioned any of these gotchas with Docker and networks.
Yeah that’s because most people are under the impression that Docker only allocates 172. addresses for its bridge network. I was under that impression too.
Turns out it also allocates 192.168 addresses.
The problem is that the default says
192.168.0.0/16, so it can also generate overlapping addresses. The fact it generated a 192.168.65.x address instead of an 192.168.1.x address was pure luck.The common-sense assumption is that you will adjust your Docker daemon settings so that you prevent overlap with your LAN. (Also, I expect lots of tutorials aren’t that high quality to begin with.)
If you edit daemon.json like I explained and explicitly only put 172. ranges in there you should be safe from overlaps in the future.
I would say a high quality tutorial doesn’t assume that I know anything about Docker, and most of them didn’t (which is good, because I don’t), but they also didn’t seem to expect me to want better logging out of my hosted services. Thanks, the /16 subnetting makes sense. I’ll try to edit that json file later today. But we seemingly don’t expect this to fix the problem by itself, so other than telling Jellyfin to expect a proxy at the new reported IP address, what should I expect for next steps?
Need to edit the proxy host for Jellyfin in NPM (advanced tab) and send the X-Forwaded-For header to it.
proxy_set_header X-Forwarded-For $remote_addr;And also in Jellyfin settings you have to set the IP that it sees the reverse proxy coming from as a “known proxy”, so it knows it can trust that header when it comes from that IP.
https://jellyfin.org/docs/general/post-install/networking/reverse-proxy/#forwarded-for-headers
Alright, I did all of the above. I added the proxy_set_header line to Custom Nginx Configuration for the Jellyfin proxy host by hitting the gear in the upper right of the edit window.
The daemon.json file wasn’t on my mini PC (where NPM lives), so I made a new one with those contents sans the 192.168 line. It was already present on my NAS (where Jellyfin lives), but only with a few other lines that weren’t our default-address-pools. I appreciate your previous instructions not assuming that I know the structure of a JSON file, but I do work with them a lot for my day job, so I was comfortable appending to that file on the NAS (though I did have to re-learn some vim basics to do it).
Anyway, after making those daemon.json file changes, I rebooted the whole machine in both places, not just Docker. That should be enough to make any IP addresses refresh themselves, right? But Jellyfin logs still show access coming from 192.168.65.1. I do find it strange that a container would be using an IP address like this one without reporting it somewhere, but in the Inspect tab of Docker Desktop on the mini PC, it just shows that 172.17.0.X address (2 for IPAddress, 1 for Gateway).