Having fought for several days with this, I’d thought it would be worth sharing while it’s still fresh in what’s left of my mind:
Problem:
- OpenWRT router running latest available version (25.12) and Luci UI.
- IP cam that I want hardwired-only (Ethernet).
- Cam must have no access to internet or rest of LAN.
- Cam must only be accessed directly by the NVR software.
- Software (NVR and reverse proxy) run as docker containers on server PC.
- Any other LAN device (laptop, phone) must be able to access the NVR only via the reverse proxy.
Network topology:
- Cam is plugged physically in lan1 port on router.
- Server is plugged physically in lan3 port on router.
- The problem above basically means we need two tagged VLANs, one for regular LAN devices, and one just for NVR and camera.
Configuring the VLANs in OpenWRT:
Before anything else: try doing the next batch of configs from wireless if possible not Ethernet, in case you mess up and lose wired connectivity to the router.
- Go to Network > Interfaces > Devices and press “Configure” for the bridge that holds your router ports. This will typically be called “br-lan”.
- Go to the “Bridge VLAN filtering” tab and check “enable Vlan filtering”.
- Press the “Add” button twice to get two VLANs.
- First VLAN: set id=1, check “local”, mark lan1=off (not member), lan2=untagged, lan3=tagged (also primary), lan4=untagged. You can use any ID but “1” is traditional for the base LAN network.
- Second VLAN: set id=100, check “local”, mark lan1=untagged, lan2=off, lan3=tagged, lan4=off. Again, you can use any ID, but it’s traditional to avoid numbers under 10 because they’re typically used for core VLANs.
- Case in point, please note that my router has two physical NICs, one for LAN ports 1-4 and one for WAN, so I have a separate br-wan device just for wan. But some routers only have one physical NIC so they only have one bridge that covers both lan1-4 and wan ports. On these routers when you get to the filtering tab you will find two VLANs (1 and 2, typically) already set up. If this is the case add the VLAN 100 and modify VLAN 1 as above and mark wan as off in 100.
- After you press “save” you will notice two new virtual devices called “br-lan.1” and “br-lan.100” type VLAN 802.1q have appeared in the device list.
- Do not apply modifications yet. First go to Network > Interfaces, edit the “lan” interface and switch it from device “br-lan” to device “br-lan.1”, otherwise you may lose connectivity to router if you’re on Ethernet.
- If you’ve applied changes early and lose connectivity, don’t panic, just wait. In recent OpenWRT versions after 90s without confirmation from the UI the router will automatically rollback the last changes.
If all went well and you’re on wireless or one of the untagged Ethernet ports (lan2 or lan4) you should have retained connectivity to the router and wireless devices.
Configuring the NVR network in OpenWRT:
- Go to Network > Interfaces, click “Add new interface”.
- Call it “NVR”, protocol “static address”, device “br-lan.100”.
- Next, edit it and set up your desired IPv4 address and netmask.
- You MUST go to “firewall settings” and use the “custom” field to add a new “NVR” zone. alternatively you can go to Network > Fireall, create the NVR zone there, and choose it here.
- Enable DHCP if you want.
- If you do, please note that some routers bind dnsmasq only explicitly to select interfaces. Please check under Network > DHCP > dnsmasq > Devices & Ports and if “non-wildcard” is checked you will have to add “NVR” to the “Listen interfaces” to actually get DHCP services on that network.
- Under Network > Firewall you should have a zone called “NVR”. Set input/output/forward to accept/accept/reject.
- Under Network > Firewall > Traffic rules you have to add a new rule, calld it “NVR DHCP”, that says that protocol UDP, source zone “NVR”, destination “device (input)”, destination port 67, action “accept”, and under advanced restrict address family to IPv4. Traffic rules have priority over zone configuration so DHCP will work no matter how you fuck up your zone access.
- We will be skipping DNS because we don’t want the NVR network to benefit from any. But if we did we’d be doing the same we did for DHCP (got to Network > DNS to enable explicitly on NVR interface, and add a traffic rule for it).
Configuring the camera:
- Plug the camera into port lan1, it should pick up a DHCP address on the network you’ve defined for interface “NVR”.
- In order to be able to access that IP from your regular LAN to configure the camera with your phone you’ll have to temporarily add the “NVR” zone to the list of forward zonez of the “lan” zone.
- Would probably be a good idea to either configure the camera to a static IP or give it a static DHCP assignment based on the Ethernet MAC, so you know where to reach it from the NVR software.
- Remember to remove the “NVR” zone from the lan’s zone forwards when you’re done.
Configuring the server for tagged VLAN connectivity:
- Your server (port lan3) was marked for tagged VLAN connectivity in OpenWRT but it doesn’t (yet) actually use tagged connections. We have to fix that or it won’t be reachable.
- Feel free to mark lan3 as “untagged” on VLAN 1 to connect to it while you change the settings, but keep in mind that direct console access may be needed if you fuck up.
- My server runs Debian so configuration basically runs down to loading module 8021q (and adding it to /etc/modules just in case, although this should be largely automated), and editing /etc/network/interfaces. Feel free to adjust the example below to your needs:
# this will bring up eth0 but leave it unconfigured, merely as a support carrier for the VLANs
auto eth0
iface eth0 inet manual
# this will set up VLAN ID 1 and tell it to use DHCP
auto eth0.1
iface eth0.1 inet dhcp
# this will set up VLAN ID 100 with a static address
auto eth0.100
iface eth0.100 inet static
address 10.234.100.2/24
Configuring docker networks and containers:
- Remember I said the proxy and the NVR are running in docker containers. If you haven’t done anything fancy with them before, you were probably using
ports:to expose a port for the proxy and one for the NVR on the host’s LAN IP, and pointing the proxy to the NVR. - First, we will want to make an ipvlan or macvlan docker network that will use the eth0.100 interface. Containers that use this network will be placed in the NVR network/zone, and have their communications tagged with VLAN ID 100. Feel free to customize the network range. You can use either ipvlan or macvlan, the latter is not very useful since docker can’t do DHCP. Note the use of
aux-addressto reserve the IP you’ve assigned the camera, in case there’s potential overlap with the range you choose for automatic allocation. It’s essential that you use theeth0.100interface as parent.
docker network create --driver ipvlan \
--ip-range=10.234.100.97/27 --subnet=10.234.100.0/24 --gateway=10.234.100.1 \
--aux-address 'cam1=10.234.100.10' \
-o parent=eth0.100 nvr-vlan
- We also need a bridge network that will allow the proxy to see the NVR container, because once the NVR container is placed on the NVR network on the VLAN ID 100 it won’t be reachable directly by the proxy.
docker network create --driver bridge \
--opt com.docker.network.bridge.name=br-docker-proxy \
--ipv4 --subnet=172.23.1.0/24 --gateway=172.23.1.1 \
proxy-bridge
- Next, in the NVR container compose, join both proxy-bridge and nvr-lan:
services:
nvr:
networks:
nvr-vlan:
proxy-bridge:
hostname: nvr
networks:
nvr-vlan:
external: true
proxy-bridge:
external: true
- Also in the proxy container compose you will want to join the proxy-bridge network.
- Remove the
ports:directive from the NVR container, it doesn’t serve any purpose now. Containers on ipvlan/macvlan are reachable by their native port assignments on the networks they’ve joined. - Please note the
hostname: nvrin the NVR compose, that’s the name you will have to use in the proxy configuration to reach the NVR. Docker will supply a DNS resolve for this automatically. - The proxy doesn’t need to be on
nvr-vlan. I mean you can, but you need to keep the proxy accessible from the main network so it will keep usingports:as usual.
Bibliography:
- https://docs.docker.com/engine/network/drivers/ipvlan/#manually-create-8021q-links
- https://openwrt.org/docs/guide-user/network/vlan/switch_configuration
- https://openwrt.org/docs/guide-user/network/dsa/converting-to-dsa
- https://openwrt.org/docs/guide-user/network/dsa/dsa-mini-tutorial
- https://forum.openwrt.org/t/openwrt-vlan-tagging/208098
- https://computingforgeeks.com/how-to-configure-vlan-interface-on-debian/
- https://wiki.debian.org/NetworkConfiguration#Howto_use_vlan
- https://manpages.debian.org/trixie/ifupdown-ng-compat/interfaces.5.en.html
- https://deepwiki.com/openwrt/firewall4/6.1-zone-configuration
- https://deepwiki.com/openwrt/firewall4/7-rules-and-nat
This is a bit more complex that it needs to be
Why didn’t you just use the openwrt Firewall? Admittedly I just skimmed this but couldn’t just allowed your server to talk to the camera via a Firewall rule?
I did initially do it with the firewall alone. I created a “br-nvr” device, moved lan1 from br-lan to it, and used br-nvr as the device for the NVR interface and firewall zone, then selectively let my phone and the NVR app from the LAN zone access the camera ports with traffic rules.
Everything else about the interface and zone stayed the same as they are now. That’s what’s great about the OpenWRT abstractions. I really appreciated how easy it was to get things working with the firewall zones alone, don’t get me wrong.
The one major issue with that approach was that the NVR app is outside the NVR zone and I wanted it in there.
- It makes broadcasting a non-issue, (I really don’t want to have to learn how to do cross-network broadcasts and I understand they’re fraught with problems anyway).
- Better security with less complexity. A single camera can have like 3 ports that need to be made accessible, and different cameras will have different ports. Making and maintaining traffic rules for multiple cameras would rapidly turn into a nightmare.
With the NVR app in the same isolated network as the cameras they can do whatever they want in there without needing explicit rules.
But I couldn’t put the NVR docker container into the NVR network, because it lives on a machine on the LAN network, and you can’t have the host machine on one network and a app on it in another network, with a single physical cable… unless you use tagged VLANs.
There are also some potential annoyances in the future if I ever want to move cables around the ports or make more complex setups, the VLAN abstraction makes things easier.
I’m going to go out in a limb and die on it, that: why overcomplicate things with Docker containers?? I didn’t read through your post; The title, to me, doesn’t require it based on what you’re doing. I then read you’re Problem box and confirmed you don’t need NVR on Docker. Seems overkill or really, pointless.
A. Do you absolutely NEED the VLAN for your NVR? Why not just isolate the devices on the network with a bogus DNS record or FW rule blocking via MAC or IP? B. Why run NVR through Docker? I hate devs who only containerize and don’t allow for options. Deal killer, personally. But it it has a native option, why not run that? It’s just extra steps for traffic to pass through which feels like a complicated bottleneck.
Idk, you do you. I’m SURE I’ll get hate for this comment, but sometimes you need to take a steep back and think about what’s necessary vs what’s for fun for learning. I’d you can, hey great! If you can’t, or it breaks, how much time do you now want to spend learning how to troubleshoot?
Tagged VLANs would be needed regardless, because I have only one server with one physical connection and I wanted to have processes on it on 2 different networks.
Docker is neither here nor there, you’re right it’s not needed for the solution. I was using it anyway, I know lots of selfhosters do, and it does make it easy to create an ipvlan and put an app on it.
Docker is not the only way to achieve containerization but I do appreciate and use containerization (and virtualization). It lets the host OS stay simpler and cleaner and prevents the various apps from messing with it. It makes it easy to control each app’s environment. You can do app containers, system containers or VMs as needed. It makes it easy to back up, restore and reproduce an app and its state, independently of the host OS or any other app.
Abstractions help… they empower you to do more. You invest some time into learning, sure, but it pays off later in time saved managing and the ability to do more complex stuff faster.
Troubleshooting is what it is. Nothing’s perfect, you’re going to end up troubleshooting something sometime not matter what you use.
A. Do you absolutely NEED the VLAN for your NVR? Why not just isolate the devices on the network with a bogus DNS record or FW rule blocking via MAC or IP?
DNS doesn’t stop direct IP comms. I have seen a few devices that talked direct IP to bypass DNS blocks. as you said, a firewall rule is a far better solution.
However, I don’t trust my Chinese cameras enough to not rootkit their way across my network, so it has to be quarantined. a VLAN is one way to successfully do that.
personally I dislike the high maintenance costs of VLANs. That’s why I opted to run a completely different network for all my IOT devices including cameras. the DVR is on that network and proxied through a dual homed server that straddles the two networks. firewall rules in place that only allow specific ports through on specific devices to specific devices.
high maintenance costs of VLANs
Said no one ever, until now.
a VLAN is assigned to a specific port on the same subnet.
now I have to know which port is for that VLAN specifically.
now I have to segment my network to ensure I know which IPs are on that VLAN.
now I have to statically assign IPs.
now I need to replace my router and rebuild everything.
or I setup a separate network on a different subnet with firewall rules to ensure nothing can talk to anything outside of the private network and let dhcp do assignments.
hmmmm…which one is easier to maintain… I wonder…
VLANs literally exist because it would be too costly to run dedicated networking hardware in enterprise solutions. from a networking perspective in a selfhosting or small business environment VLANs are overkill and high maintenance.
Networking headaches are the main reason why I’ve done everything I can to get away from docker. I’ve had easier times spinning up an LXC container and building from source than I have had fighting with docker networking.
In this particular case it wasn’t Docker that gave me the headaches, it was OpenWRT and wrapping my head around tagged VLANs.
Once you have the VLANs working on the router and the tagged interfaces up on the server, pointing a Docker network or an LXC at the
eth0.100interface is equally easy.Now, when I first got the camera I was considering adding a secondary network card to the server and plugging the camera into that, so it would be directly hardwired into the machine running the NVR. If I had done that I was given to understand that taking ownership of a physical NIC would have been much easier with LXC than with Docker.
(We’ll never know because I couldn’t find the PCI network card.)
Are you familiar with the OSI model? I would learn layer 2 vs layer 3 before getting to far in the weeds
Interesting. Any reason to prefer the driver-level VLANs over the hardware switch? That’s how I’ve done it in the past, and also how the linked wiki article starts.
I’m not an expert so take this with a grain of salt, but it seemed to me that the driver approach is the more useful abstraction and also the more modern, and that the old one will get eventually phased out (or stay there under the hood, out of the way).
I did see that some tutorials started out with “figure out if you have a hardware switch” but this way I didn’t have to care.
I’m not sure what you mean by “driver level.” Configuring vlans in openwrt will configure the hardware. (It is really the Linux kernel that does the heavy lifting)
I don’t really know, either. “driver-level VLAN” is the terminology the OpenWRT wiki (linked) uses. Seems OP went for the strategy at the bottom, with going to the bridge settings and make it add a new interface. What I’ve done before, and what that article talks about at the top is, go to the Switch settings instead, and set the port to the corresponding VLAN, or to keep the tags… But maybe I misremember what I did and which settings I touched… Yet it seems to me those are two distinct ways of doing it. Either on the switch hardware inside of the OpenWRT device, or inside of the software bridge in OpenWRT.
https://openwrt.org/docs/guide-user/network/vlan/switch_configuration
So you don’t need a hardware switch
