I’m currently running Proxmox on a single Lenovo ThinkStation P330 Tiny. As my VMs workloads have grown, the 64GB of RAM is consistently sitting near capacity.
To spread the memory load and get proper HA, seamless VMs migration, and easier node maintenance, I’m considering adding two more matching units to build a 3-node cluster.
Planned Per-Node Specs:
. Chassis: Lenovo P330 Tiny . CPU: Intel Core i9-9900T ES . RAM: 64GB DDR4-3200 (running at 2666MHz platform limit) . Boot/Local Storage: 1TB NVMe
Networking: Upgraded to 2.5GbE
Shared Storage:
Synology NAS via iSCSI (SSD cache enabled) over 2.5GbE
Does this setup still offer good value for money, or are there better alternatives in this form factor?
I’d especially appreciate feedback on:
- 9900T ES stability: Long-term reliability and virtualization quirks under Proxmox.
- 2.5GbE bandwidth: Is it sufficient for 3 nodes sharing iSCSI, especially during backups or migrations?
- Synology iSCSI + SSD cache: Practical experience with this as primary VM shared storage across multiple P330 tiny nodes.
- Any other bottlenecks, hidden traps, or limitations I might be overlooking?
For iSCSI, ita best to dedicate a port from each host and enable jumbo frames (mtu 9000) to get the most out of it.
Also try to dedicate a port for proxmox vm usage. Vnet off these 3 connections too.
jumbo frames (mtu 9000) to get the most out of it
On dedicated network segments where only jumbo frames are configured at broadcast points, like a dedicated SAN, sure.
Not the best for multi-mode networks like most home labs.
iSCSI is already good at shoveling frames. The tiny benefit one can get from using jumbo frames isn’t really worth the problems it causes in multi mode networks.
I have a similar setup, 3 mini pcs (2 dell, 1 hp) with a second 2.5Gb interface on each and a synology nas.
Initially I used iscsi for storage but I had a couple of annoyances that I encountered:
- At that time, taking snapshots on vms using iscsi storage wasn’t possible.
- My Synology NAS doesn’t have a way to natively install a 2.5Gb interface, so I used a USB 2.5Gb NIC. Occasionally (like one or twice per year, from my recollection) the NIC would stop working and my VMs would lose their storage. Easily fixed by rebooting the Synology and VMs, but annoying none the less.
Because of that I ended up installing an SSD in each node and creating a CEPH storage cluster. No more issues with VMs losing storage and it gave me the ability to create snapshots.
My current environment is 17 LXCs and 4 VMs living in a ceph cluster that runs on each proxmox node. Using PBS, I have backup jobs scheduled across 4 days each week, 2 of the days backup to my nas and the other 2 backup to Backblaze.
I’ve never encountered any noticeable bottlenecks with my current environment so I think you’ll be just fine
I don’t know about proxmox but I have 6 Pi4 8GB, 5 lenovo Tiny (i5-4x and i7-8x), and 3 laptops in a k8s cluster and love it. I actually have too many hosts but it’s a lot of fun for me to play on them. Storage is longhorn for anything with a database, everything else is over NFS onto a box I built myself. If you use the *arrs try them over the network and watch the logs. They HATE network storage and will complain a lot. If you don’t have any issues there you should be good.
I haven’t had any issues and just use the 1gb network. As for form factor, I think it’s whatever you have room for. The laptops are annoying but they have the best hardware so I put up with it.
Appreciate the reply! I’ve been running this Tiny for 2 years now and it’s been rock solid. My main hesitation is really finding the right price-to-performance sweet spot between compute and storage. Does this 3-node + Synology setup feel like a sensible middle ground, or am I overpaying/overthinking it?
Everything is super expensive these days which is why I’ve got all kinds of nodes. I lurk marketplace, kijiji, etc for “broken” or “for parts” nodes. Laptops with broken keyboards, screens, no battery, etc are perfect for me. I7 11th gen was $90 CAD. Works perfectly for a node. Found a cheap tiny with a locked bios. Removed the CMOS battery to reset and I’m good to go.
If you’re not willing to do that, the working tinys are perfect in my opinion.
The working Synology seems extra to me. You could use software like ceph, or diy it. Storage boxes don’t need crazy specs, just sata ports and power. The only reason I’d consider it is if you just want something “simple that works” and you’re willing to pay for it.
That makes a lot of sense. I already have a Synology DS920+, so there’s no additional storage hardware cost on my end. The Tiny form factor is ideal for me mainly due to footprint and low noise, but total cost and idle power are definitely things I’m weighing.
Another route I’m considering is picking up just one more Tiny instead of two, and running an external Corosync QDevice directly on the DS920+ (via a lightweight Docker container) for the tiebreaker vote. That gives me a proper 2-node quorum without the hardware cost and power draw of a third physical node. Combined with the DS920+ for shared iSCSI storage, that might actually hit the sweet spot between cost, footprint, and HA.
If you only have 2 and 1 fails, you’ll need to make sure the remaining one can take on 100% of the workload. If you’ve already maxed one and want HA, I’d suggest adding 2 more. Otherwise you’ll need to decide what services are down while you repair
Keep in mind you can’t do a migration with device passthrough, even if the VM is shut down.
I use zfs sync every 15 minutes for my replication. No issues.
