Hello, I will soon have to setup a small GPU HPC cluster with few servers. It is not something I have ever done, I did setup servers and manage them but not sometof this scale with several users.
I’ve been looking at plausible alternatives and I am looking at openhpc, Ansible and NixOs.
Out of the three I feel like NixOs could be a good choice to have the configuration clearly stored somewhere. I won’t be actively maintaining the cluster at all times and find it nice that if someone needs some software I can point them to the configuration and tell them to modify it and then just update the system.
On the other hand I appreciate how openhpc is designed for that and definitely does support all use cases we need.
Ansible seems like an in between option.
Do you have advice on this choice? I read that NixOs can have problems with cuda drivers and custom compiled software, which would definitely be a deal breaker.
I found some HPC do use NixOs but it is definitely not a widespread option. Thus I’m a bit afraid I won’t be easily able to find solutions to problems I may encounter.
Ansible certainly works here, most implementations I’ve seen are terraform with ansible or cloudinit.
Most of my stack is containers, so I personally prefer cloudinit and a handoff to ansible.