Hello, I will soon have to setup a small GPU HPC cluster with few servers. It is not something I have ever done, I did setup servers and manage them but not sometof this scale with several users.
I’ve been looking at plausible alternatives and I am looking at openhpc, Ansible and NixOs.
Out of the three I feel like NixOs could be a good choice to have the configuration clearly stored somewhere. I won’t be actively maintaining the cluster at all times and find it nice that if someone needs some software I can point them to the configuration and tell them to modify it and then just update the system.
On the other hand I appreciate how openhpc is designed for that and definitely does support all use cases we need.
Ansible seems like an in between option.
Do you have advice on this choice? I read that NixOs can have problems with cuda drivers and custom compiled software, which would definitely be a deal breaker.
I found some HPC do use NixOs but it is definitely not a widespread option. Thus I’m a bit afraid I won’t be easily able to find solutions to problems I may encounter.
I’m not sure how Ansible made it on your list. Ansible lets you apply configuration and ensure applications are installed on groups of machines. It works across a variety of Linux distros.
ansible strenght is in configuring fleets of linux machines, lending it to being one of the best tools for clusters like this.
Ansible certainly works here, most implementations I’ve seen are terraform with ansible or cloudinit.
Most of my stack is containers, so I personally prefer cloudinit and a handoff to ansible.
I guess maybe then it does not really apply. The idea is to have one CPU of one of the servers dedicated to user login and then a queue manager such as slurm managing the jobs across all servers. I will probably need a per server configuration since hardware differs across them, but all of them need to be able to run the same software.