Hello, I will soon have to setup a small GPU HPC cluster with few servers. It is not something I have ever done, I did setup servers and manage them but not sometof this scale with several users.
I’ve been looking at plausible alternatives and I am looking at openhpc, Ansible and NixOs.
Out of the three I feel like NixOs could be a good choice to have the configuration clearly stored somewhere. I won’t be actively maintaining the cluster at all times and find it nice that if someone needs some software I can point them to the configuration and tell them to modify it and then just update the system.
On the other hand I appreciate how openhpc is designed for that and definitely does support all use cases we need.
Ansible seems like an in between option.
Do you have advice on this choice? I read that NixOs can have problems with cuda drivers and custom compiled software, which would definitely be a deal breaker.
I found some HPC do use NixOs but it is definitely not a widespread option. Thus I’m a bit afraid I won’t be easily able to find solutions to problems I may encounter.
I guess maybe then it does not really apply. The idea is to have one CPU of one of the servers dedicated to user login and then a queue manager such as slurm managing the jobs across all servers. I will probably need a per server configuration since hardware differs across them, but all of them need to be able to run the same software.