Hello, I will soon have to setup a small GPU HPC cluster with few servers. It is not something I have ever done, I did setup servers and manage them but not sometof this scale with several users.
I’ve been looking at plausible alternatives and I am looking at openhpc, Ansible and NixOs.
Out of the three I feel like NixOs could be a good choice to have the configuration clearly stored somewhere. I won’t be actively maintaining the cluster at all times and find it nice that if someone needs some software I can point them to the configuration and tell them to modify it and then just update the system.
On the other hand I appreciate how openhpc is designed for that and definitely does support all use cases we need.
Ansible seems like an in between option.
Do you have advice on this choice? I read that NixOs can have problems with cuda drivers and custom compiled software, which would definitely be a deal breaker.
I found some HPC do use NixOs but it is definitely not a widespread option. Thus I’m a bit afraid I won’t be easily able to find solutions to problems I may encounter.
I have no experience with HPC clusters, just dropping in to contribute my opinions on the NixOS stuff.
I won’t be actively maintaining the cluster at all times and find it nice that if someone needs some software I can point them to the configuration and tell them to modify it and then just update the system.
This sounds good in theory, but in practice few people will have the know-how or patience to make changes to your config.
I read that NixOs can have problems with cuda drivers and custom compiled software, which would definitely be a deal breaker.
NixOS is very flexible and you can certainly bake custom compiled software into the builds, but expect a steep learning curve.
I’m not sure how Ansible made it on your list. Ansible lets you apply configuration and ensure applications are installed on groups of machines. It works across a variety of Linux distros.
ansible strenght is in configuring fleets of linux machines, lending it to being one of the best tools for clusters like this.
Ansible certainly works here, most implementations I’ve seen are terraform with ansible or cloudinit.
Most of my stack is containers, so I personally prefer cloudinit and a handoff to ansible.
I guess maybe then it does not really apply. The idea is to have one CPU of one of the servers dedicated to user login and then a queue manager such as slurm managing the jobs across all servers. I will probably need a per server configuration since hardware differs across them, but all of them need to be able to run the same software.
Firstly, you should check out what the organization you are building for uses. If they use Red Hat, or Ubuntu, then you should probably just build your solution on top of those operating systems (or a Red Hat clone like Rocky or Alma). Both of those are popular in many organizations, and it would probably be better to use what people are familiar with and know how to troubleshoot or work their way around. Potentially, they even have support contracts, giving them the option of calling the parent company for help.
Ansible is good. The learning curve is definitely less steep than Nixos, and it’s easier to teach people. One benefit is that you can reuse existing public roles and playbooks. For example this one: https://github.com/galaxyproject/ansible-slurm , which installs slurm.
You would probably have to write additional playbooks or roles to install nvidia drivers or configure the system, but then they can stick.
openhpc
Firstly, is there anything specific from here you need? Secondly, is there anything in here that’s not available in existing distro repositories, like Ubuntu or Red Hat’s?
It certainly looks like an interesting project, but a 3 node cluster is pretty small, and I find it hard to justify things like OpenMP/MPI, which is basically a special compiler that compiles programs to run across multiple machines at once. For that runtime to work, you do actually have to compile the programs, which are written for it, using it, which can require work on the side of the people who want to run applications or simulations.
The more likely setup, to me, is that Slurm is going to to run docker containers via Apptainer. Slurm would handle assigning containers to nodes based on free resources, but they wouldn’t actually share resources like memory or CPU. This setup is still plenty useful, and very common.
Original comment, from before I read that you already selected slurm below. This comment isn’t relevant, as I realized I was targeting the wrong things but I’ll just leave it here regardless.
For the platform itself, you should use either Kubernetes or Slurm. Slurm is popular in academia, and Kubernetes is popular in corporate, but they are used interchangably depending on specific needs.
Slurm is better for scheduled tasks. Like let’s say you want researchers at a school to be able to run a long running simulation. They can sign up, reserve time for that slurm cluster, and then send out a “job”, for that slurm cluster. The job will automatically be allocated to the node of the cluster with free resources, and then ran, and then stopped, and the researcher will receive the results back.
Kubernetes is better for persistent deployed services, like web services or the like. For example, AI inference. Kubernetes can also do batch jobs, but it doesn’t have the advanced time tracking or scheduling systems that slurm has (although sometimes people build that on top of Kubernetes in order to only deploy one clustered system).
Nixos is nice for configuration as code, but it lacks clustering or application/cluster orchestration features. You can use Nixos to deploy Kubernetes or Slurm, but I wouldn’t use it as the HPC platform itself. Nixos should work with Cuda in addition to that.
Thank you for the detailed explanation. This is for academia, the organization I’m building it for is my research group. About 5-10 people will have access to the system to begin with. Regarding their favored system and experience: most of my colleagues use Ubuntu or wsl on windows.
Regarding the need of openhpc, there may be things in there which are not available in repositories of most distributions. Most software we’ll have to run we can not install from repositories as we’d need custom compilation for the hardware. e.g stuff compiled with avx-512, specific cuda drivers, mps support. OpenMP is not something we’ll be able to use unless we obtain quite a few more nodes, but I don’t expect it to be a thing.
In the institutional cluster we use they have a custom built Gentoo distribution and for software they use easy build and EESSI to deploy the software, as well as manual compilation when modules are not available, and they provide the software to the users through LMOD. I do find EESSI and easy build quite nice since they provide ready made building recipes, I do find the use of LMOD quite tacky and it often starts having problems once you need more than 5 modules at once. Were I to find a better alternative to LMOD I’d take it any day. One important thing here is that we definitely want a way to install different software with conflicting library versions. Not as much as a large HPC cluster which needs tens of versions of the same software, but we’ll likely need to install about 50-100 different computing programs and it’s very unlikely they all support the same libc version when running at the needed optimization level. I’m not very familiar with the container approach to this kind of things. I have used containers for the deployment of applications, but not really to run calculations. Is is possible through containers to provide code compiled specifically for the hardware at hand? What is the advantage in using a container to say run gromacs, rather than pointing the user to the path of the compiled binary?
Regarding the kubernetes and slurm split. I definitely want to have some persistent services running: databases, a couple web applications and maybe a Jupiter webpage to visualize results. It could be nice to also be able to setup that allows persistent services to run: I have a couple applications which currently work around slurm by scheduling long running jobs and picking calculations to execute from a DB. However most calculations will be research simulations which at most execute for a few days.
I wouldn’t say it is very important to have the precise scheduling of slurm, but I definitely don’t want to manually handle a queue of jobs submitted by other people.
Containers are commonly used to distribute programs that depend on different library versions, including different libc versions.
And yes, you could compile software for specific hardware and then deploy it via containers.
What is the advantage in using a container to say run gromacs, rather than pointing the user to the path of the compiled binary
Different libc versions. There are also some sandboxing and security restrictions that are applied.
I definitely want to have some persistent services running: databases, a couple web applications and maybe a Jupiter webpage to visualize results.
Maybe docker swarm mode, with the nvidia container runtime is easier? It lets you spin up services on multiple nodes, which you could then load balance with a conventional reverse proxy. Although if you are trying to do a single instance of Jupyter, then I don’t know about it’s ability to serve multiple users, or the security of that setup. Usually, people go for things like JupyterHub + the Dockerspawner. Jupyterhub handles authentication or
There is also a Kubernetes spawner, or you can use an alternative web application that can dynamically spin up jupyterlab/notebooks, like Kubeflow, or coder, but I don’t think any of that is what you want in regards to that specific usecase for Jupyter.
If you just want to present rendered stuff, check out: https://quarto.org/ , which is a static site generator capable of executing jupyter notebooks, and rendering them to websites. It can also render I use it for my blog, which is ironic because I don’t actually use any of the data science features quarto has. I really like quarto because it’s the only static site generator which I found has fulltext search built in/easily enabled. It runs some javascript over a generated index, and you can search the website without any form of backend needed.