Hello, I will soon have to setup a small GPU HPC cluster with few servers. It is not something I have ever done, I did setup servers and manage them but not sometof this scale with several users.

I’ve been looking at plausible alternatives and I am looking at openhpc, Ansible and NixOs.

Out of the three I feel like NixOs could be a good choice to have the configuration clearly stored somewhere. I won’t be actively maintaining the cluster at all times and find it nice that if someone needs some software I can point them to the configuration and tell them to modify it and then just update the system.

On the other hand I appreciate how openhpc is designed for that and definitely does support all use cases we need.

Ansible seems like an in between option.

Do you have advice on this choice? I read that NixOs can have problems with cuda drivers and custom compiled software, which would definitely be a deal breaker.

I found some HPC do use NixOs but it is definitely not a widespread option. Thus I’m a bit afraid I won’t be easily able to find solutions to problems I may encounter.

  • ranzispa@mander.xyzOP
    1·
    1 day ago

    Thank you for the detailed explanation. This is for academia, the organization I’m building it for is my research group. About 5-10 people will have access to the system to begin with. Regarding their favored system and experience: most of my colleagues use Ubuntu or wsl on windows.

    Regarding the need of openhpc, there may be things in there which are not available in repositories of most distributions. Most software we’ll have to run we can not install from repositories as we’d need custom compilation for the hardware. e.g stuff compiled with avx-512, specific cuda drivers, mps support. OpenMP is not something we’ll be able to use unless we obtain quite a few more nodes, but I don’t expect it to be a thing.

    In the institutional cluster we use they have a custom built Gentoo distribution and for software they use easy build and EESSI to deploy the software, as well as manual compilation when modules are not available, and they provide the software to the users through LMOD. I do find EESSI and easy build quite nice since they provide ready made building recipes, I do find the use of LMOD quite tacky and it often starts having problems once you need more than 5 modules at once. Were I to find a better alternative to LMOD I’d take it any day. One important thing here is that we definitely want a way to install different software with conflicting library versions. Not as much as a large HPC cluster which needs tens of versions of the same software, but we’ll likely need to install about 50-100 different computing programs and it’s very unlikely they all support the same libc version when running at the needed optimization level. I’m not very familiar with the container approach to this kind of things. I have used containers for the deployment of applications, but not really to run calculations. Is is possible through containers to provide code compiled specifically for the hardware at hand? What is the advantage in using a container to say run gromacs, rather than pointing the user to the path of the compiled binary?

    Regarding the kubernetes and slurm split. I definitely want to have some persistent services running: databases, a couple web applications and maybe a Jupiter webpage to visualize results. It could be nice to also be able to setup that allows persistent services to run: I have a couple applications which currently work around slurm by scheduling long running jobs and picking calculations to execute from a DB. However most calculations will be research simulations which at most execute for a few days.

    I wouldn’t say it is very important to have the precise scheduling of slurm, but I definitely don’t want to manually handle a queue of jobs submitted by other people.

    • moonpiedumplings@programming.devEnglish
      1·
      5 hours ago

      Containers are commonly used to distribute programs that depend on different library versions, including different libc versions.

      And yes, you could compile software for specific hardware and then deploy it via containers.

      What is the advantage in using a container to say run gromacs, rather than pointing the user to the path of the compiled binary

      Different libc versions. There are also some sandboxing and security restrictions that are applied.

      I definitely want to have some persistent services running: databases, a couple web applications and maybe a Jupiter webpage to visualize results.

      Maybe docker swarm mode, with the nvidia container runtime is easier? It lets you spin up services on multiple nodes, which you could then load balance with a conventional reverse proxy. Although if you are trying to do a single instance of Jupyter, then I don’t know about it’s ability to serve multiple users, or the security of that setup. Usually, people go for things like JupyterHub + the Dockerspawner. Jupyterhub handles authentication or

      There is also a Kubernetes spawner, or you can use an alternative web application that can dynamically spin up jupyterlab/notebooks, like Kubeflow, or coder, but I don’t think any of that is what you want in regards to that specific usecase for Jupyter.

      If you just want to present rendered stuff, check out: https://quarto.org/ , which is a static site generator capable of executing jupyter notebooks, and rendering them to websites. It can also render I use it for my blog, which is ironic because I don’t actually use any of the data science features quarto has. I really like quarto because it’s the only static site generator which I found has fulltext search built in/easily enabled. It runs some javascript over a generated index, and you can search the website without any form of backend needed.