Hi! So I’m considering…maybe having an NPU or something similar to be hooked to my proxmox server, which runs in a mini PC. It’s a EliteDesk 800 micro form factor. It has a Core i5 8500 CPU, which at the moment of purchase was good enough for live encoding HEVC video on Jellyfin…that was my main concern back then. But I’d like to consider the possibility of hooking maybe some docker instances or other containers to some local-only AI acceleration. Is there any NPU or cheap GPU I could hook on USB to this proxmox server to run? Has it been done before?

Thanks!

  • iturnedintoanewt@lemmy.worldOPEnglish
    1·
    3 hours ago

    Thanks! I was considering using it perhaps to have a diffusion model? Or running ollama or similar without a RAM hit on my laptop or phone. Also whisper comes to mind, for Bazarr or other tools to use.

    • hendrik@palaver.p3x.deEnglish
      2·
      2 hours ago

      Aren’t diffusion models and LLMs (ollama) too big for an external NPU? As far as I know something like a Coral runs specific models only. And it’s limited to the 1 or maybe 2GB of memory on it. It’d do tasks like voice recognition, or image classification. But not generate images or text.

      If you want to run arbitrary AI models and generative AI, I think you should be looking for a graphics card?!

      • frongt@lemmy.zipEnglish
        1·
        24 minutes ago

        Agreed. OP should probably upgrade to a bigger case and discrete graphics card.