Do you have a particular risk that you are worried about?
A couple of the Docker compose files I’ve used have non-hashed secrets in the compose itself. I am assuming, should someone penetrate the firewall, and gain access to Portainer somehow, they could see these compose entries just like I can. While I feel like I have adequately hardened the server (Lynis reports a score of 87) and I have rather robust ids/ips, firewall, and assorted accoutrements to support a secure server, there’s always that ‘what if’ scenario running in my brain and it causes doubt. Perhaps a secrets manager is over kill for a single user, docker container server.
How about a remote VPS?