I am fairly new in the selfhosting business. I host a few services like Jellyfin or AdGuard, but most of my data just sits on plain SMB shares on my NAS.
I am interested in setting up things like Paperless-ngx, Nextcloud maybe Immich as well. One thing I am struggling with is how to organize all this so I don’t have the data scattered around multiple places or in the worst-case even duplicated. I like the simplicity of network shares and the fact that I am not relying on a third-party application to keep being maintained. I am currently not sure I am willing to give that up. However, I am also intrigued by the features these services provide. I know Nextcloud has the option to mount external storage, but I don’t know which downsides come with that. It gets more complicated with Paperless. As far as I know you typically have a consume directory where you throw in your data and Paperless stores it using its own system in the media directory. This means I either throw in everything or I suddenly have two locations where my documents are stored. If I would mount that media directory in things like Nextcloud I probably wouldn’t be able to find anything because of the different structure and naming scheme.
The idea that I had in mind is that I have a single directory structure of my data that can be used on its own and all these tools are just different frontends and provide different views and information of the same data. Maybe this approach is just something from the past and I should move on.
I should add that I am not planning to expose any of these services to the public. All of this is only accessed from inside my house or using a Wireguard VPN.
- How do you guys handle all this?
- How do you avoid data duplication?
- How do you avoid multiple potential file locations? Is a document in my Nextcloud, in my Paperless or just on the network share?
- Do you prepare in some ways in case an application stops being maintained?
- How has this reliance on multiple services impacted other things e.g. your backups?
@5ymm3trY@discuss.tchncs.de Excellent questions. I’m fairly new at this but I try to be organized. This may be a bit ‘meta’ for your taste but here it goes.
I use: Portainer to manage containers (FOSS, self-host of course!)
Stacks: Group by service importance or structure/db type I research each service thoroughly with AI for stability, activity and how it integrates into my stack and homelab ‘philosophy’ (see below)
Obsidian: to keep detailed notes of the entire stack (with canvas diagrams to define worfklows and keep stacks organized). I also have a LifeMap (how my homelab supports my life) section which AI vets my new services against.
Claude: as my advisor fore everything, pointed at my Obsidian vault. Makes for a much faster roundtrip, as I edit things in Obsidian (pick your .md editor) and Claude just advises/rewrites back to obsdian on the fly.
Main Setup:
Vault Folders:
Stacks: dawarich - mapping w/immich photo integration (on its own)
ghost - blogging platform, obsidian drives content high priorit backups
immich - photos high priority for backups
media - other less critical media tools i can bundle together lower priority backups since most of this stuff can be refound audiobookshelf calibre-web-nextgen jellyfin navidrome - higher priority backup of music yamtrack
tools - all the little workers doing interesting things into one stack that I can reload often without impacting more important services excalidraw immich-kiosk immich-public-proxy immich-reversegeo immich_power_tools linkding maloja miniflux miniflux-db netdata nextflux ntfy uptime-kuma rclone watchyourlan ytzero
wanderer - on its own due to complexity and more active usage by a hiking community, back end …py scripts integrating with immich, dawarich, gpslogger GPX uploads
File Structure: (all on SSDs)
Audio Audiobook backups (just a staging area for dumping stuff, it all goes to B2 via rclone) Book Course Film Image Immich Music Obsidian-RSS Podcast Show Video
Jellyfin and Immich point to the same folders I use Immich folder tools (can elaborate) to move photos / videos to match this structure, so they can both ‘share’ the same items. I find immich a wonerful front-end to crawl thru endless family videos before I process them in a video editor
Having a unified location for all your media files is really great when you start integrating your whole world holistically. I use whisper-ai to transcribe podcasts (audiobookshelf), youtube vides (YTZero), etc. and I always know where everything is.
When I add a new service (say paperless-ngx or stirling-pdf, my next project), I first ask Claude to review my stack documentation, my homelab specs (and security, network, cloudflare external, etc. setup) and then help me decide what service to pick of the basket of FOSS document management tools. I also use sumguy.com articles to review pros and cons of various services, quite helpful.
Then I get Claude to give an implementation doc with .yml, envars and all the tweaks documented in Obsidian. Then I test, and when stable, finalize documentation and make sure Obsidian is updated, and Claude has access to that new ‘folder’ or stack information for the next round. That folder is like a MOC in Obsidian, a folder in Obsidian, a project in Claude and usually a folder on the SSD as well. I use the .canvas mindmaps to help me track it all.
Then, once in a while, I tell claude to build me a workflow document with all the .py scripts, cron jobs, etc. into a document. For example Health Workflow Pull data from fitbit, drop int Fitbaus, push to postgres, analyze with Grafana or similar. Then I have it convert that into a canvas document, which makes a mindmap of the whole thing I can’t tell you how helpful it is to push mindmap boxes around and update the workflow. Of course, n8n is next, I’m just not quite there yet. But n8n will be a breeze with all my documentation.
Good luck