I would documented everything as I go.
I am a hobbyist running a proxmox server with a docker host for media server, a plex host, a nas host, and home assistant host.
I feel if It were to break It would take me a long time to rebuild.
Ansible everything and automate as you go. It is slower, but if it’s not your first time setting something up it’s not too bad. Right now I literally couldn’t care less if the SD on one of my raspberry pi’s dies. Or my monitoring backend needs to be reinstalled.
IMO ansible is over kill for my homelab. All of my docker containers live on two servers. One remote and one at home. Both are built with docker compose and are backed up along with their data weekly to both servers and third party cloud backup. In the event one of them fails I have two copies of the data and could have everything back up and running in under 30 minutes.
I also don’t like that Ansible is owned by RedHat. They’ve shown recently they have zero care for their users.
I didnlt realize that about ansible. I’ve always thought it was overkill for me as well, but I figured i’d learn it eventually. Not anymore lol.
if by “their users” you mean people who use rebuilds of RHEL ig
deleted by creator
Go with used & refurb business PCs right out of the gate instead of fucking around with SBCs like the Pi.
Go with “1-liter” aka Ultra Small Form Factor right away instead of starting with SFF. (I don’t have a permanent residence at the moment so this makes sense for me)
Ah, but now you have a stack of PiS to screw around with, separate from all the stuff you actually use.
I should have learned Ansible earlier.
Docker compose helped me get started with containers but I kept having to push out new config files and manually cycle services. Now I have Ansible roles that can configure and deploy apps from scratch without me even needing to back up config files at all.
Most of my documentation has gone away entirely, I don’t need to remember things when they are defined in code.
I would have taken a deep dive into docker and containerised pretty much everything.
I’m mostly docker. I want to selfhost Lemmy but there’s no one-click Docker Compsoe / Portainer installer yet (for Swag / Nginx proxy manager) so I won’t until it’s ready
Same for me. I’ve known about Docker for many years now but never understood why I would want to use it when I can just as easily install things directly and just never touch them. Then I ran into dependency problems where two pieces of software required different versions of the same library. Docker just made this problem completely trivial.
Converting my environment to be mostly containerized was a bit of a slow process that taught me a lot, but now I can try out new applications and configurations at such an accelerated rate it’s crazy. Once I got the hang of Docker (and Ansible) it became so easy to try new things, tear them down and try again. Moving services around, backing up or restoring data is way easier.
I can’t overstate how impactful containerization has been to my self hosting workflow.
Same, but I’ve never once touched Docker and am doing everything old skool on top of Proxmox. Others may or may not like this approach, but it has many of the benefits in terms of productivity (ease of experimentation, migration, upgrade etc)
I already did a few months ago. My setup was a mess, everything tacked on the host OS, some stuff installed directly, others as docker, firewall was just a bunch of hand-written iptables rules…
I got a newer motherboard and CPU to replace my ageing i5-2500K, so I decided to start from scratch.
First order of business: Something to manage VMs and containers. Second: a decent firewall. Third: One app, one container.
I ended up with:
- Proxmox as VM and container manager
- OPNSense as firewall. Server has 3 network cards (1 built-in, 2 on PCIe slots), the 2 add-ons are passed through to OPNSense, the built in is for managing Proxmox and for the containers .
- A whole bunch of LXC containers running all sorts of stuff.
Things look a lot more professional and clean, and it’s all much easier to manage.
Does that setup allow access to PCIe GPUs for CUDA inference from containers or VMs?
Yes, you can pass through any GPU to containers pretty easily, and if you are starting with a new VM you can also pass through easily there, but if you are trying to use an existing VM you can run into problems.
Can’t say anything about CUDA because I don’t have Nvidia cards nor do I work with AI stuff, but I was able to pass the built-in GPU on my Ryzen 2600G to the Jellyfin container so it could do hardware transcoding of videos.
You need the drivers for the GPU installed on the host OS, then link the devices on /dev to the container. For AMD this is easy, bc the drivers are open source and included in the distro (Proxmox is Debian based), for Nvidia you’d have to deal with the proprietary stuff both on the host and on the containers.
I always redo it lol, which is kind of a waste but I enjoy it.
Maybe a related question is what I wish I could do if I had the time (which I will do eventually. Some I plan to do very soon):
- self host wireguard instead of using tailscale
- self host a ACME-like setup for self signed certificates for TLS and HTTPS
- self host encrypted git server for private stuff
- setup a file watcher on clients to sync my notes on-save automatically using rsync (yes I know I can use syncthing. Don’t wanna!)
Wireguard is super quick and easy to setup and use, I’d highly recommend to do that now. I don’t understand the recent obsession with Tailscale apart from bypassing cgNAT
Tailscale is an abstraction layer built on top of Wireguard. It handles things like assigning IP addresses, sharing public kegs, and building a mesh network without you having to do any manual work. People like easy solutions, which is why it’s popular.
To manually build a mesh with Wireguard, every node needs to have every other node listed as a peer in their config. I’ve done this manually before, or you could automate it (eg using Ansible or a tool specifically for Wireguard meshes). With Tailscale, you just log in using one of their client apps, and everything just works automatically.
self host wireguard instead of using tailscale
You cam self-host a Headscale server, which is an open-source implementation of the Tailscale server. The Tailscale client apps can connect to it.
What is the downside of using tailscale over wireguard?
I don’t think there’s any significant downsides. I suppose you are dependent on their infrastructure and uptime. If they ever go down, or for any reason stop offering their services, then you’re out of luck. But yeah that’s not significant.
The reason I want to do this is it gives me more control over the setup in case I ever wanted to customize it or the wireguard config, and also teaches me more in general, which will enable me to better debug.
I suppose you are dependent on their infrastructure and uptime
AFAIK their infra is only used for configuring the VPN. The VPN itself is a regular peer-to-peer Wireguard VPN. If their infra goes down while a VPN tunnel is connected, the tunnel should keep working. I’ve never tested that, though.
You can self-host your own Headscale server to avoid using their infra.
Actually plan things and research. Too many of my decisions come back to bite me because I don’t plan out stuff like networking, resources, hard drive layouts…
also documentation for sure
My current homelab is running on a single Dell R720xd with 12x6TB SAS HDDs. I have ESXi as the hypervisor with a pfsense gateway and a trueNAS core vm. It’s compact, has lots of redundancy, can run everything I want and more, has IPMI, and ECC RAM. Great, right?
Well, it sucks back about 300w at idle, sounds like a jet engine all the time, and having everything on one machine is fragile as hell.
Not to mention the Aruba Networks switch and Eaton UPS that are also loud.
I had to beg my dad to let it live at his house because no matter what I did: custom fan curves, better c-state management, a custom enclosure with sound isolation and ducting, I could not dump heat fast enough to make it quiet and it was driving me mad.
I’m in the process of doing it better. I’m going to build a small NAS using consumer hardware and big, quiet fans, I have a fanless N6005 box as a gateway, and I’m going to convert my old gaming machine to a hypervisor using proxmox, with each VM managed with either docker-compose, Ansible, or nixOS.
…and I’m now documenting everything.
I’ve had an R710 at the foot of my bed for the past 4 years and only decommissioned it a couple of months ago. I haven’t configured anything but I don’t really notice the noise. I can tell that it’s there but only when I listen for it. Different people are bothered by different sounds maybe?
I had an r710 before the r720xd. The r710 was totally fine, the r720xd is crazy loud.
That’s crazy to me! I had an R710 and that thing was so loud. I could hear it across the house.
For me if I can hear it at all when sitting near it in a quiet room, it’s a no-go.
I’m generally pretty happy with it, though I’d have used podman rather than docker if I were starting now.
Setup for high availability. I have a hard time taking things down now since other people rely on my setup being on.
Not accidentally buy a server that takes 2.5 inch hard drives. Currently I’m using some of the ones it came with and 2 WD Red drives that I just have sitting on top of the server with SATA extension cables going down to the server.
Get a more powerful but quieter device. My 10th gen NUC is loud and sluggish when a mobile client connects.
Make sure my proxmox desktop build can do GPU passthrough.
I’d have stuck with ZFS.
