There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

You just finished setting up all your services and it works fine - how do you now prepare for eventual drive failure?

I know that for data storage the best bet is a NAS and RAID1 or something in that vein, but what about all the docker containers you are running, carefully configured services on your rpi, installed *arr services on your PC, etc.?

Do you have a simple way to automate backups and re-installs of these as well or are you just resigned to having to eventually reconfigure them all when the SD card fails, your OS needs a reinstall or the disk dies?

guitarsarereal , (edited )

The most useful philosophy I’ve come across is “make the OS instance disposable.” That means an almost backups-first approach. Everything of importance to me is thoroughly backed up so once main box goes kaput, I just have to pull the most recent copy of the dataset and provision it on a new OS, maybe new hardware if needed. These days, it’s not that difficult. Docker makes scripting backups easy as pie. You write your docker-compose so all config and program state lives in a single directory. Back up the directory, and all you need to get up and running again with your services is access to Docker Hub to fetch the application code.

Some downsides with this approach (Docker’s security model sorta assumes you can secure/segment your home network better than most people are actually able to), but honestly, for throwing up a small local service quickly it’s kind of fantastic. Also, if you decide to move away from Docker the experience will give you insight into what amounts to program state for the applications you use which will make doing the same thing without Docker that much easier.

desentizised ,

I used to (over a span of about 4 years now) just rely on a RaidZ2 (ZFS) pool (faulted drive replacements never gave any issues) but I recently did an expansion of the array plus OS reinstall and only now am I starting to incorporate Docker containers into my workflows. The live data is in ~ and nightly rsynced onto the new larger RaidZ2 pool but there is also data on that pool which I’ve thus far never stored anywhere else.

So my answer to the question would be an off-site unraid install which is still in the works. This really will only be that. A catastophe insurance. I probably won’t even rely on parity drives there in order to maximize space since I already have double parity on ZFS.

As far as reinstallation goes, I don’t feel like restoring ~ and running docker compose for all the services again would be too much of a hassle.

ikidd ,
@ikidd@lemmy.world avatar

I run everything on a 2 node proxmox cluster with ZFS mirror volumes and replication of the VMs and CTs between them, run PBS with hourly snapshots, and sync that to multuple USB drives I swap off site.

The docker VM can be ZFS snapshotted before major updates so I can rollback.

twei ,

You should get another node, otherwise when node1 fails node2 will reboot itself and then do nothing because it has no quorum

ikidd ,
@ikidd@lemmy.world avatar

pvecm expected 1

twei ,

I know, but every time I had to do that it felt like it’s a jank solution. If you have a raspberry pi or smth like that you can also set it up as a qdevice.

…and if you’re completely fine with how it is you can also just leave it like it is

ikidd ,
@ikidd@lemmy.world avatar

So I started to write a reply that said basically that I was OK doing that manually, but thought that “hell, I have a PBS box on the network that would do that fine”. So it took about 3 minutes to install the corosync-qdevice packages on all three and enable it. Good to go.

Thanks for the kick in the ass.

ikidd ,
@ikidd@lemmy.world avatar

So since I now had a “quorate” cluster again, I thought I’d try out HA. I’d always been under the impression that unless you had a shared storage LUN, you couldn’t HA anything. But I thought I’d trigger a replication and then down the 2nd node just as a test. And lo and behold, the first node brought up my OPNsense VM from the replicated image about 2 minutes after the second node lost contact, and internet starts working again.

I’m really excited about having that feature working now. This was a good night, thank you.

twei ,

If you need another thing to do, you could try to make your opnsense HA and never have your internet stop working while rebooting a node. It’s pretty simple to set up, you might finish it in 1-2 evenings. Happy clustering!

ikidd ,
@ikidd@lemmy.world avatar

I’ll look into that. I did see the option in opnsense once upon a time but never investigated it.

vividspecter ,

I put all docker data in one directory (or rather, a btrfs subvolume) and both snapshot and back it up daily to multiple machines. docker-compose files are also kept in the same subvolume.

My latest server is NixOS, so I don’t even bother backing up the root subvolume, since the actual config is tracked on git and replicated on multiple machines. If I want to reinstall, I can just install NixOS and deploy the config, then just copy over the docker subvolume, and rebuild the containers. Some of this could be automated further (nixos-anywhere and disko look promising for the actual OS install) but my systems don’t typically break often enough for that to be a significant issue.

You can go even further and either just use nix for the services, or use nix to build containers themselves, but I have a working setup already and it’s good enough, and I can easily switch to another distribution if issues start occurring in NixOS.

Decronym Bot , (edited )

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters
Git Popular version control system, primarily for code
HA Home Assistant automation software
~ High Availability
LXC Linux Containers
NAS Network-Attached Storage
Plex Brand of media server package
RAID Redundant Array of Independent Disks for mass storage
RPi Raspberry Pi brand of SBC
SBC Single-Board Computer
SSD Solid State Drive mass storage

8 acronyms in this thread; the most compressed thread commented on today has 6 acronyms.

[Thread for this sub, first seen 18th Nov 2023, 10:35] [FAQ] [Full list] [Contact] [Source code]

ad_on_is ,
@ad_on_is@lemmy.world avatar

Most of the docker services use mounted folders/files, which I usually store in the users home folder /home/username/Docker/servicename.

Now, my personal habit of choice is to have user folders on a separate drive and mount them into /home/username. Additionally, one can also mount /var/lib/docker this way. I also spin up all of these services with portainer. The benefit is, if the system breaks, I don’t care that much, since everything is on a separate drive. In case of needing to re-setup everything again, I just spin up portainer again which does the rest.

However, this is not a backup, which should be done separately in one way or the other. But it’s for sure safer than putting all the trust into one drive/sdcard etc.

Appoxo ,
@Appoxo@lemmy.dbzer0.com avatar

My whole environment is in docker-compose which is “backed” to github.
My config/system drive is backed with veeam to one drive.
The backup is backed with rsync to another drive every week.

But: I only have a 1-drive NAS because I don’t have the place for a proper PC with drive caddies and a commercial nas (synology, qnap) are not my jam because I’d need a transcoding capable gpu and those models are overpriced for what I need.
And with plain debian I get unlimited system updates (per distro release) and learn linux along the way.

ssdfsdf3488sd ,

virtualize the machine with proxmox, use proxmox backup server, load vm on new system if you get catastrophic failure on the machine running the vm currently.

emax_gomax ,

I use docker so don’t really have to worry about reproducibility of the Services or configurations. Docker will fetch the right services and versions. I’ve documented the core configurations so I can set them back up relatively easily. Anything custom I haven’t documented I’ll just have to remember or find I need to reset up.

idunnololz ,
@idunnololz@lemmy.world avatar

I eat a cyanide tablet. Drive won’t fail on me if I’m dead. Taps temple

HeartyBeast ,
@HeartyBeast@kbin.social avatar

carefully configured services on your rpi

I have a back up on an SD Card waiting for the day the SD Card fails. Slot it in and reboot

desentizised ,

I recently “upgraded” one of my raspberrys SD cards to an industrial grade one. Seems to me like those are a lot slower but for that particular use case it doesnt matter to me. What matters is that the card doesn’t die. It runs noticeably cooler when lots of data is being written to it so I feel like I must be onto something there.

ShellMonkey , (edited )
@ShellMonkey@lemmy.socdojo.com avatar

Routine backups of the VM’s and raid disk for the hypervisor running them. If the box hosting the backups went screwy there’s a problem but with something like 20TB of space used copies off-box are a bit cumbersome. To that end I just manually copy the irreplaceable stuff to a separate external storage and wish the movies and stuff good luck.

It ends up with a situation though where I’d have to lose both the disks on the hypervisor and if that happened several disks on the NAS (12 disks in a ZFS pool with each vdev being a mirror pair) or for the whole pool to get screwed up to lose the VMs fully. Depending on the day I might lose up to a week of VM state though since they only do a full copy once a week.

Skies5394 ,

On my main server: I have my SSD RAID1 ZFS snapshots of my container appdata, VM VHDs and docker image, that is also backed up as a full backup once per night to the RAID10 array, then rsynced to the backup server which then is uploaded to the cloud.

The data on the RAID is backups, repos or media that I’ve deposited there for an extra copy it for serving via Plex/Jellyfin. I have extra copies of the data, and if I were to lose the array totally, I wouldn’t be pleased, but my personal pictures/videos wouldn’t be in danger.

I run two back up servers, which both upload to the cloud. One of which takes bare metal images of all my computers (sans servers bulk drives), the other which takes live folders.

This is more due to convenience so that I can pull a bare metal image to restore a device, or easily go find a file with versioning online if necessary on both accounts.

As a wise man said, you can never have too many backups.

ehrenschwan ,

I use duplicati for docker containers. You just host it in docker and attach all the persistent volumes from the other containers to it, then you can set up backup jobs for each.

drkt ,

configs are backed up I can spin up a new container in minutes, I just accept the manual labor. It’s probably a good thing to clean out the spiders and skeletons every now and then.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • random
  • [email protected]
  • lifeLocal
  • goranko
  • All magazines
    •