There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Best way to keep a hot spare SD card for a raspberry pi?

I have a load-bearing raspberry pi on my network - it runs a DNS server, zigbee2mqtt, unifi controller, and a restic rest server. This raspberry pi, as is tradition, boots from a microSD card. As we all know, microSD cards suck a little bit and die pretty often; I’ve personally had this happen not all that long ago.

I’d like to keep a reasonably up-to-date hot spare ready, so when it does give up the ghost I can just swap them out and move on with my life. I can think of a few ways to accomplish this, but I’m not really sure what’s the best:

  • The simplest is probably cron + dd, but I’m worried about filesystem corruption from imaging a running system and could this also wear out the spare card?
  • recreate partition structure, create an fstab with new UUIDs, rsync everything else. Backups are incremental and we won’t get filesystem corruption, but we still aren’t taking a point-in-time backup which means data files could be inconsistent with each other. (honestly unlikely with the services I’m running.)
  • Migrate to BTRFS or ZFS, send/receive snapshots. This would be annoying to set up because I’d need to switch the rpi’s filesystem, but once done I think this might be the best option? We get incremental updates, point-in-time backups, and even rollback on the original card if I want it.

I’m thinking out loud a little bit here, but do y’all have any thoughts? I think I’m leaning towards ZFS or BTRFS.

ShepherdPie ,

Ditch the SD card all together and get a cheap SSD to use as the boot drive instead.

Fiivemacs ,

Or go balls to the wall and get one of these bad boys

https://lemmy.ca/pictrs/image/f6be4949-6c97-49f4-8c9b-f72ba0998789.jpeg

traches OP ,

lol raid1c10

ShepherdPie ,

Damn bro, this is like seeing an 8-slot VCR or 50 disc CD changer.

Lifebandit666 ,

I can’t remember the steps (they were simple though) but when my Home Assistant raspi SD card died, I bought a 128gb SSD from AliExpress and a usb-sata cable.

I then did something to the pi that meant it can boot from the SSD, and flashed the SSD using Balenetcher or RUFUS or whatever (same program I was using to flash my SD cards basically).

Then it was just a case of plugging in and turning it on.

Runs exactly the same as with an SD card with less dying because SD cards aren’t meant for a lot of read/write but SSDs do.

Lifebandit666 ,

Just Google “Booting a pi from an SSD” and follow the steps

traches OP ,

That’s true, just booting from an SSD would be a lot more reliable and simple.

mgrimace ,

Another option is to use Image File Utilities on the Pi to create an image backup. You can use cron + a bash script to create incremental backups using the tool (e.g., take a ‘fresh’ backup each month, with daily incremental backups in between). I mount a network ‘backup’ drive (a local NAS, but you could use anything) to save the image to so I can actually access it. Then, just use balena etcher to flash the backup iso in the event of a failure.

PracticalChameleon ,

I also use those, but (shame on me) I’ve never tested the backups. Have you ever restored such an image backup?

mgrimace ,

Ha yes, once! I just used Balena Etcher and it worked exactly as expected.

traches OP ,

Oh cool, this is exactly what I was looking for! Thanks :)

mgrimace ,

Happy to help! There’s plenty of other options too (e.g., SSD) as folks mentioned, but this works well out of the box with no additional hardware. SD has been absolutely fine for my use (Pi-Hole), while still requiring maximum uptime so the family doesn’t riot if the internet is out.

Mellow12 ,

I’ve had very bad luck with raspberry Pi’s and SDCards. They just don’t seem to last very long. I swapped to usb storage and things got somewhat better. I just had a usb drive die after 3 to 4 years of use. When I was still using SD it seemed like multiple times a year. Heat. Power loss, you can only punch holes in silicon so many times before it wears out. Whatever the reason.

My approach for this is configuration backup not the entire os. I think this approach is better for when it’s time to upgrade the os or migrate to a new system.

For my basic Pi running WireGuard and DNS, I keep an archive of documentation on steps to reconfigure the system after a total loss. Static configs are backed up once, and If there are critical configuration items that change then I back those up weekly. I’ve got two systems (media related servers, not Pi’s) that I keep ansible playbooks to configure 90% of the system from scratch so it’s as hands off as it can be.

possiblylinux127 ,

If you are killing SD cards you shouldn’t be using a raspberry pi. They aren’t exactly work horses

traches OP ,

Yeah, I’m getting a pretty strong consensus here that an SSD is the way to go. I’ve also had at least one SD card die on me, and because I didn’t have backups it was pretty inconvenient. Had to recreate my homeassistant setup from scratch.

I get the config only backup, but when I have a mondohuge nas available and we’re dealing with like less than 100 gigs, why not just take a full disk image?

possiblylinux127 ,

Why are you using a MicroSD at all? Get a used minipc with two sata ssds or a single board computer with eMMC or mpcie support

dan ,
@dan@upvote.au avatar

Get a used minipc

Why, when the Pi is working fine? Just get a USB to SATA cable. This one works great with the Pi: a.co/d/8Jv2Erj

traches OP ,

Yeah, I definitely won’t be buying a new pi anytime soon for exactly that reason but I’ve had this one for awhile and would prefer to do something useful with it.

avidamoeba ,
@avidamoeba@lemmy.ca avatar

BTW in my anecdata I’ve yet to have a failure on any of my SanDisk Extreme Pro SD cards. I have 4 in active use on different Pi 4s. The oldest one is in use since 2018-ish. It used to be a NAS till 2022 so roughly 3-4 years of 24/7 use with a bog-standard OS. It’s now running OpenWrt which does fewer writes.

noxy ,
@noxy@yiffit.net avatar

I would ditch the SD cards entirely and boot off of USB attached SATA SSDs. But your idea still sounds cool if you can’t or don’t want to invest in SSDs!

I’ve enjoyed btrfs on my laptops, definitely seems stable, and using BEES foe dedupe is rad (maybe don’t do that on an sd card tho…)

traches OP ,

I’ve also been using BTRFS for awhile, and recently I’ve been getting into zfs which IMO does a better job of handling large software raid arrays. They’re both pretty great!

Shadow ,
@Shadow@lemmy.ca avatar

Why not just connect an ssd via USB and save yourself the hassle and torment?

avidamoeba ,
@avidamoeba@lemmy.ca avatar

That wouldn’t solve the problem though would it? It might make it less likely to fail but there’s still significant downtime if there’s no hot spare for this USB drive.

Shadow ,
@Shadow@lemmy.ca avatar

I couldn’t count the number of failed sd cards I’ve seen across all my fingers and toes.

I’ve seen like 4 ssds in my entire life fail. Plus you could just do mdraid 1 / btrfs across 2 of them if you want

avidamoeba ,
@avidamoeba@lemmy.ca avatar

Any failures of SanDisk Extreme Pro / Samsung Evo Plus?

I buy the redundancy argument. I’d still use ZFS for that if possible though. 😂 All my machines use mdraid 1 for their system drives but now that I know enough about ZFS, I’d likely use it on root next time around.

Shadow ,
@Shadow@lemmy.ca avatar

Yes. I’ve always splurged on nice cards for my personal stuff. I think it’s more about the write behavior of Linux than anything else, since I’ve never had a card die in my camera.

I refuse to use a pi with SD at this point. Saving $50 isn’t worth my time to reinstall things.

avidamoeba ,
@avidamoeba@lemmy.ca avatar

You’re making me nervous.

But then again, all my Pis are OpenWrt now which barely writes, so I’m probably fine. 😅

sugar_in_your_tea ,

Do you need a backup image?

For my NAS, all I do is:

  • keep notes of what’s installed and how to configure OS things
  • automatic, offsite backups of important configs and data

Any full-disk backups just make the restore process easier, they’re hardly the primary plan. If you want that, just take a manual backup like once a year, and maybe swap them out every 2-3 years (or however long you think the SD card should last). If you keep writes down, it should last quite a while (and nothing in your use-case seems write-heavy).

But honestly, you should always have a manual backup strategy in case something terrible happens (e.g. your house burns down). Make that your primary strategy, and hot spares would just be a time-saver for the more common case where HW fails.

traches OP ,

Well, this is my DNS server which means if it’s down the internet is down and I can’t resolve hostnames to ssh into. I know that can be worked around, but I’d really like a quick and easy fix that I could even talk someone through over the phone if I had to.

My real backups are squared away, no worries. Nightly automatic restic snapshots, one to an external drive on this very pi and another to a NAS at my parents’ house.

sugar_in_your_tea ,

I ended up making my router my DNS server, so if my router goes down, the internet is down anyway. I have static routes for things on my LAN, so if I hit mydomain.com, I can route it to an internal address instead of going over the internet. So far it works pretty well.

That said, I don’t have a PiHole setup, so I don’t know if that complicates things (I’m guessing pointing the router at the PiHole with a fallback to external DNS would just show ads or whatever if the PiHole is down).

But yeah, having a quick fallback is important. I think that should be as automatic as possible.

traches OP ,

I like the DNS on the router idea, I’ll look into it. I do have some split DNS set up as well as adblocking lists (technician). Not sure what my router can do.

sugar_in_your_tea ,

I think most can do it (esp. if you flash something like OpenWRT), but I have an entry-level enterprise router from Mikrotik and that’s a pretty standard feature on that tier.

dan ,
@dan@upvote.au avatar

Get a USB to SATA cable. This one works great with the Pi: a.co/d/8Jv2Erj

Instead of having a warm spare, a better solution is to attach two drives and use them in a RAID1 config. Unfortunately, I don’t think the Pi supports RAID1.

IsoKiero ,

Unfortunately, I don’t think the Pi supports RAID1.

I haven’t ran any Pi with hard drives, but I don’t see any reason why it wouldn’t work with software raid on linux.

dan ,
@dan@upvote.au avatar

The issue is that I don’t think its standard bootloader supports booting from RAID. I guess you could use a MicroSD for booting then have everything else on the RAID1. There’s unofficial ways to boot using GRUB, which should work with RAID1 too.

IsoKiero ,

Grub supports software raid just fine. The main issue is that you need to modify grub configuration to add bootloader on both drives, but even if you don’t it’s pretty simple to recreate needed files for second drive when the primary one dies.

dan ,
@dan@upvote.au avatar

GRUB does, but Raspberry Pis don’t use GRUB by default. You should be able to install it, but it’s not officially supported.

avidamoeba ,
@avidamoeba@lemmy.ca avatar

Perhaps the best answer by far is ZFS but I don’t know how much pain it is to set it up to boot from on a Pi. The easiest to setup is probably LVM.

With ZFS you can trivially keep a hot spare even over the network. Just tell syncoid where to replicate.

possiblylinux127 ,

ZFS isn’t going to perform well on a Raspberry Pi

avidamoeba ,
@avidamoeba@lemmy.ca avatar

I used it on a Pi 4 in 2019 for an USB-connected mirror and it worked well. Unencrypted throughput was upwards from 200MB/s. Encrypted throughput dropped down to under 100MB/s due to insufficient compute. The Pi 4 is a powerful computer and the Pi 5 even more so. Pi 3 and older, not so much.

traches OP ,

It’s a 4gb pi4, think it could boot from ZFS?

possiblylinux127 ,

You could boot from from an SD and then store data on a external drive. I would go Btrfs over ZFS unless you have at least 3 or more disks

avidamoeba ,
@avidamoeba@lemmy.ca avatar

I would try it. My only issue is I have no idea how to set it up on root on a Pi. Perhaps there’s docs somewhere. If had to setup a new Pi with Pi OS/Debian/Ubuntu today I’d definitely try it. Most of my Pis are running OpenWrt though.

traches OP ,

Performance is all but irrelevant in this case

possiblylinux127 ,

You say that but when the system starts to lock up you might change your tune

avidamoeba ,
@avidamoeba@lemmy.ca avatar

Why would it lock up? ZFS will use as much RAM as you give it and it doesn’t seem CPU-bound unless you turn on encryption. It’s not Ceph. Why do you expect ZFS to lock up and Btrfs not to?

Decronym Bot , (edited )

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters
DNS Domain Name Service/System
LVM (Linux) Logical Volume Manager for filesystem mapping
NAS Network-Attached Storage
PiHole Network-wide ad-blocker (DNS sinkhole)
RAID Redundant Array of Independent Disks for mass storage
SATA Serial AT Attachment interface for mass storage
SSD Solid State Drive mass storage
ZFS Solaris/Linux filesystem focusing on data integrity

[Thread for this sub, first seen 8th Aug 2024, 15:25] [FAQ] [Full list] [Contact] [Source code]

fuckwit_mcbumcrumble ,

Why not just buy a good SD card? My dashcam has been recording ~16/5 for the past 3 years onto a Sandisk extreme dashcam SD card and it’s still going strong with no issues. If it can survive the extreme heat and cold of being in a car I’m sure it will survive in a Pi just fine.

All of my SD cards that have failed have been bargain brand cards. None of my high quality ones have failed on me, I lose them before they go bad.

traches OP ,

I have a decent Samsung card in there now, it might survive. Can’t remember what brand the one that failed was, but I don’t tend to buy crappy ones

fuckwit_mcbumcrumble ,

Which cards are you using? Just because it’s samsung doesn’t mean it’s good.

My dashcam uses mostly this “SanDisk 256GB High Endurance Video” SD card, and my backup is a 512gb Samsung Pro Plus (not rated for dashcam use). For anything that I want reliability I use one of these SanDisk cards, that Samsung, or a SanDisk extreme that I bought the other day. My “I don’t really care about this, but I’d like it to not fail” cards are Samsung Evo Select drives (or something green and Samsung). Only my “I really don’t give a shit about these” drives are those $3 Microcenter cards.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •