36
Backups, backups, backups. A learning process (and small rant). (lemmy.ml)
submitted 10 months ago by [email protected] to c/[email protected]
40 comments fedilink hide all child comments

TL;DR @ bottom

Over the last year I've dove head first into selfhosting various services and apps on an UNRAID server. It's my first real experience with Linux short of fiddling with a dual boot of Pop!_OS. So that alone has already been quite the ride. Thankfully, UNRAID supports docker compose and has it's own Community App store where there are very few extra steps to get an app to work short of installing dependencies (MariaDB, Postgres, Redis, etc).

I am trying to be militant in my backups of mostly important files. Critical documents and photos, that sort of thing. Right now I have two copies on the same Win10 PC (two different drives) that backup nightly to iDrive. I recently started testing Backblaze Personal as well from the same computer. What I want to do is get to a point where the other Win10 PCs, MacOS laptops, and my yet to be built Linux desktop are all backed up to the server locally, then shuttled off to something like B2 on a nightly basis. Seemingly there is absolutely no consensus on the best cross platform app to backup the various clients. I currently have Time Machine backing up the laptops to the server, but that's it. Nothing else in the house is touching the server outside of Immich pulling photos from phones when we're back on network.

I keep trying to wrap my head around the CLI only options like Restic, Borg, and Kopia but I can't get to a point that I'm confident enough to ensure I'm backed up and safe as far as data integrity. So that led me to apps with GUIs. I've tried UrBackup, BackupPC, KopiaUI, and Duplicacy. That last one is the only one I can get to work reliably and that is backing up server data to B2 right now. I'm still in my trial period for it so nothing set in stone in that regard.

I did see Vorta for Borgbase, but it doesn't have a Win10. UrBackup refuses to work with MacOS for whatever reason. KopiaUI can't see directories on the server which I'm guessing is permissions issues and the client side won't accept an http://ip:port address, instead requiring https and I'm not sure if that's possible to get around. BackupPC I can't get to connect to the clients and that project seems somewhat stale with no updates in the last few years.

I'm comfortable hunting for solutions but seemingly the various support forums have a lot of assumptions in regards to prior knowledge of these products and I'm just not connecting the dots.

TL;DR: I apologize that this is so lengthy, but I'm honestly just not sure what to do at this point. I need (in my mind) the following as a solution:

  • Compatibility with Win10, MacOS, and Linux clients
  • Can back up to a local repository on UNRAID
  • Has a GUI (because I apparently am too stupid for CLI-only)
  • Would be nice to also backup to remote buckets like B2 or others but I can leverage something like Rclone GUI for that

Anyone have any advice on either solutions or resources I can dig into to accomplish this?

top 20 comments
sorted by: hot top controversial new old
[-] [email protected] 8 points 10 months ago

Backing up to a local server and then shunting off to B2 is largely what I do, except I just use a local file share and try not to store much locally. The Windows 10 built in backup works quite well for this. The file share I back up using Duplicati.

[-] [email protected] 2 points 10 months ago

I have not used Duplicati at all, so grain of salt. Every time I see it mentioned someone comes along and shares horrible experiences with corrupted backups leaving them unable to restore. Which has made me extremely leery of it. I'll have to research more on that and see whats what.

[-] [email protected] 3 points 10 months ago

I did have a problem with this, but it was due to a password manager (possibly Chrome's built-in one which I had a load of passwords duplicated in at one point) auto-filling a hidden field with the wrong password.

The front-end error is backup corrupt, but it's not, it's just the encryption key has been changed at the point of restore. Sort that and error gone.

[-] [email protected] 6 points 10 months ago* (last edited 10 months ago)

CLI is the best way... it's not difficult, borg is really easy to use, what are you hung up on? I can probably help if you have questions on it.

I don't use Windows, but you could syncthing important files to your server, then use borg to make legit backups of all the important files on your server. I store all my important files on one of my servers. Basically, all my PCs act more like thin clients... as far as files go anyway. I also backup my entire phone in a similar way as I'm suggesting you for Windows using syncthing for all my photos/videos/documents/etc.

If you want, you can read this which describes my entire backup solution.

I like borg because once you figure out what directories you want to backup, then you just put them in a script and schedule to run however often you like it. I also set healthchecks to notify me if it does not run... and for even further peace of mind, I have the output from the script sent to my Telegram just in case there were any errors.

Here is an example of my config with some changes made to post publicly. This is backing up all the important files from my main server to two other servers. You could also do the same to a local location and/or a remote location like borgbase or backblaze. This server has all the important files for everything, so this one script backs up all my docker services on this server, all my phone backups, all my desktop/laptop backups, and all my music/tv/movies. Literally everything except the docker services running on the other servers.

#!/bin/bash
#get date
NOW=$(date +"%y.%m")

#notify healthchecks script ran
curl http://thinkcentre.lan:7055/ping/W0WfGWIWDUDP5blEV00pOw/thinkbox-backup-script

#setup env
export BORG_RSH="ssh -i ~/somedir/thekey"
export BORG_PASSPHRASE="password"
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes


#stop all docker containers
docker stop $(docker ps -a -q)


#run nas backup to 5tb drive (connected to rockpro64 running dietpi)
borg create --stats [email protected]:/mnt/storage/borg::tv.$NOW /srv/dev-disk-by-uuid-70d630c3-5019-446a-ab57-ef796cfdab76/media/tv/completed.shows /srv/dev-disk-by-uuid-70d630c3-5019-446a-ab57-ef796cfdab76/media/tv/misc
borg create --stats [email protected]:/mnt/storage/borg::common.$NOW /srv/dev-disk-by-uuid-3a8447fe-a660-4ac7-ae46-2de19b6d59c1/
borg create --stats [email protected]:/mnt/storage/borg::music.$NOW /srv/dev-disk-by-uuid-70d630c3-5019-446a-ab57-ef796cfdab76/media/music
borg create --stats [email protected]:/mnt/storage/borg::public.$NOW /srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/public --exclude '/srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/public/downloads/*' --exclude '/srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/public/temp'
borg create --stats [email protected]:/mnt/storage/borg::backups.$NOW /srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/backups
borg create --stats [email protected]:/mnt/storage/borg::thinkb.home.$NOW /srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/home
borg create --stats [email protected]:/mnt/storage/borg::thinkb.docker.$NOW /srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/docker


#run nas backup to 2tb drive (connected to thinkc)
#this one excludes backups/.archive/borg for space
borg create --stats [email protected]:/shared/storage/borg::tv.$NOW /srv/dev-disk-by-uuid-70d630c3-5019-446a-ab57-ef796cfdab76/media/tv/completed.shows /srv/dev-disk-by-uuid-70d630c3-5019-446a-ab57-ef796cfdab76/media/tv/misc
borg create --stats [email protected]:/shared/storage/borg::common.$NOW /srv/dev-disk-by-uuid-3a8447fe-a660-4ac7-ae46-2de19b6d59c1/
borg create --stats [email protected]:/shared/storage/borg::music.$NOW /srv/dev-disk-by-uuid-70d630c3-5019-446a-ab57-ef796cfdab76/media/music
borg create --stats [email protected]:/shared/storage/borg::public.$NOW /srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/public --exclude '/srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/public/downloads/*' --exclude '/srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/public/temp'
borg create --stats [email protected]:/shared/storage/borg::backups.$NOW /srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/backups --exclude '/srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/backups/user/.archive/borg/*' 
borg create --stats [email protected]:/shared/storage/borg::thinkb.home.$NOW /srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/home
borg create --stats [email protected]:/shared/storage/borg::thinkb.docker.$NOW /srv/dev-disk-by-uuid-b5cb9459-0144-494b-af8c-89db1f742a83/docker


#these services need to start before other services that depend on them
docker start redis
docker start nginx
#start all the other services
docker start $(docker ps -a -q)
[-] [email protected] 4 points 10 months ago

That's the rub. I keep seeing that it's easy and the best way and folks fawn over Restic and Borg (for seemingly good reasons) so I start to read the documentation on setting it up and my eyes glaze over and I'm lost because there are assumptions made regarding the readers knowledge beforehand. Which is fair in this context. Even you example, which thank you by the way, assumes I know scripting. Yeah...not so much. I will look into what you've written and linked though. Maybe it'll click for me. I also looked at some of the wrappers for both Borg (borgmatic) and Restic (autorestic) and it's still over my head. I get the general concept that instead of writing it all down, you're creating configs ahead of time, similar to your script example. But I'm not at a spot where I can dive into that sort of thing yet. I need a ELI5, for CLI stuff I guess...

[-] [email protected] 2 points 10 months ago* (last edited 10 months ago)

You can get by with a GUI wrapper for borg, that is another option. I know OpenMediaVault even has it built in to the WebGUI, but I know you are using UnRaid so you'd have to see if they have anything. Otherwise you could maybe find a docker container for Vorta or something to spin up.

But on a server platform it will greatly benefit you to learn to use a terminal because pretty much all the best and most efficient tooling is going to use it. You may possibly see yourself running into this problem over and over where you want a GUI to do something when GUI tooling on the server platform just doesn't quite exist or isn't as efficient or reliable for many different reasons.

I would say just take small steps, don't try to take it all in at once. When you have free time, sit down with it, and just be patient and try to understand what you can, and if you get frustrated just put it down and walk away. It's not a race and you can always go back later. Also, trial and error in small tests with data that won't matter. That goes for any terminal based tooling. And every new tool you learn, it will get easier and easier.

And like I said before, if you do have questions feel free to hit me up. I started a selfhosting series on YouTube and have a few videos up, but I will actually be redoing it and trying to make it even more beginner friendly. Talking to someone that doesn't have much experience with a terminal may be beneficial to knowing what needs to be addressed in them.

[-] [email protected] 3 points 10 months ago

All valid points and I appreciate the input. My frustration comes from a lot of the tutorials online for these types of tools already start ahead of my knowledge. I have some terminal knowledge and have been messing around on. VPS I rent just so that if things go really bad, I'm not hosing my production server. And that's been ok. I've even stood up an instance of Miniflux and Wallabag for my Read It Later stack and it's working really well. I absolutely do need to learn more about using terminal though. I've always had good success learning by doing, with a project in mind rather than reading courses only and using their course materials for the actual practice portions. And example would be using digital footage of my own kids to learn video editing instead of the sample clips that came with the book/online class.

I'll take you up on the offer for help. I will try again and when I get stuck I'll shout. I appreciate your input!

[-] [email protected] 5 points 10 months ago

I have a NextCloud instance running with the client on my phone. The phone backs up all my pictures etc whenever I'm on the LAN and connected to a charger. My instance also accessible via VPN, though not otherwise internet-exposed.

You can also run a client to sync folders from Linux machines, and I'm fairly sure something exists for Windows as well. I don't really use these much as most of what I have on my desktops is either synced to git repositories (for code see) or is easily restored (Steam games+saves).

When setup with the office component, NextCloud allows you to edit documents from within the cloud environment, and there are a bunch of other apps you can install into your instance. It's kinda like a locally hosted instance of M365+OneDrive+More.

My repositories and the NextCloud instance are hosted on a box with SSD's in hardware RAID5, and I also run periodic backups (rsync) to a portable drive which is disconnected when not being used

[-] [email protected] 4 points 10 months ago* (last edited 10 months ago)

Look into restic, it doesn’t do exactly what you want, but it’s a very powerful backup server and standalone tool

edit typo

[-] [email protected] 2 points 10 months ago

Before clicking the link I thought there was a new restic clone/fork out there...

[-] [email protected] 1 points 10 months ago

My bad! Thanks for pointing out my typo

[-] [email protected] 1 points 10 months ago

Yeah, that's the struggle I'm having though. Understanding the CLI-only tools. I've heard great things about it though. I guess it's mostly I'm in over my head so much that I don't even know where to start on learning it since the actual Restic docs start with an unspoken understanding that you know how to script (to an extent).

[-] [email protected] 1 points 10 months ago

Deja dupe is a good gui for this

[-] [email protected] 4 points 10 months ago

Maybe you can sync all the clients with syncthing? It's cross platform and can be configured to keep copies on all of your machines.

[-] [email protected] 2 points 10 months ago

I did get Syncthing running at one point to test but more as a dropbox sort of replacement. Can you have it only write one way? So the clients would append only? I only did two devices and it was a bit cantankerous to get working, but I eventually got it.

[-] [email protected] 3 points 10 months ago

The last time I used it was months ago but I think I remember a client sided option to receive only.

[-] [email protected] 2 points 10 months ago

Can you have it only write one way?

This can be configured on the web UI at least.

[-] [email protected] 3 points 10 months ago

kiss (keep it simple )

rclone will do all that cli. Just write a simple script. Write once, run many.

Take the time to learn rclone cli and it will simplify everything

[-] [email protected] 2 points 10 months ago* (last edited 10 months ago)

I'm still procrastinating on figuring out the majority of my backup solution, mainly because I get overwhelmed too. But there is a backblaze personal docker template on the Unraid Community App store. You could do one way syncthing and have that backup to backblaze.

The only thing I currently backup offsite is an encrypted archive of my paperless-ngx content. I use rclone for that.

I found this guide on reddit that goes over borg and rclone. Seems to walk you through most of it. https://www.reddit.com/r/unRAID/comments/e6l4x6/tutorial_borg_rclone_v2_the_best_method_to/

But, i'm also making assumptions on your technical abilities, which I guess is part of the problem you have with cli tools....

[-] [email protected] 1 points 10 months ago

I'll have a look at that link, thank you. I actually did try the Backblaze Personal and it worked really well for about 2 weeks. Then it refused to see the D:\ drive, and no manner of uninstall/purge and reinstall would bring it back. No idea what happened but it scared me enough to back away from that option

load more comments
view more: next ›
this post was submitted on 05 Aug 2023
36 points (95.0% liked)

Linux

45443 readers
1312 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]