Running the right command on the wrong SSH session/machine.

I wanted to try inserting and removing kernel modules, so I looked around and thought "well, I don't have a USB stick in right now so I can safely try removing the usb kernel module." So I did that, and after pressing enter I realized my keyboard is connected with USB.

1. have an nvidia GPU

2. have Fedora

3. download RPM package of drivers for Red Hat (after all, Fedora and Red Hat are... compatible, right?)

4. Everything goes fine

5. Six months later, upgrade to a new version of Fedora

6. oops, kernel panic at boot after the upgrade, and no video to troubleshoot after UEFI boot

7. figure out how to boot into a recovery partition from UEFI

8. figure out how to enable a serial console over a USB device

9. figure out how to connect to the serial console from another computer using another USB device

10. figure out what the kernel panic is from (not the upgrade, but the driver which wasn't upgraded)

11. figure out how to uninstall the incorrectly installed driver

12. figure out how to install the correct driver

That was a fun three week OS upgrade.

I tried to install an OS to a USB stick. This is Kubuntu specific.

You need to create a GPT partition on the stick, then you should be able to just use the installer and install on another USB stick.

I went through it, selected the usb stick... was not sure if everything was right and went a menu back, was correct, went forth again, past the install target selection and installed.

Well... turns out the Kubuntu installer (Calamares) selects the first disk always. And that selection seems to reset to default when going a menu back....

I deleted my complete normal disk, with like everything I had.

No Backup no mercy. Luckily did one only a few weeks before. The first since half a year! Damn... had my uni stuff on Nextcloud, a lot of personal stuff synced to my phone with syncthing.

rm *.c when I meant rm *.o

ntfsclone /dev/sdc /dev/sdb

/dev/sdb was a blank filesystem and /dev/sdc was my Windows filesystem.

ntfsclone man page

It ran for less than a second and didn't take me long to figure out what happened. That's the story of how I stopped using Windows.

If you count Android too, then this: I got my first Android phone when I was 10 or 11 and rooted it on the first day of having it. This was during a time when we were all still using ClockworkMod because TWRP didn't exist yet, and I somehow ended up with a system without a kernel. Panic ensued, and I spent that entire night (like 10 hours) digging through xda in order to find a tutorial on how to get this damn phone to run again. Imagine having to tell your parents "I broke my phone I got yesterday." I did get it working at like 6:30 AM. Fun times.

An older friend of mine told me years back about an incident that happened on a university VAX running Unix. In those days, everyone was using vt100 terminals, and the disk drives weren't all that quick. He was working on his own terminal when without warning, he got this error when trying to run a common command (e.g. ls)

$ ls -l
sh: ls: command not found

So he went on over to the system admin's office, where he found the sysadmin and his assistant, staring at their terminal in frozen horror. Their screen had something like:

# rm -rf / tmp/*.log
^C^C^C^C^C^C^C^C^C^C
# ls -l
sh: ls: command not found
# stat /bin/ls
sh: stat: command not found

A few seconds after hitting return, and the rm command not finishing immediately, he realised about the errant space, and then madly hammered Ctrl-C to try to stop it. It turns out that the disk was slow enough that not everything was lost, and by careful use of the commands that hadn't been deleted, managed to copy the executables off another server without having to reinstall the OS.

I totally pulled a LTT and removed my kernel. >_< There was a "real time" kernel listed in apt, and I installed it because I was curious if it would reduce lock latency for a project I was working on. (I wasn't trying to solve a problem, just curious) It didn't and I figured it was probably a bad idea to leave it installed. So I did an apt remove, and the rest went something like this.

Apt: Are you sure you want to remove the your kernel? Y/N

Me: Oh jeez... I don't want to do that.

Motor Memory: Y

Apt: Are you really really sure? Your computer will not boot if you do this. Y/N

Me: Oh, crap! That's not what I meant to do. Definitely not!

Motor Memory: Y

Me: No! Why would my brain betray me!?

Fortunately this was on a PopOS machine, so I booted into the recovery partition. Even if fixing it only took a minute, I still felt very very dumb. >_<

The dumbest must have been when I went through the list of installed packages on Debian and removed everything named "python-..." since I don't program in python.

I’m still stuck in vi

Everyone here is talking about rm, but when’s the last time you dd’ed the wrong thing by accident?

You can get tripped up by tab completion, hda vs sda, sda vs sdb, flipping the articles around, he’ll, I’ve even blasted a good drive with /dev/random because I did t pay attention to what computer I’m logged into.

My killer app for multiple terminals open at once, weather through several ttys, xterms, tmux or the other one I don’t use was to type out my dd commands with a ls or something safe making in front of it while I look back and forth compulsively to verify that all the targets are correct.

I once removed all groups from my user by using usermod -g instead of usermod -G

Way back when I was just beginning to experiment with Linux back in the 90s I installed ZipSlack, which was a GUIless 100MB distro based on Slackware that ran from a folder on Windows. It was okay but I couldn't really do much with it and back then 100MB was a chunk of space, so i went to delete it. But i thought I would give it one last hurrah by deleting it from Linux. So I made use of the infamous rm -rf and sat there thinking "this is taking a long time"... then realised I had my Windows drive mounted as a sub folder and I was in the process of wiping my hard drive of everything!

How I lost a Postgres database:

1. Installed Postgres container without configuring a volume
2. Made a mental note that I need to configure a volume
3. After a few days of usage, restarted the container to configure the volume
4. ...
5. Acceptance

Reformatted windows and installed Linux.

Wait before the hate. This was the first time I did it and knew nothing about it and didn't know it would wipe my system. So I lost everything.

Trusting tab-completion and pressing enter just a tiny bit too early, resulting in overwriting the work of at least four hours, because the files' names started the same. That whilst trying to initiate a git repository to prevent that kind of mistake....

You know, I'm in a constant battle with bootloaders. I've both deleted grub multiple times and windows boot manager by accident and by believing "I could fix it by re-generating it"... More like re-installing.

I once tried to restore replication on a broken MySQL cluster by restoring the backup on the only good, running node.

I have a faint memory of once uninstalling python2 on an Ubuntu system trying to switch to python3. That was a fun learning moment.

Deleted the certs from the sshd daemon which locked me out of a remote server that required and a 2 hour drive to fix.

dd if=/dev/zero of=/dev/sda status=progress

hmm why is it so fast

OH

CTRL-C

and then a kernel panic yeah my fs was gone

"I have 200 GB of unused space in the windows partition, I'll just plug a live CD, divide that partition and merge it with the Linux one, ggez"

Yes, dividing the windows partition destroyed it. Yes, mixing the windows and ext destroyed Linux. I might have been able to recover something but I was like 18 and I just reinstalled windows in a fit of misdirected anger against Linux.

Not me, and not Linux, but a school mate found the following bash snippet online :(){ :|:& };:.

Naturally, he tried it on the SunOS servers we had access to for schoolwork. He got his account suspended for the rest of the year.

I think most Linux distros are configured to kill fork bombs nowadays.

I do dumb things like edit my network configuration do some stuff and log out. Then I can't login the next weekend because the IP address is wrong. Also:

Ifconfig eth0 down

And I am booted from ssh.

I installed timeshift to have a way to create restore points just in case I mess something up while fiddling with my Archbox.

I used it for a while before I decided to remove it. After that, I realized it didn't remove the "restore points" (I didn't fully understand how it worked) and thought it would be good idea to rm -rf /run/timeshift.

My whole /home was smited (it uses symlinks to create these "restore points"). Before I realized, it removed gigabytes of data.

Lesson learned: always understand how something works and always be careful when using rm -rf.

I distrohopped once and wanted to try OpenSuse Tumbleweed. Would have went really well if I didn't by accident deleted all my partitions...

I lost all of my curated music files I gathered over the span of the last 15 years.

I'll never get those back.

Due to some poorly placed quotes, I managed to create a subdirectory named ~ in my home folder. You can imagine what happened next. Luckily, I had just gotten my backup system up and running the day before, so nothing was lost.

Always do an echo first before you put rm in a script. You know the story.

My first experience with installing Linux on a hard drive involved wiping the wrong hard drive (my dad’s) and installing on it. Then panicking when Windows 95 didn’t boot up. Thank goodness my dad was understanding lol.

Trying to solve any Linux problem via ...

~Anti~ ~Commercial-AI~ ~license~ ~(CC~ ~BY-NC-SA~ ~4.0)~

I mounted a disk of a server in rescue mode, since I needed to extract everything (the provider didn't have the option to dump everything as a zip). Then installed an FTP server, added a user/pass, it worked.

But I couldn't access the files of the original disk, even though I could see them. So I just chgrp/chown the original files, since the disk was just "mounted" in the rescue disk /mnt, I thought it was alright (at the time I thought permissions were volatile, stored separately from the files). I could now download the entire disk, yay!

Upon booting the original disk again, a bunch of errors: shell not starting, tools not running, because they were owned by user and not root...

Well we reinstalled all the server from scratch that day.

i accidentally mark iwd as dependency rather than explicit. after that someday im removing a package depends on iwd, iwd itself been removed as a dependency as well. and im been blocked out of my remote machine until someday i have physical acces to it

I updated a manjaro system.

Wiping out Windows trying to install Linux seems to be a common thing. Did that while trying to install Ubuntu as a teen.
Funnily, I didn't check which Ubuntu iso I downloaded and ended up installing Ubuntu server. I should've noticed with the gui-less installer, but then I thought that it was just Linux being hightech.

Using sudo when it isn't necessary, and the real cannon: sudo su.. Adding sudo to your command lines indiscriminately causes files you create to be owned by root even though they are in your home directory, and then you end up using sudo to make changes to the files... and then the filesystem permissions cannot prevent you from successfully running an accidental "sudo rm -rf /" command.

Seriously... sudo is not a "habit" to develop in order to avoid dealing with filesystem permissions problems.

I accidentally overwrote /etc/passwd once and I allowed /boot to run out of space during a kernal update and I created a local user with the same user that was also on the realm/domain that I had joined and various bash script issues.
Some stuff I've had to fix that someone else did:

• named a file rm -rf
• rm -rf /bin instead of ./bin -- Also the fact that they had sudo was crazy and also I guess this was the second time
• chmod -R 777 /
• Various software bugs running swap out of space or hitting the inode limit by creating files over and over again with a timestamp in the filename and having to remove all of them because there was no backup to the OS
• Someone disabled SELinux because something wasn't working but didn't tell anyone -- ugh
• Compiled java because they googled some issue and followed some old tutorial without understanding anything instead of using alternatives and symlinked the old java from /bin to /home/theiruser/java -- had sudo because he was a Windows domain admin.
• Cybersecurity guy didn't know what some VMs did so he turned them off and figured he'd find out if/when someone complained. Caused a massive core services outage.
• Same Cybersecurity guy deleted a bunch of data because he wanted to see how the sysadmins would respond and witness backup restorations. He did not inform anyone.
• Cybersecurity guy above still has Domain Admin and sudo everywhere. I would have personally removed his privileged access regardless of what 'CyberSecurity' management thought but I was leaving for a new job by then anyway so I figured I'd just let them eventually lie in the bed they made.

There's more but I don't want to keep going because it is Sunday and I don't want to ruin it.

Tip: don't put important things in just 1 place.

That aside!

Years ago when I first tried out Linux (I was around the age of 10), I didn't really pay much attention while installing Linux back then, so I wiped my entire data disk D:...

The Arch installation tutorial I followed originally advised using LVM to have separate root and user logical volumes. However, after some time my root volume started getting full, so I figured I would take 10GB off of my home volume and add it to the root one. Simple, right?

It turns out that lvreduce --size 10G volgroup0/lv_home doesn't reduce the size by 10GB, it sets the absolute size to 10GB, and since I had way more than 10GB in that volume, it corrupted my entire system.

There was a warning message, but it seems my past years of Windows use still have me trained to reflexively ignore dire warnings, and so I did it anyway.

Since then I have learned enough to know that I really don't do anything with LVM, nor do I see much benefit to separate root/home partitions for desktop Linux use, so I reinstalled my system without LVM the next time around. This is, to date, the first and only time I have irreparably broken my Linux install.