Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 130 additions & 0 deletions EMERGENCY_RECOVERY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# Emergency Recovery Guide for sliceanddice

## Problem Analysis

Your sliceanddice NixOS system is in emergency mode. Based on the repository analysis:

### Most Likely Causes:

1. **Missing disko.nix import**: If you pulled/applied configuration from the `development` branch (commit `ba8fb7cb` or later), it references `./disko.nix` which doesn't exist on `master` branch.

2. **Filesystem mounting failure**: The recent changes included:
- New disko partitioning scheme (btrfs subvolumes)
- Modified filesystem UUIDs and mount options
- New swap configuration

3. **Boot generation limit**: Commit `963b83f1` capped systemd-boot generations to 5 to prevent `/boot` from filling (511M EFI partition).

## Immediate Recovery Steps

### Option 1: Boot into a previous generation (RECOMMENDED FIRST STEP)

1. At boot, when you see the systemd-boot menu, press `Space` or `Esc` immediately
2. Select an older NixOS generation from before your recent changes
3. Press `Enter` to boot
4. Once booted, you can investigate what went wrong

### Option 2: Emergency mode recovery

If you're already in emergency mode:

```bash
# 1. Check which filesystems are mounted
mount | grep -E "/(boot|nix|home)"

# 2. Check for filesystem errors
journalctl -xb | grep -i "failed\|error\|emergency"

# 3. Check if /boot is full
df -h /boot

# 4. Check filesystem mounts status
systemctl list-units --type=mount --failed

# 5. Try to mount root as read-write if needed
mount -o remount,rw /
```

### Option 3: Rollback to stable master branch configuration

If you're using `/etc/nixos/.dotfiles`:

```bash
# In emergency mode or from an older generation:
cd /etc/nixos/.dotfiles

# Check current branch and status
git branch
git status

# If you're on development branch, switch to master
git checkout master
git pull origin master

# Rebuild with stable config
nixos-rebuild switch --flake /etc/nixos/.dotfiles#sliceanddice
```

## Known Issues from Recent Commits

### Issue 1: disko.nix import on master branch
- **Commit**: `ba8fb7cb` (on development branch only)
- **Problem**: Imports `./disko.nix` which doesn't exist on master
- **Solution**: Ensure you're using master branch, or cherry-pick disko.nix from development

### Issue 2: Boot partition space
- **Commit**: `963b83f1`
- **Problem**: 511M EFI partition fills with initrds (~50MB each)
- **Solution**: Already fixed with `boot.loader.systemd-boot.configurationLimit = 5`
- **Check**: Run `df -h /boot` to verify space

### Issue 3: Filesystem UUID changes
- **Location**: `hosts/nixos/sliceanddice/hardware-configuration.nix`
- **Current UUIDs**:
- Root: `b89f5dca-4b37-4062-bf1d-9e4ebfd61916` (ext4)
- Boot: `8824-4C5F` (vfat)
- Swap: `c570ec29-6025-456b-99d1-8f16b677835a`

## Verification Commands

Once you can boot normally, verify the configuration:

```bash
# Check current branch
cd /etc/nixos/.dotfiles && git branch

# Verify filesystem mounts
findmnt / /boot /home /nix

# Check boot partition space
df -h /boot
du -sh /boot/loader/entries/*

# List available boot generations
ls -lh /boot/loader/entries/

# Check system status
systemctl status
journalctl -p err -b
```

## Prevention

1. **Always test major changes**: Use `nixos-rebuild test` before `switch`
2. **Keep old generations**: The configurationLimit=5 is set, but you can temporarily increase it
3. **Monitor /boot space**: Run `df -h /boot` before rebuilds
4. **Stick to one branch**: Use either `master` or `development`, not a mix

## Contact Information

If you need to restore from scratch:
- Bootstrap script: `scripts/install-sliceanddice.sh`
- Secrets: `secrets/sliceanddice-secrets.yaml` (encrypted with sops)
- Hardware config: `hosts/nixos/sliceanddice/hardware-configuration.nix`

## Architecture Notes

- Laptop: MSI with Intel Tiger Lake UHD + NVIDIA RTX 3050 Ti Mobile
- Storage: Samsung 870 EVO 500GB (`ata-Samsung_SSD_870_EVO_500GB_S62ANJ0R238724D`)
- Graphics: Hybrid PRIME render offload (Intel for display, NVIDIA on-demand)
- Kernel: Latest mainline (`pkgs.linuxPackages_latest`)
94 changes: 94 additions & 0 deletions QUICKSTART.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Quick Recovery Guide - sliceanddice Emergency Mode

## Fastest Solution (Do This First!)

### Step 1: Boot Previous Generation

1. **Reboot** your sliceanddice machine
2. When you see the boot menu, press **Space** or **Esc**
3. **Select an older NixOS generation** (one from before the issue)
4. Press Enter to boot

That's it! You should now be back in a working system.

---

## Step 2: Prevent It From Happening Again

Once you're booted into the working generation:

```bash
# Go to your dotfiles
cd /etc/nixos/.dotfiles

# Check which branch you're on
git branch
# If you see "development", that's likely the issue

# Switch to stable master branch
git checkout master
git pull origin master

# Make this the current system
sudo nixos-rebuild switch --flake /etc/nixos/.dotfiles#sliceanddice

# Reboot to test
sudo reboot
```

---

## What Happened?

The `development` branch has new features but requires additional files:
- `disko.nix` - disk configuration
- `kbd-bl-ask.nix` - keyboard backlight
- MSI-specific kernel patches
- Power management features

If you rebuilt with the development branch but those files weren't present, NixOS couldn't complete the build and entered emergency mode.

---

## Alternative: Fix in Emergency Mode

If you can't access the boot menu:

```bash
# You should be at an emergency shell prompt
# Remount root as writable
mount -o remount,rw /

# Fix the configuration
cd /etc/nixos/.dotfiles
git checkout master
git reset --hard origin/master

# Rebuild
nixos-rebuild switch --flake .#sliceanddice

# Reboot
reboot
```

---

## Need More Help?

See the detailed guides:
- **SOLUTION.md** - Full explanation and multiple recovery options
- **EMERGENCY_RECOVERY.md** - Detailed technical recovery procedures
- **diagnose-emergency.sh** - Run this script to diagnose the issue

Or run the diagnostic script:
```bash
bash diagnose-emergency.sh
```

---

## Summary

**Problem**: Configuration mismatch between branches
**Solution**: Boot previous generation, switch to master branch
**Prevention**: Stick to one branch (master is stable)
173 changes: 173 additions & 0 deletions README-EMERGENCY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
# πŸ†˜ sliceanddice Emergency Mode - Recovery Instructions

> **STATUS**: Your sliceanddice NixOS system is in emergency mode
> **CAUSE**: Configuration mismatch between master and development branches
> **SOLUTION**: Multiple recovery paths available (choose the easiest one for you)

---

## πŸš€ Quick Fix (Recommended - 2 minutes)

### Option 1: Boot Previous Generation (FASTEST!)

1. **Reboot** the sliceanddice machine
2. At the systemd-boot menu, press **`Space`** or **`Esc`**
3. Select an **older NixOS generation** (from before the problem)
4. Press **Enter** to boot

βœ… **Done!** You're back in a working system.

Then prevent it from happening again:
```bash
cd /etc/nixos/.dotfiles
git checkout master
sudo nixos-rebuild switch --flake .#sliceanddice
```

---

## πŸ”§ Automated Fix Script

If you're in a working shell (emergency mode or booted from old generation):

```bash
cd /etc/nixos/.dotfiles
sudo bash fix-emergency-auto.sh
```

This script will:
- βœ… Detect the configuration problem
- βœ… Switch to stable master branch
- βœ… Rebuild your system
- βœ… Verify everything works

---

## πŸ“– Detailed Documentation

Choose the guide that fits your situation:

| File | Purpose | When to Use |
|------|---------|-------------|
| **QUICKSTART.md** | Fast recovery steps | You want the TL;DR |
| **SOLUTION.md** | Complete explanation | You want to understand what happened |
| **EMERGENCY_RECOVERY.md** | Technical deep-dive | You need detailed recovery procedures |
| **diagnose-emergency.sh** | Diagnostic script | You want to see what's wrong |
| **fix-emergency-auto.sh** | Automated fix | You want one-command recovery |

---

## πŸ” What Happened?

**Short version**: The `development` branch has new features that require additional files. If you rebuilt using development branch config but the files weren't present, NixOS couldn't activate and dropped into emergency mode.

**Files required by development branch**:
- `disko.nix` - Disk/filesystem management
- `kbd-bl-ask.nix` - Keyboard backlight config
- `msi-ec-sword-kbd-disable.patch` - MSI laptop kernel patch
- Power management features
- Boot generation limiting

**Current master branch**: βœ… Stable, working, doesn't need these files

---

## πŸ› οΈ Recovery Paths

### Path A: Previous Generation (Easiest)
Boot from systemd-boot menu β†’ Select older generation β†’ Boot

### Path B: Emergency Shell Fix
```bash
mount -o remount,rw /
cd /etc/nixos/.dotfiles
git checkout master
nixos-rebuild switch --flake .#sliceanddice
reboot
```

### Path C: Automated Script
```bash
sudo bash fix-emergency-auto.sh
```

### Path D: Rescue Media
Boot NixOS installer β†’ Mount filesystems β†’ Chroot β†’ Fix config

---

## ⚑ After Recovery

Once your system is working again:

1. **Verify you're on master branch**:
```bash
cd /etc/nixos/.dotfiles
git branch # Should show: * master
```

2. **Stay on master** (recommended):
- It's stable and tested
- No configuration changes needed
- Everything works

3. **OR move to development** (advanced):
- Merge development features properly
- Test in VM first
- Ensure all files are present

---

## πŸ“ž Still Having Issues?

1. Run the diagnostic script:
```bash
bash diagnose-emergency.sh
```

2. Check the detailed guides:
- Read `SOLUTION.md` for full explanation
- Read `EMERGENCY_RECOVERY.md` for technical details

3. Verify filesystem integrity:
```bash
lsblk -o NAME,UUID,FSTYPE,SIZE,MOUNTPOINT
df -h
mount | grep -E "/(boot|nix|home)"
```

---

## πŸ“Š System Information

**Machine**: MSI laptop with Intel Tiger Lake + NVIDIA RTX 3050 Ti
**Storage**: Samsung 870 EVO 500GB
**Current filesystem**: ext4 on root
**Boot**: systemd-boot (511M EFI partition)

**Expected UUIDs**:
- Root: `b89f5dca-4b37-4062-bf1d-9e4ebfd61916`
- Boot: `8824-4C5F`
- Swap: `c570ec29-6025-456b-99d1-8f16b677835a`

---

## βœ… Success Indicators

You'll know recovery worked when:
- βœ… System boots to desktop without emergency mode
- βœ… All applications work normally
- βœ… `systemctl status` shows no critical failures
- βœ… You're on master branch: `git branch` in dotfiles shows `* master`

---

## 🎯 Summary

| Problem | Fix | Time |
|---------|-----|------|
| Emergency mode | Boot previous generation | 1 min |
| + Want to prevent | Switch to master branch | 5 min |
| + Want automated fix | Run fix-emergency-auto.sh | 10 min |

**Bottom line**: Boot an older generation from the boot menu. Problem solved! πŸŽ‰
Loading
Loading