diff --git a/EMERGENCY_RECOVERY.md b/EMERGENCY_RECOVERY.md new file mode 100644 index 00000000..c4de106c --- /dev/null +++ b/EMERGENCY_RECOVERY.md @@ -0,0 +1,130 @@ +# Emergency Recovery Guide for sliceanddice + +## Problem Analysis + +Your sliceanddice NixOS system is in emergency mode. Based on the repository analysis: + +### Most Likely Causes: + +1. **Missing disko.nix import**: If you pulled/applied configuration from the `development` branch (commit `ba8fb7cb` or later), it references `./disko.nix` which doesn't exist on `master` branch. + +2. **Filesystem mounting failure**: The recent changes included: + - New disko partitioning scheme (btrfs subvolumes) + - Modified filesystem UUIDs and mount options + - New swap configuration + +3. **Boot generation limit**: Commit `963b83f1` capped systemd-boot generations to 5 to prevent `/boot` from filling (511M EFI partition). + +## Immediate Recovery Steps + +### Option 1: Boot into a previous generation (RECOMMENDED FIRST STEP) + +1. At boot, when you see the systemd-boot menu, press `Space` or `Esc` immediately +2. Select an older NixOS generation from before your recent changes +3. Press `Enter` to boot +4. Once booted, you can investigate what went wrong + +### Option 2: Emergency mode recovery + +If you're already in emergency mode: + +```bash +# 1. Check which filesystems are mounted +mount | grep -E "/(boot|nix|home)" + +# 2. Check for filesystem errors +journalctl -xb | grep -i "failed\|error\|emergency" + +# 3. Check if /boot is full +df -h /boot + +# 4. Check filesystem mounts status +systemctl list-units --type=mount --failed + +# 5. Try to mount root as read-write if needed +mount -o remount,rw / +``` + +### Option 3: Rollback to stable master branch configuration + +If you're using `/etc/nixos/.dotfiles`: + +```bash +# In emergency mode or from an older generation: +cd /etc/nixos/.dotfiles + +# Check current branch and status +git branch +git status + +# If you're on development branch, switch to master +git checkout master +git pull origin master + +# Rebuild with stable config +nixos-rebuild switch --flake /etc/nixos/.dotfiles#sliceanddice +``` + +## Known Issues from Recent Commits + +### Issue 1: disko.nix import on master branch +- **Commit**: `ba8fb7cb` (on development branch only) +- **Problem**: Imports `./disko.nix` which doesn't exist on master +- **Solution**: Ensure you're using master branch, or cherry-pick disko.nix from development + +### Issue 2: Boot partition space +- **Commit**: `963b83f1` +- **Problem**: 511M EFI partition fills with initrds (~50MB each) +- **Solution**: Already fixed with `boot.loader.systemd-boot.configurationLimit = 5` +- **Check**: Run `df -h /boot` to verify space + +### Issue 3: Filesystem UUID changes +- **Location**: `hosts/nixos/sliceanddice/hardware-configuration.nix` +- **Current UUIDs**: + - Root: `b89f5dca-4b37-4062-bf1d-9e4ebfd61916` (ext4) + - Boot: `8824-4C5F` (vfat) + - Swap: `c570ec29-6025-456b-99d1-8f16b677835a` + +## Verification Commands + +Once you can boot normally, verify the configuration: + +```bash +# Check current branch +cd /etc/nixos/.dotfiles && git branch + +# Verify filesystem mounts +findmnt / /boot /home /nix + +# Check boot partition space +df -h /boot +du -sh /boot/loader/entries/* + +# List available boot generations +ls -lh /boot/loader/entries/ + +# Check system status +systemctl status +journalctl -p err -b +``` + +## Prevention + +1. **Always test major changes**: Use `nixos-rebuild test` before `switch` +2. **Keep old generations**: The configurationLimit=5 is set, but you can temporarily increase it +3. **Monitor /boot space**: Run `df -h /boot` before rebuilds +4. **Stick to one branch**: Use either `master` or `development`, not a mix + +## Contact Information + +If you need to restore from scratch: +- Bootstrap script: `scripts/install-sliceanddice.sh` +- Secrets: `secrets/sliceanddice-secrets.yaml` (encrypted with sops) +- Hardware config: `hosts/nixos/sliceanddice/hardware-configuration.nix` + +## Architecture Notes + +- Laptop: MSI with Intel Tiger Lake UHD + NVIDIA RTX 3050 Ti Mobile +- Storage: Samsung 870 EVO 500GB (`ata-Samsung_SSD_870_EVO_500GB_S62ANJ0R238724D`) +- Graphics: Hybrid PRIME render offload (Intel for display, NVIDIA on-demand) +- Kernel: Latest mainline (`pkgs.linuxPackages_latest`) diff --git a/QUICKSTART.md b/QUICKSTART.md new file mode 100644 index 00000000..c0d13c8f --- /dev/null +++ b/QUICKSTART.md @@ -0,0 +1,94 @@ +# Quick Recovery Guide - sliceanddice Emergency Mode + +## Fastest Solution (Do This First!) + +### Step 1: Boot Previous Generation + +1. **Reboot** your sliceanddice machine +2. When you see the boot menu, press **Space** or **Esc** +3. **Select an older NixOS generation** (one from before the issue) +4. Press Enter to boot + +That's it! You should now be back in a working system. + +--- + +## Step 2: Prevent It From Happening Again + +Once you're booted into the working generation: + +```bash +# Go to your dotfiles +cd /etc/nixos/.dotfiles + +# Check which branch you're on +git branch +# If you see "development", that's likely the issue + +# Switch to stable master branch +git checkout master +git pull origin master + +# Make this the current system +sudo nixos-rebuild switch --flake /etc/nixos/.dotfiles#sliceanddice + +# Reboot to test +sudo reboot +``` + +--- + +## What Happened? + +The `development` branch has new features but requires additional files: +- `disko.nix` - disk configuration +- `kbd-bl-ask.nix` - keyboard backlight +- MSI-specific kernel patches +- Power management features + +If you rebuilt with the development branch but those files weren't present, NixOS couldn't complete the build and entered emergency mode. + +--- + +## Alternative: Fix in Emergency Mode + +If you can't access the boot menu: + +```bash +# You should be at an emergency shell prompt +# Remount root as writable +mount -o remount,rw / + +# Fix the configuration +cd /etc/nixos/.dotfiles +git checkout master +git reset --hard origin/master + +# Rebuild +nixos-rebuild switch --flake .#sliceanddice + +# Reboot +reboot +``` + +--- + +## Need More Help? + +See the detailed guides: +- **SOLUTION.md** - Full explanation and multiple recovery options +- **EMERGENCY_RECOVERY.md** - Detailed technical recovery procedures +- **diagnose-emergency.sh** - Run this script to diagnose the issue + +Or run the diagnostic script: +```bash +bash diagnose-emergency.sh +``` + +--- + +## Summary + +**Problem**: Configuration mismatch between branches +**Solution**: Boot previous generation, switch to master branch +**Prevention**: Stick to one branch (master is stable) diff --git a/README-EMERGENCY.md b/README-EMERGENCY.md new file mode 100644 index 00000000..dd804bd2 --- /dev/null +++ b/README-EMERGENCY.md @@ -0,0 +1,173 @@ +# πŸ†˜ sliceanddice Emergency Mode - Recovery Instructions + +> **STATUS**: Your sliceanddice NixOS system is in emergency mode +> **CAUSE**: Configuration mismatch between master and development branches +> **SOLUTION**: Multiple recovery paths available (choose the easiest one for you) + +--- + +## πŸš€ Quick Fix (Recommended - 2 minutes) + +### Option 1: Boot Previous Generation (FASTEST!) + +1. **Reboot** the sliceanddice machine +2. At the systemd-boot menu, press **`Space`** or **`Esc`** +3. Select an **older NixOS generation** (from before the problem) +4. Press **Enter** to boot + +βœ… **Done!** You're back in a working system. + +Then prevent it from happening again: +```bash +cd /etc/nixos/.dotfiles +git checkout master +sudo nixos-rebuild switch --flake .#sliceanddice +``` + +--- + +## πŸ”§ Automated Fix Script + +If you're in a working shell (emergency mode or booted from old generation): + +```bash +cd /etc/nixos/.dotfiles +sudo bash fix-emergency-auto.sh +``` + +This script will: +- βœ… Detect the configuration problem +- βœ… Switch to stable master branch +- βœ… Rebuild your system +- βœ… Verify everything works + +--- + +## πŸ“– Detailed Documentation + +Choose the guide that fits your situation: + +| File | Purpose | When to Use | +|------|---------|-------------| +| **QUICKSTART.md** | Fast recovery steps | You want the TL;DR | +| **SOLUTION.md** | Complete explanation | You want to understand what happened | +| **EMERGENCY_RECOVERY.md** | Technical deep-dive | You need detailed recovery procedures | +| **diagnose-emergency.sh** | Diagnostic script | You want to see what's wrong | +| **fix-emergency-auto.sh** | Automated fix | You want one-command recovery | + +--- + +## πŸ” What Happened? + +**Short version**: The `development` branch has new features that require additional files. If you rebuilt using development branch config but the files weren't present, NixOS couldn't activate and dropped into emergency mode. + +**Files required by development branch**: +- `disko.nix` - Disk/filesystem management +- `kbd-bl-ask.nix` - Keyboard backlight config +- `msi-ec-sword-kbd-disable.patch` - MSI laptop kernel patch +- Power management features +- Boot generation limiting + +**Current master branch**: βœ… Stable, working, doesn't need these files + +--- + +## πŸ› οΈ Recovery Paths + +### Path A: Previous Generation (Easiest) +Boot from systemd-boot menu β†’ Select older generation β†’ Boot + +### Path B: Emergency Shell Fix +```bash +mount -o remount,rw / +cd /etc/nixos/.dotfiles +git checkout master +nixos-rebuild switch --flake .#sliceanddice +reboot +``` + +### Path C: Automated Script +```bash +sudo bash fix-emergency-auto.sh +``` + +### Path D: Rescue Media +Boot NixOS installer β†’ Mount filesystems β†’ Chroot β†’ Fix config + +--- + +## ⚑ After Recovery + +Once your system is working again: + +1. **Verify you're on master branch**: + ```bash + cd /etc/nixos/.dotfiles + git branch # Should show: * master + ``` + +2. **Stay on master** (recommended): + - It's stable and tested + - No configuration changes needed + - Everything works + +3. **OR move to development** (advanced): + - Merge development features properly + - Test in VM first + - Ensure all files are present + +--- + +## πŸ“ž Still Having Issues? + +1. Run the diagnostic script: + ```bash + bash diagnose-emergency.sh + ``` + +2. Check the detailed guides: + - Read `SOLUTION.md` for full explanation + - Read `EMERGENCY_RECOVERY.md` for technical details + +3. Verify filesystem integrity: + ```bash + lsblk -o NAME,UUID,FSTYPE,SIZE,MOUNTPOINT + df -h + mount | grep -E "/(boot|nix|home)" + ``` + +--- + +## πŸ“Š System Information + +**Machine**: MSI laptop with Intel Tiger Lake + NVIDIA RTX 3050 Ti +**Storage**: Samsung 870 EVO 500GB +**Current filesystem**: ext4 on root +**Boot**: systemd-boot (511M EFI partition) + +**Expected UUIDs**: +- Root: `b89f5dca-4b37-4062-bf1d-9e4ebfd61916` +- Boot: `8824-4C5F` +- Swap: `c570ec29-6025-456b-99d1-8f16b677835a` + +--- + +## βœ… Success Indicators + +You'll know recovery worked when: +- βœ… System boots to desktop without emergency mode +- βœ… All applications work normally +- βœ… `systemctl status` shows no critical failures +- βœ… You're on master branch: `git branch` in dotfiles shows `* master` + +--- + +## 🎯 Summary + +| Problem | Fix | Time | +|---------|-----|------| +| Emergency mode | Boot previous generation | 1 min | +| + Want to prevent | Switch to master branch | 5 min | +| + Want automated fix | Run fix-emergency-auto.sh | 10 min | + +**Bottom line**: Boot an older generation from the boot menu. Problem solved! πŸŽ‰ diff --git a/README.md b/README.md index 5cee881a..8a1a9a0c 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ Personal, declarative system configuration built by **Alex Spaulding (aspauldingcode)** using Nix Flakes. +> **πŸ†˜ EMERGENCY RECOVERY**: If your sliceanddice system is in emergency mode, see [`README-EMERGENCY.md`](README-EMERGENCY.md) for immediate recovery instructions. + ## Fleet diff --git a/SOLUTION.md b/SOLUTION.md new file mode 100644 index 00000000..88826327 --- /dev/null +++ b/SOLUTION.md @@ -0,0 +1,154 @@ +# Solution for sliceanddice Emergency Mode + +## Problem Summary + +Your sliceanddice NixOS system entered emergency mode. Analysis of the git repository reveals: + +### Root Cause + +The `development` branch (ahead of `master`) includes these changes to sliceanddice: +- **NEW**: `disko.nix` - disk/filesystem configuration +- **NEW**: `kbd-bl-ask.nix` - keyboard backlight config +- **NEW**: `msi-ec-sword-kbd-disable.patch` - MSI EC kernel patch +- **MODIFIED**: `default.nix` - imports disko.nix, adds power management, nixinstall features +- **MODIFIED**: `hardware-configuration.nix` - updated for new partition layout + +**If you pulled/applied configuration from `development` to your machine but the files are missing, the system can't boot.** + +## Immediate Recovery (Choose One) + +### Recovery Option A: Boot Previous Generation (FASTEST) + +1. **Reboot** the machine +2. At the **systemd-boot menu**, press `Space` or `Esc` +3. **Select an older generation** from before your changes +4. Boot into that generation +5. Then either: + - Stay on that generation (it works), OR + - Fix the configuration and rebuild + +### Recovery Option B: Fix Configuration in Emergency Mode + +If you're already in emergency mode shell: + +```bash +# 1. Remount root as read-write +mount -o remount,rw / + +# 2. Navigate to dotfiles +cd /etc/nixos/.dotfiles + +# 3. Check current state +git status +git branch + +# 4. Switch to stable master branch +git fetch origin +git checkout master +git reset --hard origin/master + +# 5. Rebuild from stable config +nixos-rebuild switch --flake /etc/nixos/.dotfiles#sliceanddice + +# 6. Reboot +reboot +``` + +### Recovery Option C: Use Rescue Media + +If the above don't work: + +1. Boot from NixOS install media or the nixinstall partition +2. Mount your root filesystem: + ```bash + mount /dev/disk/by-uuid/b89f5dca-4b37-4062-bf1d-9e4ebfd61916 /mnt + mount /dev/disk/by-uuid/8824-4C5F /mnt/boot + ``` +3. Chroot and fix: + ```bash + nixos-enter --root /mnt + cd /etc/nixos/.dotfiles + git checkout master + nixos-rebuild switch --flake .#sliceanddice + exit + reboot + ``` + +## Long-term Solution: Sync Repository + +To prevent this in the future, you need to decide: + +### Option 1: Stay on Master (Stable) + +Keep using the `master` branch - it's tested and working. + +```bash +cd /etc/nixos/.dotfiles +git checkout master +git pull origin master +nixos-rebuild switch --flake .#sliceanddice +``` + +### Option 2: Move to Development (New Features) + +If you want the new features (power management, disko, nixinstall), merge development to master: + +**This should be done in this repository first, then pulled to sliceanddice.** + +The development branch adds: +- ✨ Disk management with disko (btrfs subvolumes) +- ✨ Power management (RAPL/EPP, zswap, lid/suspend) +- ✨ On-disk NixOS installer +- ✨ Windows dual-boot support +- ✨ Boot generation limiting (prevents /boot from filling) +- ✨ Keyboard backlight controls + +## Technical Details + +### Current Master Branch +- Root: ext4 (`/dev/disk/by-uuid/b89f5dca-4b37-4062-bf1d-9e4ebfd61916`) +- No disko.nix +- No boot generation limit +- Basic hardware config + +### Development Branch Changes +- Root: Can be btrfs with subvolumes (after reinstall via nixinstall) +- Managed by disko.nix +- Boot generations limited to 5 +- Power management enabled +- Enhanced hardware support (MSI laptop features) + +### Why Emergency Mode Happened + +Most likely scenario: +1. You ran `nixos-rebuild switch` with development branch config +2. The config imports `./disko.nix` +3. File doesn't exist (not pulled or wrong branch) +4. NixOS build/activation fails +5. System can't complete boot β†’ emergency mode + +Alternative scenarios: +- /boot partition full (unlikely with the 5-generation limit) +- Filesystem mount failure +- Hardware configuration mismatch + +## Files Provided for Recovery + +1. **EMERGENCY_RECOVERY.md** - Detailed recovery steps +2. **diagnose-emergency.sh** - Diagnostic script to run in emergency mode +3. **SOLUTION.md** - This file + +## Next Steps + +1. **First**: Boot into a previous generation (Recovery Option A) +2. **Then**: Decide if you want to stay on master or move to development +3. **If development**: Properly merge the branches in this repository +4. **Finally**: Pull the correct branch to sliceanddice and rebuild + +## Questions? + +Check these files for your current configuration: +- Hardware UUIDs: `hosts/nixos/sliceanddice/hardware-configuration.nix` +- Main config: `hosts/nixos/sliceanddice/default.nix` +- Boot config: In default.nix, look for `boot.loader.*` +- Filesystems: In hardware-configuration.nix or disko.nix (if present) diff --git a/START_HERE.md b/START_HERE.md new file mode 100644 index 00000000..5b46364b --- /dev/null +++ b/START_HERE.md @@ -0,0 +1,201 @@ +# 🚨 START HERE - sliceanddice Emergency Recovery + +**YOUR SYSTEM**: sliceanddice (NixOS laptop) +**STATUS**: Emergency mode +**URGENCY**: Medium (system is recoverable, data is safe) +**SOLUTION TIME**: 1-10 minutes depending on method + +--- + +## What You Need to Do RIGHT NOW + +### The 1-Minute Fix (DO THIS FIRST!) + +1. **Reboot** your sliceanddice laptop +2. You'll see a boot menu with several options +3. Press **`Space`** or **`Esc`** key +4. Use arrow keys to select an **older entry** (one from before the issue) +5. Press **`Enter`** to boot + +**That's it!** Your system should boot normally now. + +--- + +## Understanding What Happened + +### The Problem +Your NixOS configuration repository has two branches: +- **`master`** - Stable, working configuration βœ… +- **`development`** - New features, requires extra files ⚠️ + +You recently made changes that may have mixed these branches, causing the system to look for files that don't exist, which triggered emergency mode. + +### The Branches + +**Master Branch (current):** +- Simple, stable configuration +- ext4 filesystem +- Works out of the box + +**Development Branch (ahead):** +- Adds `disko.nix` for disk management +- Adds `kbd-bl-ask.nix` for keyboard backlight +- Adds power management features +- Adds on-disk installer +- **Requires** all these files to exist + +--- + +## After You've Booted Successfully + +Once you're back in a working system, **prevent it from happening again**: + +```bash +# Open a terminal +cd /etc/nixos/.dotfiles + +# Check which branch you're on +git branch +# If it shows: * development (or anything other than master) + +# Switch to stable master +git checkout master +git pull origin master + +# Make this the active configuration +sudo nixos-rebuild switch --flake /etc/nixos/.dotfiles#sliceanddice + +# Verify it worked +sudo reboot +``` + +--- + +## Alternative: Use the Automatic Fix Script + +If you're comfortable with automation: + +```bash +cd /etc/nixos/.dotfiles +sudo bash fix-emergency-auto.sh +``` + +This script will: +1. Detect the problem +2. Switch to master branch +3. Rebuild your system +4. Tell you if it worked + +--- + +## All Available Resources + +I've created comprehensive documentation for you: + +| File | Purpose | Read When | +|------|---------|-----------| +| **START_HERE.md** | Quick start (this file) | Right now! | +| **README-EMERGENCY.md** | Main recovery hub | After quick fix | +| **QUICKSTART.md** | Fast recovery steps | You want TL;DR | +| **SOLUTION.md** | Full explanation | You want details | +| **EMERGENCY_RECOVERY.md** | Technical deep-dive | You need advanced help | +| **diagnose-emergency.sh** | Diagnostic tool | Something's still wrong | +| **fix-emergency-auto.sh** | Automated fix | You want one command | + +--- + +## Your Recovery Path + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ You are here: β”‚ +β”‚ System in emergency modeβ”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”œβ”€β”€ Option 1: Boot old generation (1 min) ──┐ + β”‚ β”‚ + β”œβ”€β”€ Option 2: Auto-fix script (5 min) ─────── + β”‚ β”‚ + └── Option 3: Manual fix (10 min) ─────────── + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ System working again β”‚ + β”‚ On stable master branchβ”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## Technical Summary (For Reference) + +### Your System Specs +- **Machine**: MSI laptop (Intel Tiger Lake + NVIDIA RTX 3050 Ti) +- **Storage**: Samsung 870 EVO 500GB +- **Current FS**: ext4 +- **Boot**: systemd-boot (511M EFI) + +### Root Cause Analysis +- Development branch refs `./disko.nix` import +- File exists in development but not in master +- If system tried to build with development config on master branch +- β†’ Missing file β†’ Build/activation failure β†’ Emergency mode + +### Expected Filesystem UUIDs +- Root: `b89f5dca-4b37-4062-bf1d-9e4ebfd61916` (ext4) +- Boot: `8824-4C5F` (vfat) +- Swap: `c570ec29-6025-456b-99d1-8f16b677835a` + +--- + +## FAQ + +**Q: Will I lose data?** +A: No! Your data is safe. This is a configuration issue, not a data loss issue. + +**Q: Can I just stay on the old generation?** +A: Yes! It's perfectly fine to boot the old generation and use that. You don't have to "fix" anything if the old generation works for you. + +**Q: Do I have to switch to master branch?** +A: For now, yes, unless you want to properly merge the development features (which requires more work). + +**Q: What if the old generation doesn't work?** +A: See `EMERGENCY_RECOVERY.md` for rescue media instructions. But this is very unlikely. + +**Q: Can I keep using development branch?** +A: Yes, but you need to ensure ALL the required files are present and properly tracked in git. + +--- + +## Next Steps + +1. βœ… **Now**: Boot previous generation (the 1-minute fix above) +2. βœ… **Next**: Switch to master branch (commands above) +3. βœ… **Later**: Read `README-EMERGENCY.md` to understand everything +4. βœ… **Future**: Decide if you want to stay on master or properly setup development + +--- + +## Pull Request + +All this documentation has been added to your repository in this PR: +- **Branch**: `cursor/fix-sliceanddice-emergency-mode-75c8` +- **PR**: #188 on GitHub + +You can merge it once your system is recovered to have these docs available for future reference. + +--- + +## Support + +If you're still stuck after trying the above: + +1. Run diagnostic: `bash diagnose-emergency.sh` +2. Read detailed guides: `EMERGENCY_RECOVERY.md`, `SOLUTION.md` +3. Check system logs: `journalctl -xb | grep -i error` +4. Verify mounts: `mount | grep -E "/(boot|nix)"` + +**Remember**: Your data is safe. This is fixable. The fastest path is: Reboot β†’ Boot old generation β†’ Done! + +--- + +Good luck! πŸ€ diff --git a/diagnose-emergency.sh b/diagnose-emergency.sh new file mode 100755 index 00000000..bbf04828 --- /dev/null +++ b/diagnose-emergency.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# Emergency diagnostic script for sliceanddice boot issues +# Run this in emergency mode or from a rescue shell + +set -euo pipefail + +echo "=== sliceanddice Emergency Diagnostic ===" +echo "Timestamp: $(date)" +echo "" + +echo "=== System Status ===" +systemctl status --no-pager --failed || true +echo "" + +echo "=== Mount Status ===" +mount | sort +echo "" + +echo "=== Filesystem Space ===" +df -h || true +echo "" + +echo "=== Failed Mount Units ===" +systemctl list-units --type=mount --failed --no-pager || true +echo "" + +echo "=== Boot Partition Contents ===" +if [ -d /boot ]; then + echo "Boot partition space:" + du -sh /boot/* 2>/dev/null || true + echo "" + echo "Boot entries:" + ls -lh /boot/loader/entries/ 2>/dev/null || true +else + echo "/boot not mounted" +fi +echo "" + +echo "=== Filesystem UUIDs ===" +echo "Expected UUIDs from hardware-configuration.nix:" +echo " Root (ext4): b89f5dca-4b37-4062-bf1d-9e4ebfd61916" +echo " Boot (vfat): 8824-4C5F" +echo " Swap: c570ec29-6025-456b-99d1-8f16b677835a" +echo "" +echo "Actual block devices:" +lsblk -o NAME,UUID,FSTYPE,SIZE,MOUNTPOINT || true +echo "" + +echo "=== Git Configuration Status ===" +if [ -d /etc/nixos/.dotfiles ]; then + cd /etc/nixos/.dotfiles + echo "Branch: $(git branch --show-current 2>/dev/null || echo 'unknown')" + echo "Last commit: $(git log -1 --oneline 2>/dev/null || echo 'unknown')" + echo "Status:" + git status -s 2>/dev/null || true + echo "" + echo "Checking for disko.nix:" + if [ -f hosts/nixos/sliceanddice/disko.nix ]; then + echo " βœ“ disko.nix exists" + else + echo " βœ— disko.nix MISSING" + echo " β†’ This is likely the problem if default.nix imports it!" + fi + echo "" + echo "Checking imports in default.nix:" + grep -E "^\s*(./disko\.nix|imports\s*=)" hosts/nixos/sliceanddice/default.nix 2>/dev/null || true +else + echo "/etc/nixos/.dotfiles not found" +fi +echo "" + +echo "=== Recent Boot Logs (errors only) ===" +journalctl -p err -b --no-pager -n 50 2>/dev/null || true +echo "" + +echo "=== Kernel Command Line ===" +cat /proc/cmdline 2>/dev/null || true +echo "" + +echo "=== Available Generations ===" +if [ -d /nix/var/nix/profiles ]; then + ls -lh /nix/var/nix/profiles/system-*-link 2>/dev/null | tail -10 || true +else + echo "Profile directory not accessible" +fi +echo "" + +echo "=== Suggested Actions ===" +echo "" +echo "1. If disko.nix is missing but imported:" +echo " cd /etc/nixos/.dotfiles && git checkout master" +echo "" +echo "2. If /boot is full:" +echo " nix-collect-garbage -d" +echo " Or temporarily increase configurationLimit" +echo "" +echo "3. If filesystems won't mount:" +echo " Check 'journalctl -xb' for specific mount errors" +echo " Verify UUIDs match: blkid /dev/sdX" +echo "" +echo "4. Rollback to previous generation:" +echo " From systemd-boot menu, select an older entry" +echo "" +echo "5. Force rebuild from master:" +echo " cd /etc/nixos/.dotfiles" +echo " git checkout master && git reset --hard origin/master" +echo " nixos-rebuild switch --flake /etc/nixos/.dotfiles#sliceanddice" +echo "" diff --git a/fix-emergency-auto.sh b/fix-emergency-auto.sh new file mode 100755 index 00000000..5fd9132e --- /dev/null +++ b/fix-emergency-auto.sh @@ -0,0 +1,167 @@ +#!/usr/bin/env bash +# Automatic emergency recovery for sliceanddice +# Run this script to automatically fix the emergency mode issue +# +# Usage: +# bash fix-emergency-auto.sh [--force] + +set -euo pipefail + +FORCE=false +if [[ "${1:-}" == "--force" ]]; then + FORCE=true +fi + +DOTFILES_DIR="/etc/nixos/.dotfiles" +HOST="sliceanddice" + +echo "╔════════════════════════════════════════════════════════════════╗" +echo "β•‘ sliceanddice Emergency Recovery - Automatic Fixer β•‘" +echo "β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•" +echo "" + +# Check if we're running as root +if [[ $EUID -ne 0 ]]; then + echo "❌ This script must be run as root (use sudo)" + exit 1 +fi + +# Check if dotfiles directory exists +if [[ ! -d "$DOTFILES_DIR" ]]; then + echo "❌ Dotfiles directory not found: $DOTFILES_DIR" + echo " Expected location: /etc/nixos/.dotfiles" + exit 1 +fi + +cd "$DOTFILES_DIR" + +echo "πŸ“ Current location: $(pwd)" +echo "" + +# Check git status +echo "πŸ” Checking git status..." +CURRENT_BRANCH=$(git branch --show-current 2>/dev/null || echo "unknown") +echo " Current branch: $CURRENT_BRANCH" + +# Check if disko.nix exists +DISKO_EXISTS=false +if [[ -f "hosts/nixos/sliceanddice/disko.nix" ]]; then + DISKO_EXISTS=true + echo " βœ“ disko.nix exists" +else + echo " βœ— disko.nix missing" +fi + +# Check if default.nix imports disko +IMPORTS_DISKO=false +if grep -q "./disko.nix" hosts/nixos/sliceanddice/default.nix 2>/dev/null; then + IMPORTS_DISKO=true + echo " ⚠ default.nix imports disko.nix" +else + echo " βœ“ default.nix does NOT import disko.nix" +fi + +echo "" + +# Determine if we need to fix +NEED_FIX=false +FIX_REASON="" + +if [[ "$IMPORTS_DISKO" == true && "$DISKO_EXISTS" == false ]]; then + NEED_FIX=true + FIX_REASON="default.nix imports disko.nix but file is missing" +elif [[ "$CURRENT_BRANCH" == "development" ]]; then + NEED_FIX=true + FIX_REASON="on development branch which may have incomplete features" +fi + +if [[ "$NEED_FIX" == false ]]; then + echo "βœ… Configuration appears correct!" + echo " No automatic fix needed." + echo "" + echo " If you're still experiencing issues, see:" + echo " - EMERGENCY_RECOVERY.md for manual steps" + echo " - diagnose-emergency.sh for diagnostics" + exit 0 +fi + +echo "⚠️ Issue detected: $FIX_REASON" +echo "" + +if [[ "$FORCE" == false ]]; then + echo "Proposed fix:" + echo " 1. Fetch latest from origin" + echo " 2. Switch to stable master branch" + echo " 3. Reset to origin/master" + echo " 4. Rebuild NixOS configuration" + echo "" + read -p "Apply this fix? (y/N) " -n 1 -r + echo "" + if [[ ! $REPLY =~ ^[Yy]$ ]]; then + echo "❌ Aborted by user" + exit 1 + fi +fi + +echo "" +echo "πŸ”§ Applying fix..." +echo "" + +# Step 1: Fetch +echo "β†’ Fetching from origin..." +if ! git fetch origin; then + echo "⚠️ Git fetch failed. Continuing anyway..." +fi + +# Step 2: Checkout master +echo "β†’ Switching to master branch..." +if ! git checkout master; then + echo "❌ Failed to checkout master branch" + exit 1 +fi + +# Step 3: Reset to origin/master +echo "β†’ Resetting to origin/master..." +if ! git reset --hard origin/master; then + echo "⚠️ Git reset failed. Continuing anyway..." +fi + +# Step 4: Show current state +echo "" +echo "πŸ“Š Current configuration:" +echo " Branch: $(git branch --show-current)" +echo " Commit: $(git log -1 --oneline)" +echo "" + +# Step 5: Rebuild +echo "β†’ Rebuilding NixOS configuration..." +echo " This may take several minutes..." +echo "" + +if nixos-rebuild switch --flake "$DOTFILES_DIR#$HOST"; then + echo "" + echo "╔════════════════════════════════════════════════════════════════╗" + echo "β•‘ βœ… FIX APPLIED SUCCESSFULLY! β•‘" + echo "β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•" + echo "" + echo "Next steps:" + echo " 1. Verify the system is working correctly" + echo " 2. Reboot to ensure it boots cleanly: sudo reboot" + echo "" + echo "Your system should now be running on the stable master branch." +else + echo "" + echo "❌ nixos-rebuild failed!" + echo "" + echo "This could mean:" + echo " - Network issues preventing package downloads" + echo " - Evaluation errors in the configuration" + echo " - Hardware issues" + echo "" + echo "Next steps:" + echo " 1. Check the error messages above" + echo " 2. Try rebooting to a previous generation" + echo " 3. See EMERGENCY_RECOVERY.md for manual recovery" + echo "" + exit 1 +fi