{{Header}} __FORCETOC__ {{#seo: |description=Recovery Mode, Emergency, Chroot, Debugging, Capturing Diagnostic Output from Serial Console |image=Recovery312123.jpg }} [[image:Recovery312123.jpg|thumb]] {{intro| The goal of this wiki page is to provide the user with knowledge for higher system reliability, robustness, increases uptime, backup, restoration, recovery and repair skills. }} = Introduction = It is prudent to acquire system recovery skills prior to an eventual later grave issue such as an unbootable VM or even unbootable host operating system, data loss or similar. This is because naturally for most users once they are affected by a grave issue, the mind goes into "issue solving mode", which can be stressful and less productive, specifically when it is currently a bad time to acquire these skills due to other more pressing priorities. By learning these skills in advance, one can the skills during a more appropriate time in in a much more positive, relayed environment which might potentially greatly increase a user's ability and lower the time required to fix their broken system, should this happen in the future. = Useful Skills = * Boot an {{os}} from an external USB / DVD boot drive. * Boot an {{os}} from an external USB / DVD boot drive and mount the OS which is installed on the internal hard drive to access or backup its data. For Qubes, see: https://www.qubes-os.org/doc/mount-from-other-os/ Yes, this is also possible when using [[Full Disk Encryption]]. * Remove an internal drive from one computer, move it into a USB enclosure and access its data from a different computer. * [[Grub#Kernel_Boot_Parameter_Change|Temporary Kernel Boot Parameter Change]] using [[grub]] bootloader boot menu. * [[Grub#Permanent_Configuration_Changes|Permanent Kernel Boot Parameter Configuration Changes]]. * [[Grub#Inspect_Grub_Configuration_Changes|Inspect Grub Configuration Changes]]. * [[Raw Disk Backup|Raw Disk Backup and Restoration]]. * [[Recovery#Recovery_Mode|How to boot into recovery mode]]. * How to switch to a [[Desktop#Virtual_Consoles|virtual console]] in a {{VM}}. * How to switch to a virtual console on the host operating system. * [[Troubleshooting#Enable_Persistent_Systemd_Journal_Log|Enable Persistent Systemd Journal Log]] * [[Troubleshooting#Check_Systemd_Journal_Log_of_Previous_Boot|Check Systemd Journal Log of Previous Boot]] * [[Troubleshooting#Check_Systemd_Journal_Log_of_Failed_Boot|Check Systemd Journal Log of Failed Boot]] * hard drive health check (S.M.A.R.T) * Topics mentioned on the [[Troubleshooting]] wiki page. * Other topics mentioned on this wiki page. = Useful Habits = * Keep the log of any [[Operating System Software and Updates|system upgrades]] as well as [[Install Software|software installation]] processes. * Keep one or multiple root virtual terminal tabs in a terminal emulator open during actions which might break to unbootable systems, data loss or other issues during software installation or upgrades. The same goes for one or multiple root [[Desktop#Virtual_Consoles|virtual consoles]]. This advice (for higher ability to easily recover from issuing gaining root) conflicts with the [[Root#General_Security_Advice|general security advice]] from the [[root]] wiki page (for better security). = Recovery Mode = If an error occurs that prevents {{project_name_long}} from booting, it is possible to try and boot {{project_name_long}} in recovery mode to fix the problem. It is advisable to learn how to use recovery mode while everything is still functional in case of future issues. {{Box|text= # After powering on a {{project_name_short}}, you will see the virtual BIOS for a second, then the grub boot menu. The grub boot menu is easily identified by the first line of text which begins with GNU GRUB. # Use the arrow keys to navigate and select Advanced Options for {{project_name_short}} GNU/Linux. # Press Enter. # Choose the second option which at the end displays (recovery mode). # Press Enter. # Somewhere in the output, a message similar to the following will appear (it might be entangled with other debug output):
Give root password for maintenance (Or press control + d to continue):
# Enter the root password. # While typing, no asterisk symbols (*) will appear; the password must be typed "blind". See also: [[Default_Passwords|Default Passwords]]. # Press Enter. # The default keyboard layout will be en-US. To change this setting, see: [[Keyboard Layout]]. # Enter exit to continue booting or poweroff or reboot. }} TODO: Explore whether networking is possible and how to transfer files out of the VM. = Virtual Consoles = An easier and more lightweight solution as alternative to [[#Recovery Mode|recovery mode]] might be [[Desktop#Virtual_Consoles|virtual consoles]]. If the graphical user interface is no longer starting, login to a virtual console might still be possible. '''1.''' Prerequisite knowledge: [[Desktop#Virtual Consoles|Virtual Consoles]]. '''2.''' Try to login in a virtual console in a different, still functional VM (virtual machine) as an exercise. '''3.''' If that works, try login to virtual console in the broken VM. = Hard Drive Recovery = If no data needs be recovered, a complete re-installation might be easier. '''A)''' If the host got unbootable due to hard drive issues: # Damage control. Stop using the hard drive. This is because further use might add further damage. # Make a full 1:1 raw clone of the hard drive. # Data recovery. [[Unspecific]] to {{project_name_short}}. '''B)''' If a VM got unbootable due to hard drive issues: Stop using the VM. This is because further use might add further damage. # Create a full raw copy of the hard drive. # Make a full 1:1 raw clone of the VM. # Data recovery. [[Unspecific]] to {{project_name_short}}. = Unlock User Account: Excessive Wrong Password Entry Attempts = See: [[Root#Unlock_User_Account:_Excessive_Wrong_Password_Entry_Attempts|Unlock User Account: Excessive Wrong Password Entry Attempts]]. = Reset User Account Password = See [[Root#Reset_User_Account_Password|Reset User Account Password]]. = Boot Virtual Machine from ISO instead of Virtual Hard Drive = == VirtualBox == Booting a Debian-based VirtualBox VM (such as Kicksecure or Whonix, which are derivatives based on Debian) using an ISO file instead of the internal disk involves changing the boot order and attaching the ISO file as a virtual CD/DVD drive. This is useful for purposes of recovery, [[Grow Virtual Harddisk]], [[Shrink Virtual Harddisk]], or installing a new operating system. Which ISO to use? The [[ISO|Kicksecure ISO]] should be suitable for these purposes. Here are the steps to do this: [[Untested]]! '''1.''' Open VirtualBox. Launch VirtualBox from your applications menu. '''2.''' Select the VM. In the VirtualBox Manager, click on the VM that you want to boot from an ISO. '''3.''' Open Settings. Click on the "Settings" button (a gear icon) at the top. '''4.''' Configure Storage. * In the Settings window, go to the "Storage" tab. * You will see a tree structure with a controller (like IDE or SATA) and attached storage devices. * Click on the empty CD/DVD drive (usually labeled as "Empty" under the appropriate controller). '''5.''' Add ISO File. * On the right side of the window, you will see an icon of a CD/DVD. Click on this icon. * Choose "Choose a disk file..." and navigate to the location of your ISO file on your host machine. Select the ISO file and click "Open." * The ISO file should now be attached to the VM as a virtual CD/DVD drive. '''6.''' Configure Boot Order. * Go to the "System" tab in the Settings window. * Click on the "Motherboard" tab within the System settings. * In the "Boot Order" section, ensure that "Optical" is checked and listed above "Hard Disk." You can reorder them by dragging or using the up/down arrows. '''7.''' Save and Start VM. * Click "OK" to save the settings. * Start the VM by clicking the "Start" button. '''8.''' Verify ISO Boot. * As the VM starts, watch for any prompts to "Press any key to boot from CD or DVD." * You should see the boot menu or installation screen of the ISO you've selected. * If the VM boots into the existing operating system instead, shut down the VM and double-check steps 5 and 6. '''9.''' Complete Your Task. * Proceed with your intended task (installation, recovery, etc.) using the booted ISO. * Once you've finished and no longer need to boot from the ISO, shut down the VM. '''10.''' Detach ISO After Use. * After the VM has been powered off, go back to the "Storage" tab in the VM settings. * Click on the ISO file that you previously attached. * Click on the CD/DVD icon on the right and choose "Remove disk from virtual drive." '''11.''' Revert Boot Order (if needed). * Go to the "System" tab in the Settings window. * Click on the "Motherboard" tab within the System settings. * In the "Boot Order" section, ensure that "Hard Disk" is checked and listed above "Optical." '''12.''' Done. Your VM should now boot from the internal disk instead of the ISO file. '''Troubleshooting:''' * If the VM fails to boot from the ISO, ensure the ISO file is not corrupted and is compatible with your VM configuration. = Chroot = If an error occurs that prevents {{project_name_short}} from booting, it is possible to [https://www.computerhope.com/unix/chroot.htm chroot] into {{project_name_short}} from a live CD to fix the problem. A disk image must be downloaded for this purpose -- [https://www.debian.org/CD/live/ Debian Live images] or [https://www.archlinux.org/download/ Arch ISO] are useful options for this procedure. === VirtualBox === In VirtualBox, navigate to the VM storage settings. At the second controller, click the optical disk icon and add the disk image. In the Motherboard settings, move the optical disk component to the top of the boot order, press OK and start the VM. === KVM === In virt-manager, click on the VM, click the settings icon at the top and click "Add Hardware". In "Storage", change the device type to "CDROM device" and add the disk image. Click "Finish" and at the boot options setting, move the CDROM to the top of the boot order. === {{project_name_short}} === This procedure is [[undocumented]] and unspecific to {{project_name_short}}. See [[unspecific]]. == Booting into the Live CD == After booting into the Live CD, follow these steps. {{Box|text= '''1.''' List the available drives and partitions. {{CodeSelect|code= fdisk -l }} '''2.''' Mount the partition. {{CodeSelect|code= mount /dev/sda1 /mnt }} Replace "/dev/sda1" with the drive partition name. '''3.''' Chroot into the partition. {{CodeSelect|code= chroot /mnt /bin/bash }} '''4.''' ''Optional:'' If the PATH variable is incorrect, it can be fixed with the following command. {{CodeSelect|code= source /etc/profile }} }} = Serial Console = == VirtualBox == === Preparation === ==== Host Preparation ==== Install socat on the host operating system. The following steps apply to Linux distributions. {{Install Package|package= socat }} It is advised to research these using a non-{{project_name_short}} VM such as Debian {{Stable project version based on Debian codename}} and [[unspecific|learn it in a generic way]] which could then also be applied to {{project_name_short}}. ==== VM Preparation ==== Inside the {{project_name_short}} virtual machine where you want to enable the serial console, install the serial-console-enable package. {{Install Package|package= serial-console-enable }} (Installed by default would be nice but it is not installed by default due to issue. TODO: add reference) === Read Only Serial Console === This option is very useful for capturing diagnostic output from a virtual machine. It allows a log of everything written to the kernel console to be obtained -- all console output is written to a plain text file from boot until shutdown. The logfile persists after VM shutdown, but is overwritten after the VM is shutdown and powered on again. {{Box|text= '''1.''' Enable the serial ports option. 3 {{project_name_short}} VM settings → Serial Ports → Tick enable Configure the following settings: * Port Number: COM1 (default) * Port Mode: Raw File * Path/Address: /home/user/vbox-raw-file '''2.''' View the file with a text editor. Any text editor can be used for this purpose. For example to view the file with mousepad, run. {{CodeSelect|code= mousepad /home/user/vbox-raw-file }} '''3.''' ''Optional:'' View the file as it is being appended to from a terminal emulator. The logfile is also compatible with other standard linux utlitites such as tail. To view it as it is being written, run. {{CodeSelect|code= tail -f /home/user/vbox-raw-file }} '''4.''' Retain the file contents. After VM shutdown it may be useful to retain its contents by copying the vbox-raw-file elsewhere to make it a persistent log file. If this is desirable, run. {{CodeSelect|code= cp /home/user/vbox-raw-file /home/user/vbox-console-log }} }} === Interactive Serial Console === This procedure does not yet work during the grub boot menu. {{Box|text= '''1.''' Enable serial ports on the host. {{project_name_short}} VM settings → Serial Ports → Tick enable Configure the following settings: * Port Number: COM1 (default) * Port Mode: Host Pipe * Option: '''un'''check Connect to existing pipe/socket * Path/Address: /home/user/vbox-socket-file If an error like the following appears. 
Failed to open a session for the virtual machine Kicksecure-Xfce_15.0.0.4.9.

NamedPipe#0 failed to connect to local socket /home/user/vbox-socket-file (VERR_FILE_NOT_FOUND).

Result Code: NS_ERROR_FAILURE (0x80004005)
Component: ConsoleWrap
Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
Then you must uncheck Connect to existing pipe/socket. {{CodeSelect|code= vboxmanage modifyvm {{project_name_short}}-Xfce --uart1 0x3F8 4 --uartmode1=server /home/user/vbox-socket-file }} '''2.''' Connect to the relevant unix domain socket file. On the host, run the following socat command to connect to the unix domain socket file which is connected to the operating system running inside the virtual machine. {{CodeSelect|code= socat - UNIX-CONNECT:/home/user/vbox-socket-file }} {{CodeSelect|code= socat - UNIX-CONNECT:/home/user/vbox-socket-file }} Depending on when the above command is run, nothing might appear. The reason is an interactive serial console will only show messages once connected to the serial console; old messages cannot be viewed that way. If the above command is run during early boot, then verbose messages will appear during boot. However, if you press Enter that should result in the virtual console asking for authentication. '''3.''' Log in to the {{project_name_short}} session. Press Enter. The following prompt will appear. 
host login:
Do not enter your host login username! Enter your {{project_name_short}} user login name, which is most likely user. Press Enter. The following prompt will appear. 
Password:
Enter the password for that user account; see [[Default Passwords]]. Press Enter. Warning: the password will not be hidden by asterisk ("*") symbols. In other words, the password will be written in cleartext and could be read by anyone looking over your shoulder. Also note that [[Root|root logins]] are not possible by default. }} === Forum Discussion === https://forums.whonix.org/t/serial-console-in-virtualbox/8021 == KVM == See [[KVM]], [[KVM#Command_Line_Interface_.28CLI.29|serial console]]. = Qubes specific = == Non-Bootable Qubes VM == This is [[unspecific|unspecific to {{project_name_short}}]]. Therefore the primary contact of support must be the Qubes community, not {{project_name_short}}. Always have a full, tested (can be restored) backup first before proceeding. A lot of the following ideas are [[undocumented]]. Ideas: * '''A)''' Set the broken VM's NetVM setting to "none" (optional), change to a different Template (perhaps a re-installed Template). * '''B)''' Can a Qubes VM be booted from a virtual ISO? (Such as for example the Kicksecure [[ISO]]? * '''C)''' Can a Qubes VM be booted into recovery mode? * '''D)''' Show full console window ("desktop") of Qubes VM by using Qubes "debug mode"? * '''E)''' Can Qubes boot from a different VM, assign the other VMs virtual hard drive, mount it, make changes to fix it, unmount it, assign back to original VM and boot it? -> https://www.qubes-os.org/doc/mount-lvm-image/ * '''F)''' If the Qubes /rw/rc.local mechanism is broken inside the VM: ** Boot the Template, disable Qubes' mechanism to process /rw/rc.local. ** As a hack, an early running systemd unit configured in the Template could delete /rw/rc.local at VM startup before Qubes processes it. = See Also = * [[SysRq|System Recovery using SysRq Key]] * [[debug-misc]] * [[systemcheck]] * [[Factory Reset]] * [[Broken Boot]] = Footnotes = {{reflist|close=1}} {{Footer}} [[Category:Documentation]]