{{Header}} __FORCETOC__ {{#seo: |description=Recovery Mode, Emergency, Chroot, Debugging, Capturing Diagnostic Output from Serial Console |image=Recovery312123.jpg }} [[image:Recovery312123.jpg|thumb]] {{intro| The goal of this wiki page is to provide the user with knowledge for higher system reliability, robustness, increases uptime, backup, restoration, recovery and repair skills. }} = Introduction = It is prudent to acquire system recovery skills prior to an eventual later grave issue such as an unbootable VM or even unbootable host operating system, data loss or similar. This is because naturally for most users once they are affected by a grave issue, the mind goes into "issue solving mode", which can be stressful and less productive, specifically when it is currently a bad time to acquire these skills due to other more pressing priorities. By learning these skills in advance, one can the skills during a more appropriate time in in a much more positive, relaxed environment which might potentially greatly increase a user's ability and lower the time required to fix their broken system, should this happen in the future. = Useful Skills = * Boot an {{os}} from an external USB / DVD boot drive. * Boot an {{os}} from an external USB / DVD boot drive and mount the OS which is installed on the internal hard drive to access or backup its data. For Qubes, see: https://www.qubes-os.org/doc/mount-from-other-os/ Yes, this is also possible when using [[Full Disk Encryption]]. * Remove an internal drive from one computer, move it into a USB enclosure and access its data from a different computer. * [[Grub#Kernel_Boot_Parameter_Change|Temporary Kernel Boot Parameter Change]] using [[grub]] bootloader boot menu. * [[Grub#Permanent_Configuration_Changes|Permanent Kernel Boot Parameter Configuration Changes]]. * [[Grub#Inspect_Grub_Configuration_Changes|Inspect Grub Configuration Changes]]. * [[Raw Disk Backup|Raw Disk Backup and Restoration]]. * [[Recovery#Recovery_Mode|How to boot into recovery mode]]. * How to switch to a [[Desktop#Virtual_Consoles|virtual console]] in a {{VM}}. * How to switch to a virtual console on the host operating system. * [[Troubleshooting#Enable_Persistent_Systemd_Journal_Log|Enable Persistent Systemd Journal Log]] * [[Troubleshooting#Check_Systemd_Journal_Log_of_Previous_Boot|Check Systemd Journal Log of Previous Boot]] * [[Troubleshooting#Check_Systemd_Journal_Log_of_Failed_Boot|Check Systemd Journal Log of Failed Boot]] * hard drive health check (S.M.A.R.T) * Topics mentioned on the [[Troubleshooting]] wiki page. * Other topics mentioned on this wiki page. = Useful Habits = * Keep the log of any [[Operating System Software and Updates|system upgrades]] as well as [[Install Software|software installation]] processes. * Keep one or multiple root virtual terminal tabs in a terminal emulator open during actions which might break to unbootable systems, data loss or other issues during software installation or upgrades. The same goes for one or multiple root [[Desktop#Virtual_Consoles|virtual consoles]]. This advice (for higher ability to easily recover from issuing gaining root) conflicts with the [[Root#General_Security_Advice|general security advice]] from the [[root]] wiki page (for better security). = Recovery Mode = If an error occurs that prevents {{project_name_long}} from booting, it is possible to try and boot {{project_name_long}} in recovery mode to fix the problem. This is possible in many cases but not for example if grub is broken. It is advisable to learn how to use recovery mode while everything is still functional in case of future issues. {{Box|text= # After powering on a {{project_name_short}}, you will see the virtual BIOS for a second, then the grub boot menu. The grub boot menu is easily identified by the first line of text which begins with GNU GRUB. # Use the arrow keys to navigate and select Advanced Options for {{project_name_short}} GNU/Linux. # Press Enter. # Choose the second option which at the end displays (recovery mode). # Press Enter. # Somewhere in the output, a message similar to the following will appear (it might be entangled with other debug output):
Give root password for maintenance (Or press control + d to continue):
# Enter the root password. # While typing, no asterisk symbols (*) will appear; the password must be typed "blind". See also: [[Default_Passwords|Default Passwords]]. # Press Enter. # The default keyboard layout will be en-US. To change this setting, see: [[Keyboard Layout]]. # Enter exit to continue booting or poweroff or reboot. }} TODO: Explore whether networking is possible and how to transfer files out of the VM. = Virtual Consoles = An easier and more lightweight solution as alternative to [[#Recovery Mode|recovery mode]] might be [[Desktop#Virtual_Consoles|virtual consoles]]. If the graphical user interface is no longer starting, login to a virtual console might still be possible. '''1.''' Prerequisite knowledge: [[Desktop#Virtual Consoles|Virtual Consoles]]. '''2.''' Try to login in a virtual console in a different, still functional VM (virtual machine) as an exercise. '''3.''' If that works, try login to virtual console in the broken VM. = Hard Drive Recovery = If no data needs be recovered, a complete re-installation might be easier. '''1.''' Choose unbootable host or {{VM}}. {{Tab |type=controller |content= {{Tab |title= ==Unbootable Host Operating System== |type=section |image= |addToClass=info-box |active= |content= If the host got unbootable due to hard drive issues: '''2.''' Damage control. Stop using the hard drive. This is because further use might add further damage. '''3.''' Make a full 1:1 raw clone of the hard drive. }} {{Tab |title= ==Unbootable VM== |type=section |image= |addToClass=info-box |active= |content= If a VM got unbootable due to hard drive issues: Stop using the VM. This is because further use might add further damage. '''2.''' Damage control. Stop using the VM. Create a full clone of the VM or hard drive. '''3.''' Make a clone of the VM and/or hard drive. }} }} '''4.''' Data recovery. [[Unspecific]] to {{project_name_short}}. Often [[#Boot from External Drive|Boot from External Drive]] will help. == Boot from External Drive == {{Anchor|Boot Virtual Machine from ISO instead of Virtual Hard Drive}} This is useful for purposes of data recovery, [[Grow Virtual Harddisk]], [[Shrink Virtual Harddisk]], or installing a new operating system. Which ISO to use? The [[ISO|{{project_name_short}} ISO]] should be suitable for these purposes. {{Tab |type=controller |content= {{Tab |title= ===Host Operating System=== |type=section |image= |addToClass=info-box |active= |content= Note: You need to burn a Kicksecure ISO on an external Flash or DVD before proceeding with the below steps. '''1.''' Place the bootable Flash or DVD into your PC's. '''2.''' Restart your computer if it's already on. If it's off, turn it on. '''3.''' During the boot process, a message will typically appear on the screen telling you to press a specific key to enter the BIOS/UEFI setup. Common keys are F2, F12, Delete, or Esc. '''4.''' Once inside the BIOS/UEFI, look for a section related to "Boot Order" or "Boot Priority" (Use the keyboard to navigate). '''5.''' Set the Flash or DVD drive as the first boot device. This tells your computer to try booting from the Flash or DVD drive before any other devices. '''6.''' After setting the boot order, find the option to Save Changes and Exit. The exact wording might vary, but it’s typically something like "Save & Exit" or "Exit Saving Changes". Confirm your changes, and your PC will restart. '''7.''' If everything is set correctly, your PC should now boot from the Flash or DVD. You might see a message like "Press any key to boot from CD/DVD..." If you see this message, press any key on your keyboard to start booting from the DVD. '''8.''' Done. }} {{Tab |title= ===VirtualBox=== |type=section |image=[[File:Logo-virtualbox.png|25px]] |addToClass=info-box |active= |content= Booting a Debian-based VirtualBox VM (such as {{project_name_short}} or Whonix, which are derivatives based on Debian) using an ISO file instead of the internal disk involves changing the boot order and attaching the ISO file as a virtual CD/DVD drive. Here are the steps to do this: '''1.''' Launch VirtualBox from your applications menu. In the VirtualBox Manager, click on the VM that you want to boot from an ISO, then click on the "Settings" button (a gear icon) at the top. [[File:btext1.png|600px]] '''2.''' Configure Storage: In the Settings window, go to the "Storage" tab. You will see a tree structure with a controller (like IDE or SATA) and attached storage devices. Click on the CD/DVD drive icon with a "+" symbol attached to it. [[File:btext2.png|470px]] [[File:btext3.png|470px]] '''3.''' Add ISO File: On the top right corner of the window, you will see an icon of a CD/DVD. Click on this icon. Navigate to the location of your ISO file on your host machine. Select the ISO file, then press "Choose" The ISO file should now be attached to the VM as a virtual CD/DVD drive. [[File:btext4.png|600px]] [[File:btext5.png|600px]] [[File:btext6.png|600px]] '''4.''' Configure Boot Order: Go to the "System" tab in the Settings window. Click on the "Motherboard" tab within the System settings. In the "Boot Order" section, ensure that "Optical" is checked and listed above "Hard Disk." You can reorder them by dragging or using the up/down arrows.Then click "OK" to save the settings. [[File:btext7.png|470px]] '''5.''' Save and Start VM. [[File:btext8.png|470px]] [[File:btexd9.png|490px]] '''6.''' Complete Your Task. * Proceed with your intended task (installation, recovery, etc.) using the booted ISO. * Once you've finished and no longer need to boot from the ISO, shut down the VM. '''7.''' Detach ISO After Use: After the VM has been powered off, go back to the "Storage" tab in the VM settings. Click on the ISO file that you previously attached. Click on the CD/DVD icon on the right and choose "Remove disk from virtual drive." [[File:btexd10.png|470px]] '''8.''' Revert Boot Order (if needed): Go to the "System" tab in the Settings window. Click on the "Motherboard" tab within the System settings. In the "Boot Order" section, ensure that "Hard Disk" is checked and listed above "Optical." [[File:btexd11.png|470px]] '''9.''' Done. Your VM should now boot from the internal disk instead of the ISO file. '''Troubleshooting:''' * If the VM fails to boot from the ISO, ensure the ISO file is not corrupted and is compatible with your VM configuration. }} {{Tab |title= ===KVM=== |type=section |image=[[File:Logo-kvm.png|25px]] |addToClass=info-box |active= |content= Note: Kicksecure ISO downloaded in /home/user/Downloads on the host. '''1.''' Launch KVM (called Virtual Machine Manager) from your applications menu. [[File:vmiso.png|400px]] '''2.''' Click on the VM that you want to boot from an ISO, then click on the "Open" button (Monitor icon) at the top. [[File:vmiso2.png|600px]] '''3.''' Click on VM settings (bulb icon), then "SATA CDROM 1", and then "Browse" to choose the ISO. [[File:vmiso3.png|600px]] '''4.''' "Downloads" represents the path /home/user/Downloads where the ISO is located. If you click on it and find it empty, then click on the refresh icon. Next, select the ISO and press "Choose Volume". [[File:vmiso4.png|600px]] '''5.''' Make sure you have chosen the correct ISO, then click on "Apply". [[File:vmiso4-1.jpg|600px]] '''6.''' Press on "Boot Options", then tick "Enable boot menu", Tick and move up "SATA CDROM 1," and finally, press "Apply". [[File:vmiso5.png|600px]] '''7.''' Run the VM, and it should boot the Kicksecure ISO. [[File:vmiso6.png|600px]] [[File:vmiso7.png|600px]] '''8.''' If you want to go back to the original operating system, just revert the steps above. '''9.''' Done }} {{Tab |title= ===Qubes=== |type=section |image=[[File:Logo-qubes.png|25px]] |addToClass=info-box |active= |content= Note: You need to download the {{project_name_short}} ISO into a separate AppVM/Qube that will be used to boot {{project_name_short}} from (in the example below, the ISO has been downloaded in "debian-personal"). '''1.''' Launch "Qube Manager," click on the VM that you want to boot the ISO on (Kali-Linux StandaloneVM will be used as an example), then click on the "Qube Settings" button (a gear icon) at the top. [[File:qubesbootiso1.png|800px]] '''2.''' Click on "Advanced" then "Boot qube from CD-ROM" [[File:qubesbootiso2.png|600px]] '''3.''' Choose "from file in qube" then select your AppVM where you downloaded the ISO ("debian-personal" is used here) then Press "..." to choose the path of the downloaded ISO. [[File:qubesbootiso3.png|600px]] '''4.''' Choose the downloaded ISO from the AppVM path where you have downloaded it. [[File:qubesbootiso4.png|600px]] '''5.''' Make sure everything is as it should be, then press "OK". [[File:qubesbootiso5.png|600px]] '''6.''' {{project_name_short}} ISO should boot perfectly. [[File:qubesbootiso6.png|600px]] [[File:qubesbootiso7.png|600px]] '''7.''' If you shut down {{project_name_short}} live mode and reopen the VM, the ISO will be detached automatically, and the VM will boot the installed/default operating system. '''8.''' Done. }} }} = Chroot = If an error occurs that prevents {{project_name_short}} from booting, it is possible to [https://www.computerhope.com/unix/chroot.htm chroot] into {{project_name_short}} from a live CD to fix the problem. A disk image must be downloaded for this purpose -- [https://www.debian.org/CD/live/ Debian Live images] or [https://www.archlinux.org/download/ Arch ISO] are useful options for this procedure. === VirtualBox === In VirtualBox, navigate to the VM storage settings. At the second controller, click the optical disk icon and add the disk image. In the Motherboard settings, move the optical disk component to the top of the boot order, press OK and start the VM. === KVM === In virt-manager, click on the VM, click the settings icon at the top and click "Add Hardware". In "Storage", change the device type to "CDROM device" and add the disk image. Click "Finish" and at the boot options setting, move the CDROM to the top of the boot order. === {{project_name_short}} === This procedure is [[undocumented]] and unspecific to {{project_name_short}}. See [[unspecific]]. == Booting into the Live CD == After booting into the Live CD, follow these steps. {{Box|text= '''1.''' List the available drives and partitions. {{CodeSelect|code= fdisk -l }} '''2.''' Mount the partition. {{CodeSelect|code= mount /dev/sda1 /mnt }} Replace "/dev/sda1" with the drive partition name. '''3.''' Chroot into the partition. {{CodeSelect|code= chroot /mnt /bin/bash }} '''4.''' ''Optional:'' If the PATH variable is incorrect, it can be fixed with the following command. {{CodeSelect|code= source /etc/profile }} }} = Serial Console = == VirtualBox == === Preparation === ==== Host Preparation ==== Install socat on the host operating system. The following steps apply to Linux distributions. {{Install Package|package= socat }} It is advised to research these using a non-{{project_name_short}} VM such as Debian {{Stable project version based on Debian codename}} and [[unspecific|learn it in a generic way]] which could then also be applied to {{project_name_short}}. ==== VM Preparation ==== Inside the {{project_name_short}} virtual machine where you want to enable the serial console, install the serial-console-enable package. {{Install Package|package= serial-console-enable }} (Installed by default would be nice but it is not installed by default due to issue. TODO: add reference) === Read Only Serial Console === This option is very useful for capturing diagnostic output from a virtual machine. It allows a log of everything written to the kernel console to be obtained -- all console output is written to a plain text file from boot until shutdown. The logfile persists after VM shutdown, but is overwritten after the VM is shutdown and powered on again. {{Box|text= '''1.''' Enable the serial ports option. 3 {{project_name_short}} VM settings → Serial Ports → Tick enable Configure the following settings: * Port Number: COM1 (default) * Port Mode: Raw File * Path/Address: /home/user/vbox-raw-file '''2.''' View the file with a text editor. Any text editor can be used for this purpose. For example to view the file with mousepad, run. {{CodeSelect|code= mousepad /home/user/vbox-raw-file }} '''3.''' ''Optional:'' View the file as it is being appended to from a terminal emulator. The logfile is also compatible with other standard linux utlitites such as tail. To view it as it is being written, run. {{CodeSelect|code= tail -f /home/user/vbox-raw-file }} '''4.''' Retain the file contents. After VM shutdown it may be useful to retain its contents by copying the vbox-raw-file elsewhere to make it a persistent log file. If this is desirable, run. {{CodeSelect|code= cp /home/user/vbox-raw-file /home/user/vbox-console-log }} }} === Interactive Serial Console === This procedure does not yet work during the grub boot menu. {{Box|text= '''1.''' Enable serial ports on the host. {{project_name_short}} VM settings → Serial Ports → Tick enable Configure the following settings: * Port Number: COM1 (default) * Port Mode: Host Pipe * Option: '''un'''check Connect to existing pipe/socket * Path/Address: /home/user/vbox-socket-file If an error like the following appears. 
Failed to open a session for the virtual machine Kicksecure-Xfce_15.0.0.4.9.

NamedPipe#0 failed to connect to local socket /home/user/vbox-socket-file (VERR_FILE_NOT_FOUND).

Result Code: NS_ERROR_FAILURE (0x80004005)
Component: ConsoleWrap
Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
Then you must uncheck Connect to existing pipe/socket. {{CodeSelect|code= vboxmanage modifyvm {{project_name_short}}-Xfce --uart1 0x3F8 4 --uartmode1=server /home/user/vbox-socket-file }} '''2.''' Connect to the relevant unix domain socket file. On the host, run the following socat command to connect to the unix domain socket file which is connected to the operating system running inside the virtual machine. {{CodeSelect|code= socat - UNIX-CONNECT:/home/user/vbox-socket-file }} {{CodeSelect|code= socat - UNIX-CONNECT:/home/user/vbox-socket-file }} Depending on when the above command is run, nothing might appear. The reason is an interactive serial console will only show messages once connected to the serial console; old messages cannot be viewed that way. If the above command is run during early boot, then verbose messages will appear during boot. However, if you press Enter that should result in the virtual console asking for authentication. '''3.''' Log in to the {{project_name_short}} session. Press Enter. The following prompt will appear. 
host login:
Do not enter your host login username! Enter your {{project_name_short}} user login name, which is most likely user. Press Enter. The following prompt will appear. 
Password:
Enter the password for that user account; see [[Default Passwords]]. Press Enter. Warning: the password will not be hidden by asterisk ("*") symbols. In other words, the password will be written in cleartext and could be read by anyone looking over your shoulder. Also note that [[Root|root logins]] are not possible by default. }} === Forum Discussion === https://forums.whonix.org/t/serial-console-in-virtualbox/8021 == KVM == See [[KVM]], [[KVM#Command_Line_Interface_.28CLI.29|serial console]]. = Qubes specific = == Non-Bootable Qubes VM == This is [[unspecific|unspecific to {{project_name_short}}]]. Therefore the primary contact of support must be the Qubes community, not {{project_name_short}}. Always have a full, tested (can be restored) backup first before proceeding. A lot of the following ideas are [[undocumented]]. Ideas: * '''A)''' Set the broken VM's NetVM setting to "none" (optional), change to a different Template (perhaps a re-installed Template). * '''B)''' Can a Qubes VM be booted from a virtual ISO? (Such as for example the Kicksecure [[ISO]]? * '''C)''' Can a Qubes VM be booted into recovery mode? * '''D)''' Show full console window ("desktop") of Qubes VM by using Qubes "debug mode"? * '''E)''' Can Qubes boot from a different VM, assign the other VMs virtual hard drive, mount it, make changes to fix it, unmount it, assign back to original VM and boot it? -> https://www.qubes-os.org/doc/mount-lvm-image/ * '''F)''' If the Qubes /rw/rc.local mechanism is broken inside the VM: ** Boot the Template, disable Qubes' mechanism to process /rw/rc.local. ** As a hack, an early running systemd unit configured in the Template could delete /rw/rc.local at VM startup before Qubes processes it. = Unlock User Account: Excessive Wrong Password Entry Attempts = See: [[Root#Unlock_User_Account:_Excessive_Wrong_Password_Entry_Attempts|Unlock User Account: Excessive Wrong Password Entry Attempts]]. = Reset User Account Password = See [[Root#Reset_User_Account_Password|Reset User Account Password]]. = See Also = * [[SysRq|System Recovery using SysRq Key]] * [[debug-misc]] * [[systemcheck]] * [[Factory Reset]] * [[Broken Boot]] = Footnotes = {{reflist|close=1}} {{Footer}} [[Category:Documentation]]