8.7. Recovering a Broken System

8.7. Recovering a Broken System
	Chapter 8. Next Steps and Where to Go From Here

Sometimes, things go wrong, and the system you've carefully installed is no longer bootable. Perhaps the boot loader configuration broke while trying out a change, or perhaps a new kernel you installed won't boot, or perhaps cosmic rays hit your disk and flipped a bit in /sbin/init. Regardless of the cause, you'll need to have a system to work from while you fix it, and rescue mode can be useful for this.

There are several options to rescue a broken Ubuntu system on s390x:

The zipl boot menu offers by default an option to boot an old kernel via the boot option number 2.
```
zIPL v2.3.0-build-20180425 interactive boot menu
  
 0. default (ubuntu) 
 
 1. ubuntu 
 2. old 
Please choose (default will boot in 10 seconds):
```
Boot problems are often introduced with a kernel change. So there is a good chance that the prior / old kernel will still work.

Another option is to use the installation kernel and initrd in it's rescue mode (sometimes also known as recovery mode). To activate the rescue mode, one needs to boot with the “rescue/enable=true” boot parameter, and this parameter needs to be added to the parmfile.ubuntu file. This is slightly different for an LPAR compared to a z/VM guest.

For an LPAR change to the network install server, where the image got extracted and enter the image's ./boot folder. You can now just open the parmfile.ubuntu, add the “rescue/enable=true” boot parameter and proceed like you would usually do for an LPAR installation, means with the HMC task Load from Removable Media or Server.

But at this point it's better to add a dedicated boot entry for the rescue mode. Therefore you need to copy the parmfile.ubuntu and the ubuntu.ins to create seperate versions for the rescue mode which finally result in a separate rescue mode entry at the Load from Removable Media or Server task:

$ cd <path to the extracted ubuntu server image>/ubuntu-server-20.04/boot
$ cp ubuntu.ins rescue.ins
$ cp parmfile.ubuntu parmfile.rescue

Modify the new rescue.ins file from the default:

$ cat rescue.ins
* Ubuntu for z Series (default kernel)
kernel.ubuntu 0x00000000
initrd.off 0x0001040c
initrd.siz 0x00010414
parmfile.ubuntu 0x00010480
initrd.ubuntu 0x01000000

and point to the new parmfile.rescue:

$ cat rescue.ins
* Ubuntu for z Series (default kernel)
kernel.ubuntu 0x00000000
initrd.off 0x0001040c
initrd.siz 0x00010414
parmfile.rescue 0x00010480
initrd.ubuntu 0x01000000

And open parmfile.rescue and add “rescue/enable=true” (the parmfile is empty by default):

$ cat parmfile.rescue
rescue/enable=true

Now start the Load from Removable Media or Server task, select FTP source, enter the data of your network boot (FTP) server, and the step Select the software to install you will find in addition to the default entry another entry for the rescue mode:

Load from Removable Media or Server - Select Software to Install - P00ABCDE:LINLPAR
Select the software to install.
Select | Name                                | Description
  *    | ubuntu-server-20.04/boot/rescue.ins | Ubuntu for z Series (default kernel)
  o    | ubuntu-server-20.04/boot/ubuntu.ins | Ubuntu for z Series (default kernel)

Make sure ubuntu-server-20.04/boot/rescue.ins is selected and proceed with OK.

For a z/VM guest you need to access the Minidisk where the installer files, including the parmfile.ubuntu is located, add the “rescue/enable=true” boot parameter and proceed like you would usually do for a z/VM installation, means by executing the ubuntu exec REXX script. But here it's again better to add a dedicated boot script for the rescue mode. Therefore you need to copy the parmfile ubuntu and the ubuntu exec to create seperate versions for the rescue mode:

Logon to your z/VM system, have CMS active (ipl-ed) and make sure you have r/w acces to the Minidisk where the installations files are located. In this example “File Mode” (Fm) “A” is used, that might be different on your system. Copy the following files:

copyfiles ubuntu exec a rescue exec a 
copyfiles parmfile ubuntu a parmfile rescue a

listfiles rescue * a
RESCUE EXEC     A1
listfiles * rescue a
PARMFILE RESCUE A1

Change PARMFILE RESCUE (by default it's empty):

x PARMFILE RESCUE A
* * * Top of File * * *
                      
* * * End of File * * *

And add the rescue parameter:

x PARMFILE RESCUE A
* * * Top of File * * *
rescue/enable=true     
* * * End of File * * *

Now modify the RESCUE EXEC file as well so that it point to the new parmfile rescue file:

x RESCUE EXEC A
* * * Top of File * * *
/* REXX EXEC TO IPL Ubuntu's Rescue Mode */
/* z Systems FROM THE VM READER.         */
/*                                       */
'CP CLOSE RDR'                          
'PURGE RDR ALL'                         
'SPOOL PUNCH * RDR'                     
'PUNCH KERNEL    UBUNTU * (NOHEADER'   
'PUNCH PARMFILE  RESCUE * (NOHEADER'   
'PUNCH INITRD    UBUNTU * (NOHEADER'   
'CHANGE RDR ALL KEEP NOHOLD'            
'CP IPL 000C CLEAR'                     
* * * End of File * * *

Now start the z/VM guest installation as usual, but using the newly created REXX script for the rescue mode, hence just type: rescue <Enter>

Don't worry, your system is not about to be overwritten! “Rescue” mode simply takes advantage of the hardware detection facilities available in the installer to ensure that your disks, network devices, and so on are available to you while repairing your system.

Using either of the two installation types you'll be shown the first few screens of the installer, with a note in the corner of the display to indicate that this is rescue mode, not a full installation:

Rescue mode
                        [?] Ubuntu installer main menu                         
                                                                              
              Choose the next step in the install process:                    
                                                                              
                   Configure the network device                               
                   Configure the network                                      
                   Choose language                                            
                   Choose a mirror of the Ubuntu archive                      
                   Download installer components                              
                   Change debconf priority                                    
                   Save debug logs                                            
                   Execute a shell                                            
                   Abort the installation

Don't worry, your system is not about to be overwritten! Rescue mode simply takes advantage of the hardware detection facilities available in the installer to ensure that your disks, network devices, and so on are available to you while repairing your system.

Instead of the partitioning tool, you should now be presented with a list of the partitions on your system, and asked to select one of them. Normally, you should select the partition containing the root file system that you need to repair. You may select partitions on RAID and LVM devices as well as those created directly on disks:

Rescue mode                                                                     
                            [!!] Enter rescue mode
                                                                              
   Enter a device you wish to use as your root file system. You will be       
   able to choose among various rescue operations to perform on this          
   file system.                                                               
                                                                              
   If you choose not to use a root file system, you will be given a           
   reduced choice of operations that can be performed without one. This       
   may be useful if you need to correct a partitioning problem.               
                                                                              
   Device to use as root file system:                                         
                                                                              
                      /dev/sda1                                               
                      /dev/sdb1                                               
                      Assemble RAID array                                     
                                                                              
       <Go Back>

If possible, the installer will now present you with a shell prompt in the file system you selected, which you can use to perform any necessary repairs.

If the installer cannot run a usable shell in the root file system you selected, perhaps because the file system is corrupt, then it will issue a warning and offer to give you a shell in the installer environment instead. You may not have as many tools available in this environment, but they will often be enough to repair your system anyway. The root file system you selected will be mounted on the /target directory.

A trivial option is to just boot the standard installation kernel and initrd without any additional kernel parameter, and select from the inital screen the Start shell entry.

                             [!!] Configuring d-i
                                                                              
   This is the network console for the Debian installer. From here, you       
   may start the Debian installer, or execute an interactive shell.           
                                                                              
   To return to this menu, you will need to log in again.                     
                                                                              
   Network console option:                                                    
                                                                              
                      Start installer                                         
                      Start installer (expert mode)                           
                      Start shell

The functionality in the debian-installer shell is limited, however, it can still act as a rescue system to fix a broken installation. While not using the rescue mode, be careful to not accidentially repartition or format any disk which may cause data loss.

In either case, after you exit the shell, the system will reboot.

Finally, note that repairing broken systems can be difficult, and this manual does not attempt to go into all the things that might have gone wrong or how to fix them. If you have problems, consult an expert.


8.6. Compiling a New Kernel		Appendix A. Installation Howto