RHEL 6 Rescue Mode LVM Recovery

RHEL 6 Rescue Mode LVM Recovery

So there I was on on Friday taking a day off when I get a call that one of our key database servers was down. On the console we were getting the error 'telinit: did not receive a reply'. This was the first time I'd ever come across this error. A bit of internet digging and I found that this is the process that escalates through the run levels. I also found a Red Hat article pointing out that this could be a corruption with the passwd file. Time to break out that RHEL install disc!

To boot into rescue mode, insert/mount your RHEL media and boot to it. One of the menu options will be to boot into rescue mode.

Once in rescue mode you will need to set language options then you will be given the option to setup networking which you will need to do if you want to restore from a backup. Finally it will ask if you want to scan for linux partitions or if you want to skip this. Let it try and find your linux partitions but I didn't have any luck with this. After failing to find your partitions, select the option to go to the rescue shell.

From here you will need to try and mount your partitions so you can get at your filesystem and start looking for the problem.

First off, run the command lvm vgscan -v to scan for volume groups

Next, activate your volume groups by using the command lvm vgchange -a y

Now you can list your logical volumes using the command lvm lvs

Finally, create a directory to mount your volume to using the mkdir command such as mkdir /recovery and mount your volume group to it using the mount command such as mount /dev/mapper/volgrp-vg1/logvol-lv1 /recovery

With your logical volume now mounted you can start looking for your issue. I found that my passwd file had gone blank! I restored this by taking a copy of my passwd from a backup which i restored to another server then scp'd this back to this server (why it is a good idea to network rescue mode!).

With the passwd file restored I issued the reboot command and the server came up successfully. I did find that all of my group permissions were also missing so had to also restore the /etc/group file.