RHEL 6 Rescue Mode LVM Recovery

RHEL 6 Rescue Mode LVM Recovery

So there I was on on Friday taking a day off when I get a call that one of our key database servers was down. On the console we were getting the error 'telinit: did not receive a reply'. This was the first time I'd ever come across this error. A bit of internet digging and I found that this is the process that escalates through the run levels. I also found a Red Hat article pointing out that this could be a corruption with the passwd file. Time to break out that RHEL install disc!

To boot into rescue mode, insert/mount your RHEL media and boot to it. One of the menu options will be to boot into rescue mode.

Once in rescue mode you will need to set language options then you will be given the option to setup networking which you will need to do if you want to restore from a backup. Finally it will ask if you want to scan for linux partitions or if you want to skip this. Let it try and find your linux partitions but I didn't have any luck with this. After failing to find your partitions, select the option to go to the rescue shell.

From here you will need to try and mount your partitions so you can get at your filesystem and start looking for the problem.

First off, run the command lvm vgscan -v to scan for volume groups

Next, activate your volume groups by using the command lvm vgchange -a y

Now you can list your logical volumes using the command lvm lvs

Finally, create a directory to mount your volume to using the mkdir command such as mkdir /recovery and mount your volume group to it using the mount command such as mount /dev/mapper/volgrp-vg1/logvol-lv1 /recovery

With your logical volume now mounted you can start looking for your issue. I found that my passwd file had gone blank! I restored this by taking a copy of my passwd from a backup which i restored to another server then scp'd this back to this server (why it is a good idea to network rescue mode!).

With the passwd file restored I issued the reboot command and the server came up successfully. I did find that all of my group permissions were also missing so had to also restore the /etc/group file.

Installing VMWare Tools from a Red Hat Shell

Installing VMWare Tools from a Red Hat Shell

Installing and Upgrading VMware tools on Red Hat is very straightforward. In the steps below I will demonstrate what to do assuming that you are happy to install under /opt and accepting the defaults during the installation. There are some advanced features you can choose to enable during installation as well as choosing different locations for the software to find key files.

From vSphere right click the Virtual Machine, guest, Install/Upgrade VMWare Tools. Next run the commands below.

1. mkdir -p /mnt/cdrom          - This creates the directory /mnt/cdrom which we will mount the VMtools installer to
2.  mount /dev/cdrom /mnt/cdrom -This mountes the media the virtual machine see's as it's CDROM drive to the directory /mnt/cdrom
3. cd /opt
4. cp /mnt/cdrom/VMversionname.tar.gz /opt
5. tar -xvzf VMversionname.tar.gz
6. cd vmware-tools-distrib
7. ./VMware-install.pl
8.  Hit enter to accept each default answer through the installation

Rebalance databases in a DAG on Exchange 2010

Sometime you will find that your mailbox database will all be mounted on one of your mail servers where they should be split between members of your DAG. This can be caused but network outages, server crashes and more.

Microsoft have included a powershell script that you can run to quickly fix this. By running this script with the options below, Exchange will attempt to put all mailbox databases back on their preferred servers based on Activation Preference.

From an exchange server, launch the Exchange Management Shell

1. cd "C:\Program Files\Microsoft\Exchange Server\V14\Scripts"

2. .\RedistributeActiveDatabases.ps1 -DagName nameofyourdag -BalanceDbsByActivationPreference

3. For each mailbox that it says Considering move of 'DATABASE NAME' from 'SERVER' to 'SERVER' type Y and press enter. There is an option to say yes to all but i've never had any success out of it.

4. When its finished, open the exchange management console and check the screen above. It should show half mounted on one and half on the other.

If you get a message come up saying that a move has failed because the content index state is failed you will need to run the following command in the exchange management shell on the server which the content state is failed on.

Update-MailboxDatabaseCopy "MAILBOX_DATABASE_NAME\SERVER_NAME" -CatalogOnly

Creating Shared Mailboxes in Microsoft Exchange 2010

Creating Shared Mailboxes in Microsoft Exchange 2010

In Exchange 2010 you can create a shared mailbox which doesn't rely on a user account being active to use it. This is better security and will stop exchange from telling you that you need a CAL for the account. Once the account is created you will see a disabled user in the OU you specify below. This user should not be touched.


To do this, log onto you Exchange server and launch the Exchange Management Shell. This will open a special powershell module with the exchange cmdlets loaded.

In the example I give you, you will be creating a mailbox called Quarantine. In this window, run a modified version of this:

New-Mailbox –Name Quarantine –Alias quarantine –OrganizationalUnit “yourdomain/path/to/organizational/unit” –Database “name of database to store mailbox” –UserPrincipalName quarantine@youremaildomain.com -Shared

Below is a breakdown of what this will do:

New-Mailbox at the start indicates that you are creating a new mailbox
-Name is the flag to say whatever follows it will be the name of the mailbox, in this case Quarantine
-Alias is the flag to say whatever follows it will be the alias/user logon name of the mailbox, in this case quarantine
–OrganizationalUnit is the flag to say whatever follows it will be where the account is stored in active directory
-Database is the mailbox store in which the mailbox will be created
–UserPrincipalName is the flag to say whatever follows it will be the email address created for the account
-Shared is the flag that goes on the end to say this is going to be created as shared type of mailbox (free of particular ownership)

How to enable/perform a rollback of Yum updates on RHEL/CentOS

How to enable/perform a rollback of Yum updates on RHEL/CentOS

The ability to rollback updates is not enabled by default. To enable the rollback feature follow these steps:

Use a text editor (vi for example) to open /etc/yum.conf

Add the line

tsflags=repackage

Save and exit the file

Add the following to /etc/rpm/macros (if the file doesn't exist, create it)

%_repackage_all_erasures 1


You can now rollback to previous packages by using the following commands to go back to a specified time

rpm -Uvh -rollback '6 hours ago'
rpm -Uvh -rollback 'january 1'
rpm -Uvh -rollback 'yesterday'
rpm -Uvh -rollback '18:00'


Be aware that the repackaged software will be stored in /var/spool/repackage so make sure you have free disk space on this partition!