wiki:OpenvzUmountLustrePatch
Last modified 3 years ago

Background

Lustre filesystem can be easily instructed to stopped when -f parameter is provided on umount command. As we can read in the lustre documentation:

To stop a server:

$ umount -f /mnt/test/ost0 The '-f' flag means "force"; force the server to stop WITHOUT RECOVERY.

Although it is a handy feature that might be used when dealing with unresponsive lustre server, it creates a possibility of carrying out a DOS attack by a malicious user.

The problem is only real if we have a hosting server running OpenVZ and a group of containers (used by outside users) mounting the same lustre filesystem. In such setup one of the users can type: 

umount -f /mnt/shared/lustre/fs

inside a container which will cause the lustre filesystem to stop without any warning, preventing all the other containers that use the same filesystem from accessing the data.

Such event gets logged to /var/log/messsages as: 

kernel: Lustre: setting import lustre-server-MDT0000_UUID INACTIVE by administrator request

Important: the above is also likely to happen when one of the containers gets stopped since a lot of linux distributions, as a part of of shutdown process, run umount -f command on the filesystems found in the /etc/mtab file.

Patch

This patch is for OpenVZ kernel. It prevents users from running umount -f command on lustre filesystems inside containers. If umount -f gets run inside a container -f option gets removed and a standard, not forced umount is performed writing an appropriate message to the logs. If umount -f gets run on the host server it will work as expected - forcing a lustre filesystem to stop.

Installation on CentOS

  1. Download the patch.
  1. Download OpenVZ  kernel.
  1. Install kernel src rpm: 
rpm -ihv ovzkernel-2.6.18-92.1.18.el5.028stab060.2.src.rpm
  1. Download a SPEC file to

/usr/src/redhat/SPECS replacing existing spec file.

  1. Copy patch: 
cp ovz-kernel-lustre-umount.patch /usr/src/redhat/SOURCES/
  1. Build kernel RPMs: 
    rpmbuild -ba --target=i686 /usr/src/redhat/SPECS/kernel-ovz.spec

It will compile the kernel apply the patch and produce ready to install rpms in /usr/src/redhat/RPMS.

  1. Install ovz kernel: 
rpm -ihv /usr/src/redhat/RPMS/i686/ovzkernel-PAE-2.6.18-92.1.18.el5.028stab060.2.i686.rpm
  1. Make sure that grub is setup to run the newly installed kernel and reboot your server.

Test

To verify that the installation has been successful and the patch is working as it should you can mount a lustre filesystem in one of your containers and try doing: 

umount -f /mnt/lustre/fs

You should get a message similar to: 

kernel: Forced umount of lustre fs is not allowed inside container (999). Overriding MNT_FORCE flag.

in your /var/log/messages file on the host server.

Blog entry describing the problem in more detail -  http://hoston.it/blog/2009/03/31/openvz-forced-umount-of-lustre-mount-problem/