Posts Tagged: ‘Luster’

Building a Small Lustre storage cluster

January 6, 2011 Posted by admin

 

Building Clusters with distributed storage is one of the hottest things IT professionals are doing these days. One of the key reasons is scaling storage and improving disk I/O. This posting outlines the basics needed for creating a Lustre storage Cluster using CentOS 5.5.

Luster Network Design

MDS (Metadata Server)
MDT(Metadata Target)
OSS(Object Storage Servers)
OST(Object Storage Targets)
For this setup we will be combining services on because of lack of hardware

On the MDT the following luster software needs to be installed

rpm -ivh kernel-2.6.18-164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-modules-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-ldiskfs-3.2.0-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm

On the OST’s the following luster software needs to be installed.

rpm -ivh kernel-2.6.18-164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-modules-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-ldiskfs-3.2.0-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm

On the client the following luster software needs to be installed.

rpm -ivh kernel-2.6.18-164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-modules-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-client-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm
rpm -ivh lustre-client-modules-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm

After installing the Lustre software and kernel on each server edit your grub.conf so that it boots the lustre kernel and reboot each hosts.

vim /etc/grub.conf
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu

title CentOS (2.6.18-164.11.1.el5_lustre.2.0.0.1)
root (hd0,0)
kernel /vmlinuz-2.6.18-164.11.1.el5_lustre.2.0.0.1 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-164.11.1.el5_lustre.2.0.0.1.img

title CentOS (2.6.18-194.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.el5 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-194.el5.img

reboot

Next lets edit the hosts files so we have the hosts names of each server and ip address.

vim /etc/hosts

192.168.0.10 client01.your.org
192.168.0.11 mds.your.org
192.168.0.12 ost1.your.org
192.168.0.13 ost2.your.org

Also add the following line to your modprobe.conf to tell lustre on boot to load the tcp modules

vi /etc/modprobe.conf
options lnet networks=tcp0

Next lets format the filesystems on the MDS and OST servers.

On the MDS (mds.your.org@tcp0):

mkfs.lustre –mdt –mgs –fsname=large-fs /dev/sdb
mount -t lustre /dev/sda /mnt/mdt

On OSS1:

mkfs.lustre –ost –fsname=large-fs –mgsnode=mds.your.org@tcp0 /dev/sdb
mount -t lustre /dev/sdb/mnt/ost1

On OSS2:

mkfs.lustre –ost –fsname=large-fs –mgsnode=mds.your.org@tcp0 /dev/sdc
mount -t lustre /dev/sdc/mnt/ost2

Client

mkdir -p /mnt/test
mount -t lustre mdss.your.org:/large-fs /mnt/test