From there we are presented with a list of services and applications that can be clustered.

Simply choose the service or application you want to make highly available…

A quick system check will ensure you have the required components in place.  The proper install order would be:

• Clustering Role
• Role, Service or Application to be clustered

We will choose the File Server role as it is already installed.

You will have to provide a name (used by NetBIOS and DNS) as well as the public IP address you wish to use.  This will be the IP that users connect to gain access to resources.

Next select the disk you wish to assign to this cluster.  This will be the disk that contains the shares and files.

Finish out the wizard and once it is complete you will see the service or application appear in the list as well as it’s status, the type and the current owner (aka the node holding the role at this time).

If you wish to move it to the other node in the cluster simply right-click the name and choose Move this Service or Application to Another Node and choose the node.

The resources will be re-assigned to the other node and after a brief outage be back online for users to access.

For more information see:

TechNet Virtual Lab: Creating a Highly Available Infrastructure

TechNet Virtual Lab: Windows Server 2008 Enterprise Failover Clustering Lab

Step-by-Step Guide for Configuring a Two-Node File Server Failover Cluster in Windows Server 2008

Failover Clustering Resource Center

1. 3 servers at minimum (1DC, 2 Nodes)
2. Shared storage (iSCSI or FibreChannel)
3. 3 Networks (Public, iSCSI, Heartbeat)

With that in place the first thing to do is get the cluster role installed on both nodes.  On Sever Core you can do that with the following command (case sensitive):

OCSETUP FailoverCluster-Core

We can verify that the install was successful with the following command (case sensitive):

OCLIST

With that complete launch the Failover Cluster Administrator MMC from the DC and click on Create  a Cluster under the Management section.

Add the two servers that will act as the cluster nodes…

…and perform a validation.  You don’t have to perform one but it is a best practice to ensure everything is configured properly and it is required to get support with your cluster from Microsoft.  It is also a handy troubleshooting tool to boot!

Once the validation is complete and everything checks out, give the cluster a name and set the IP address.  This is the public IP that clients will use to connect.

Confirm the settings and click Next to start the cluster creation.

Once complete you can view the report and the install summary.

That is it!  You now have a 2 node failover cluster installed!  Don’t believe it is that easy?  Check out the TechNet Virtual Labs below and do it yourself!

For more information see:

TechNet Virtual Lab: Creating a Highly Available Infrastructure

TechNet Virtual Lab: Windows Server 2008 Enterprise Failover Clustering Lab

This guide attempts to provide a Xen based test environment where you can practice setting up a two node cluster (cluster setup itself is not discussed here – I’m merely giving you what you need to set it up).

XEN can host two type of guest systems para-virtualized and fully-virtualized:

• for para-virtualized guests you require the Red Hat Enterprise Linux 5 installation tree available over NFS, FTP or HTTP.
• for fully-virtualized guest installations you will require DVD or CD-ROM distribution media or a bootable .iso file and a network accessible installation tree

For details, please refer to the RHEL5 Virtualization Manual.

I’ll be using para-virtualized guests here in my setup. There will be three systems involved here:

• node00 – physical system
  virtual IPs: 192.168.222.1 (public1 vlan)
  192.168.100.1 (private1 vlan)
• node01 – para-virtualized guest 1
  virtual IPs: 192.168.222.10 (public1 vlan)
  192.168.100.10 (private1 vlan)
• node02 – para-virtualized guest 1
  virtual IPs: 192.168.222.20 (public1 vlan)
  192.168.100.20 (private1 vlan)

B. What I used

• an HP Blade bl25p machine with 4G of RAM (this is actually an AMD64 blade machine). A machine with decent amount of RAM and processing speed should do.
• Centos i386 5 update 1 www.centos.org DVD ISO downloaded HTTP, NFS and FTP installation sources were created from this iso. Also, the yum repository that can be used by host and guest systems will be generated from the centos iso image.
• logical volumes hosting the guests and the “virtual luns” via iscsi (you can also use disk partitions – please refer to the virtualization guide for details).

1. My LVM setup

The following is my LVM configuration. The lvLUN0* entries are the ones I used for iSCSI setup and will be shared by the two virtual guest systems.

lvs

LV VG Attr LSize Origin Snap% Move Log Copy%
lvLUN01 Virtual00VG -wi-ao 50.00G
lvLUN02 Virtual00VG -wi-ao 50.00G
lvNODE01 Virtual00VG -wi-ao 30.00G
lvNODE02 Virtual00VG -wi-ao 30.00G
lvNODE03 Virtual00VG -wi-ao 15.00G
lvsys00 vg00 -wi-ao 512.00M
lvsys01 vg00 -wi-ao 8.00G
lvsys02 vg00 -wi-ao 8.00G
lvsys03 vg00 -wi-ao 512.00M
lvsys04 vg00 -wi-ao 128.00M
lvsys05 vg00 -wi-ao 1.00G
lvsys06 vg00 -wi-ao 256.00M

C. Host Preparation

I’m assuming that you know how to install CentOS or other RHEL based distributions and that you are familiar with rpm installation. Since I do a lot of setup for test/dev environments at work, I already have an installation server making it easy to do a network based install via PXE. The kickstart file for node00 is provided below. You can do a local media install (you have the ISO so you can burn it to a DVD) and just refer to the kickstart file for some of the configuration. The list of packages I used is in the %packages section of node00‘s kickstart file. You can install them manually using yum, like:
# will list centos installation groups

yum grouplist

# will install Virtualization group

yum groupinstall Virtualization

1. ks file and installation

1.a kickstart file I use for the host (node00)

You’ll have to modify the following to suit your setup.

## START node00_ks.cfg
#modify for your own settings
install
nfs –server=remote_server –dir=/path/to/CENTOS5U1/i386
lang en_US.UTF-8
keyboard us
skipx
reboot
network –device eth2 –bootproto static –ip a.b.c.1 –netmask 255.255.255.0 –gateway a.b.c.2 –nameserver x.y.z.n –hostname node00.example.com
# grub and root password is a1s2d3f4g5
rootpw –iscrypted $1$3CXK2$CG9WlX2PuPpp7nxYMQGwP0
firewall –disabled
authconfig –enableshadow
selinux –disabled
timezone Asia/Singapore
bootloader –location=mbr –driveorder=cciss/c0d0 –append=”rhgb quiet” –md5pass=$1$3CXK2$CG9WlX2PuPpp7nxYMQGwP0
clearpart –all –initlabel –drives=cciss/c0d0
part /boot –fstype ext3 –size=100 –ondisk=cciss/c0d0
part pv.100000 –size=100 –grow –ondisk=cciss/c0d0 –asprimary
volgroup vg00 –pesize=32768 pv.100000
logvol /tmp –fstype ext3 –name=lvsys05 –vgname=vg00 –size=1024
logvol /opt –fstype ext3 –name=lvsys04 –vgname=vg00 –size=128
logvol /var –fstype ext3 –name=lvsys03 –vgname=vg00 –size=512
logvol /usr –fstype ext3 –name=lvsys02 –vgname=vg00 –size=8192
logvol swap –fstype swap –name=lvsys01 –vgname=vg00 –size=8192
logvol /home –fstype ext3 –name=lvsys06 –vgname=vg00 –size=256
logvol / –fstype ext3 –name=lvsys00 –vgname=vg00 –size=512
%packages
@development-libs
@editors
@system-tools
@text-internet
@x-software-development
@virtualization
@dns-server
@core
@base
@ftp-server
@network-server
@legacy-software-development
@base-x
@web-server
@printing
@server-cfg
@sql-server
@admin-tools
@development-tools
lsscsi
createrepo
audit
net-snmp-utils
iptraf
tftp
lynx
mesa-libGLU-devel
kexec-tools
bridge-utils
device-mapper-multipath
vnc-server
xorg-x11-server-Xnest
xorg-x11-server-Xvfb
imake
openmotif
-vim-enhanced
-zisofs-tools
-zsh
-bluez-hcidump
-sysreport
## END of node00_ks.cfg

2. Host configuration

Setting up an HTTP, NFS and FTP installation server:

2.a web server

#/etc/httpd/conf.d/centos5u1.conf
Alias /centos5u1 /var/ftp/pub/centos5u1
<Location /centos5u1>
Options Indexes FollowSymLinks MultiViews
IndexOptions FancyIndexing
Order deny,allow
Deny from all
Allow from 127.0.0.1 ::1 all
<Location>

Then start the httpd service and make sure it does during startup:

service httpd start
chkconfig httpd on

2.b NFS server

Edit /etc/exports and put the following into it:

# /etc/exports
/var/ftp/pub/centos5u1 192.168*(ro)

service nfs start
chkconfig nfs on

2.c FTP server

Since we already have the source in /var/ftp/pub/centos5u1, all that is needed is to start vsftpd:

service vsftpd start
chkconfig vsftpd on

2.d YUM repository

For this setup, I only use a local yum repository from the Centos DVD ISO I downloaded. First, I loopback mount it in /var/ftp/pub/centos5u1/i386/:

cd /var/ftp/pub/centos5u1/
mkdir temp
mount -o loop CentOS-5.1-i386-bin-DVD.iso temp
cp -pr temp i386
umount temp
createrepo -g i38

(i386/repodata/ will then be updated.)6

For RHEL5, it’s different:

createrepo -g repodata/comps-rhel5-server-core.xml Server

You need to do this inside the i386 directory (after loopback mounting and copying the whole directory structure).

2.d.1 the yum repo configuration:

I renamed the default repo files in /etc/yum.repos.d/ to *-repo (instead of *.repo) to disable them. I then created this file:

#/etc/yum.repos.d/CentOS5.repo
[centos5-Server]
name=CentOS5 Server
baseurl=http://node00/centos5u1/i386
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

node00 needs to be defined in /etc/hosts for the above file to work. Or just replace node00 with its IP address.

2.e VNC server

You won’t be needing a vnc connection if you have local console access to the physical machine. All you need to do is switch into gui mode:

telinit 5

and open a gui terminal (like gnome-terminal or kde konsole). But since I do everything remotely, I use vncserver and vncviewer to do gui based stuff.

2.e.1 run vncserver:

This will bringvup a vncserver in node00 that is accessible via “vncviewer” at node00:1 (assuming node00 is resolveable from your vncviewer host).

vncserver

You will require a password to access your desktops.

Password:
Verify:
xauth: creating new authority file /root/.Xauthority

New ‘node00.example.com:1 (root)’ desktop is node00.example.com:1

Creating default startup script /root/.vnc/xstartup
Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/node00.example.com:1.log

D. Virtualization

1. virtual networks

As root, run

virt-manager

The Virtual Machine Manager window should appear. You’ll see Domain-0 and the resources it is using.

1.a to create the virtual network:

• On the menu, click on Edit and then “Host details”.
• In the Host Details window, you will only see “default” on the left frame. Below, click on “Add”.
• The “Create a new virtual network” window will appear, click forward.
• Use “public1″ (no quotes) and then hit forward.
• Network should be “192.168.222.0/24″ then hit forward.
• DHCP range: Start: 192.168.222.128 end: 192.168.222.254 then hit forward.
• This will be an “Isolated virtual network”. Hit forward.
• Summary:Network Name: public1
  IPV4 network:
  Network: 192.168.222.0/24
  Gateway: 192.168.222.1
  Netmask: 255.255.255.0
  DHCP
  Start address: 192.168.222.128
  End address : 192.168.222.254
  Forwarding:
  Connectivity: Isolated virtual network
• Hit Finish.

You’ll go back to the Host Details window and the public1 entry will appear. Now to the same steps for network private1 with the following settings:

Network Name: private1
IPV4 network:
Network: 192.168.100.0/24
Gateway: 192.168.100.1
Netmask: 255.255.255.0
DHCP
Start address: 192.168.100.128
End address : 192.168.100.254
Forwarding:
Connectivity: Isolated virtual network

When you are done, in the Host Details window, click on “File > Close” to go back to the Virtual Machine Manager Window. Then click on “File > Quit”. NOTE: Don’t leave the Virtual Manager window running if you are not going to use it. It will eat up a lot of memory. If this happens, you need to restart Xen.

Once done, vnet0 and vnet1 can be seen when you run

ifconfig

vnet0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:192.168.222.1 Bcast:192.168.222.255 Mask:255.255.255.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:7782 (7.5 KiB)

vnet1 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:192.168.100.1 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:7712 (7.5 KiB)

2. kickstart files

I’m providing the kickstart file for node01 that you can also use for node02. You just need to mofiy the IP addresses and hostname entries.

2.a for node01

#START of node01_ks.cfg
install
text
reboot
#uncomment the line you want to use
# for nfs
nfs –server=192.168.222.1 –dir=/var/ftp/pub/centos5u1
##url –url ftp://<username>:<password>@<server>/<dir>
# this will be an anonymous ftp access
#url –url ftp://192.168.222.1/pub/centos5u1/i386
#key –skip
lang en_US.UTF-8
keyboard us
skipx
# private
network –device eth1 –bootproto static –ip 192.168.100.10 –netmask 255.255.255.0
# public – disabled on initial install
network –device eth0 –bootproto static –ip 192.168.222.10 –netmask 255.255.255.0 –gateway 192.168.222.1 –nameserver 192.168.222.1 –hostname node01.example.com
## password is n0de01pass
rootpw –iscrypted $1$Lqk1Y$Y8TIWCMLiiPjVt1GjRS0F1
## password is n0de02pass
#rootpw –iscrypted $1$Rn47b$DDwgrOv3IFGf3HVhsxv9X0
firewall –disabled
authconfig –enableshadow –enablemd5
selinux –disabled
timezone –utc Asia/Singapore
services –disabled ipsec,iptables,bluetooth,hplip,firstboot,cups,sendmail,xfs
bootloader –location=mbr –driveorder=xvda,xvdb –append=”rhgb quiet”
clearpart –all –initlabel –drives=xvda
part /boot –fstype ext3 –size=100 –ondisk=xvda
part pv.2 –size=0 –grow –ondisk=xvda
volgroup VolGroup00 –pesize=32768 pv.2
logvol swap –fstype swap –name=LogVol01 –vgname=VolGroup00 –size=1000 –grow –maxsize=1984
logvol / –fstype ext3 –name=LogVol00 –vgname=VolGroup00 –size=1024 –grow
%packages
@development-libs
@system-tools
@gnome-software-development
@text-internet
@x-software-development
@dns-server
@core
@authoring-and-publishing
@base
@ftp-server
@network-server
@legacy-software-development
@java
@legacy-software-support
@smb-server
@base-x
@web-server
@printing
@server-cfg
@sql-server
@admin-tools
@development-tools
emacs
lsscsi
gnutls-utils
hwbrowser
audit
iptraf
mesa-libGLU-devel
kexec-tools
device-mapper-multipath
vnc-server
xorg-x11-utils
xorg-x11-server-Xnest
xorg-x11-server-Xvfb
imake
iscsi-initiator-utils
ypserv
-sysreport

%post
cat <<EOT >> /etc/hosts
# private or replace with nodeXY-
192.168.100.10 node01-priv
192.168.100.20 node02-priv
192.168.100.1 node00-priv

#public or replace with nodeXY
192.168.222.10 node01
192.168.222.20 node02
192.168.222.1 node00
EOT

# yum local repo
mv /etc/yum.repos.d/*.repo /tmp
cat > /etc/yum.repos.d/centos5.repo << EOF
[centos5-Server]
name=CEntos5 Server
baseurl=http://node00/centos5u1/i386
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
EOF

#change default runlevel
ed /etc/inittab << EOF

,s/id:5:initdefault:/id:3:initdefault:/g
.
w
EOF

# vncserver stuff
cat << EOT > /opt/vnc_xstartup
#!/bin/sh

# run vncserver and copy to your $HOME/.vnc/xstartup file
# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup

[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 130×30+12+12 -ls -bg black -fg green -title “$VNCDESKTOP Desktop” &
mwm &
EOT
# END of nod01_ks.cfg

2.b for node02

Copy the node01_ks.cfg file above to node02_ks.cfg and change the appropriate entries for node02 (hostname and IP addresses).

3. Installing the guest systems (node01 and node02)

For the installation, we’ll be invoking it in the CLI, using

virt-install

But first, generate the MAC addresses for the NICS of the virtual systems:

3.a MAC Address generation

We’ll use a python script provided by the Red Hat Virtualization Guide:

#!/usr/bin/python
# macgen.py script to generate a MAC address for Red Hat Virtualization guests
import random
#
def randomMAC():
mac = [ 0x00, 0x16, 0x3e,
random.randint(0x00, 0x7f),

random.randint(0x00, 0xff),
random.randint(0x00, 0xff) ]
return ‘:’.join(map(lambda x: “%02x” % x, mac))
#
print randomMAC()
# careful with the indention
# this is from the Virtualization guide from redhat.com
# http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.2/html/Virtualization/index.html
node01 NICs
# for eth0 . public1
[root@node00 ~]# ./macgen.py
00:16:3e:33:32:07
# for eth1 . private1
[root@ node00 ~]# ./macgen.py
00:16:3e:55:6b:83

Then do the same for node02‘s virtual NICs.

3.b virt-install command for node01

virt-install -p -n node01 -r 768 -f /dev/Virtual00VG/lvNODE01 -m 00:16:3e:33:32:07 \
-w network:public1 -m 00:16:3e:55:6b:83 -w network:private1 \
-l nfs:192.168.222.1:/var/ftp/pub/centos5u1/i386 \
-x "ksdevice=eth0 ks=http://192.168.222.1/centos5u1/node01_ks.cfg" --vnc

Parameters:
-n node01 = name of the guest system
-r 768 = amount of RAM in MB
-f /dev/Virtual00VG/lvNODE01 = disk to be used by the guest system (can be an unused partition in your system, like /dev/sda3 or /dev/hda9).
-m 00:16:3e:33:32:07 = mac address for eth0
-w network:public1 = eth0′s network
-m 00:16:3e:55:6b:83 = mac address for eth1
-w network:private1 = eth0′s network
-l nfs:192.168.222.1:/var/ftp/pub/centos5u1/i386 = installation source (not the actual one used during ks install
-x “ksdevice=eth0 ks=http://192.168.222.1/centos5u1/node01_ks.cfg” = kickstart directives. This will use dhcp to startup installation.
–vnc = will launch a gui window for you to view (if you are running virt-install from a vnc session or a gui terminal).

3.c virt-install command for node02

virt-install -p -n node02 -r 768 -f /dev/Virtual00VG/lvNODE02 -m 00:16:3e:1e:05:b6 \
-w network:public1 -m 00:16:3e:40:3d:b0 -w network:private1 \
-l nfs:192.168.222.1:/var/ftp/pub/centos5u1/i386 \
-x "ksdevice=eth0 ks=http://192.168.222.1/centos5u1/node02_ks.cfg" --vnc

I ran the above virt-install inside a vnc session in the physical host so the the guest installation screens automatically appear. You can start it in the physical host’s console and the installation of the guest systems will run in the background. Their installation can be viewed by running virt-manager and then opening the guests.

E. iSCSI

iSCSI is a Storage Area Network protocol allowing shared storage going through an exising network infrastructure. In my setup, I used iscsitarget from http://iscsitarget.sourceforge.net.

1. iSCSI server installation and configuration

1.a compiling the iscsi application tarball

This needs to be done on the physical host.

• Get the tarball from SourceForge and put it in /usr/local/src.
• cd to /usr/local/src:cd /usr/local/src
• Then extract the files:tar xvf iscsitarget-0.4.16.tar.gz
  cd iscsitarget-0.4.16
• Then run:make
  make install

1.b configuration needed

This is my ietd.conf configuration defining the “LUNs” to be allocated to the guests from the physical host’s disks:

#/etc/ietd.conf
# NOTE: the config files has more entries than what i’m showing here.
# but i’ve commented out the original entries and made the following
Target iqn.2008-07.NODE00:LUN01.NODE00
MaxConnections 2
Lun 1 Path=/dev/Virtual00VG/lvLUN01,Type=fileio
Alias LUN01
Target iqn.2008-07.NODE00:LUN02.NODE00
MaxConnections 2
Lun 2 Path=/dev/Virtual00VG/lvLUN02,Type=fileio
Alias LUN02
# end of ietd.conf

In my physical host system, I have created two logical volumes 50G each in size. You can also use files or disk partitions, just change the Path entries in the ietd.conf file.

1.c ACL

iscsitarget has /etc/initiators.allow and /etc/initiators.deny that work like hosts.allow and hosts.deny. In my setup, I will allow node01 and node02 to access the two LUNs defined in ietd.conf.

#/etc/initiators.allow
#this should correspond to the definition in your /etc/ietd.conf
iqn.2008-07.NODE00:LUN01.NODE00 192.168.100.10, 192.168.100.20
iqn.2008-07.NODE00:LUN02.NODE00 192.168.100.10, 192.168.100.20
# endof initiators.allow

• Start the iscsi-target service:service iscsi-target start
• and make sure it starts during bootup:
  chkconfig --add iscsi-target
chkconfig iscsi-target on
chkconfig --list iscsi-target
  iscsi-target 0:off 1:off 2:on 3:on 4:on 5:on 6:off

2 Client Side

The package iscsi-initiator-utils-6.2.0.865-0.8.el5 should already be installed (as it is included in the kickstart file above).

2.a configuration

• Edit the file /etc/iscsi/initiatorname.iscsi to define the targets.
• My /etc/iscsi/initiatorname.iscsi is as follows:
  

  #/etc/iscsi/initiatorname.iscsi
  InitiatorName=iqn.2008-07.NODE00:LUN01.NODE00
  InitiatorName=iqn.2008-07.NODE00:LUN02.NODE00
  # end of #/etc/iscsi/initiatorname.iscsi

• Run iscsid service and try to discover the LUNs:service iscsid start

  Turning off network shutdown. Starting iSCSI daemon: [ OK ]

  iscsiadm -m discovery -t st -p node00

  192.168.222.1:3260,1 iqn.2008-07.NODE00:LUN01.NODE00
  192.168.222.1:3260,1 iqn.2008-07.NODE00:LUN02.NODE00

• Then start the iscsi service. You’ll then see the LUN definitions created earlier:service iscsi start

  will then show the following:

  iscsid (pid 964 963) is running…
  Setting up iSCSI targets: Login session [iface: default, target: \

  iqn.2008-07.NODE00:LUN02.NODE00, portal: 192.168.222.1,3260]
  Login session [iface: default, target: iqn.2008-07.NODE00:LUN01.\
  NODE00, portal: 192.168.222.1,3260] [ OK ]

• Check system logs to see if the disks have been seen:dmesg
  

  scsi0 : iSCSI Initiator over TCP/IP
  Vendor: IET Model: VIRTUAL-DISK Rev: 0
  Type: Direct-Access ANSI SCSI revision: 04
  scsi 0:0:0:2: Attached scsi generic sg0 type 0
  SCSI device sda: 104857600 512-byte hdwr sectors (53687 MB)
  sda: Write Protect is off
  sda: Mode Sense: 77 00 00 08
  SCSI device sda: drive cache: write through
  SCSI device sda: 104857600 512-byte hdwr sectors (53687 MB)

  sda: Write Protect is off
  sda: Mode Sense: 77 00 00 08
  SCSI device sda: drive cache: write through
  sda: unknown partition table
  sd 0:0:0:2: Attached scsi disk sda
  scsi1 : iSCSI Initiator over TCP/IP
  Vendor: IET Model: VIRTUAL-DISK Rev: 0
  Type: Direct-Access ANSI SCSI revision: 04
  SCSI device sdb: 104857600 512-byte hdwr sectors (53687 MB)

  sdb: Write Protect is off
  sdb: Mode Sense: 77 00 00 08
  SCSI device sdb: drive cache: write through
  SCSI device sdb: 104857600 512-byte hdwr sectors (53687 MB)
  sdb: Write Protect is off
  sdb: Mode Sense: 77 00 00 08
  SCSI device sdb: drive cache: write through
  sdb: unknown partition table
  sd 1:0:0:1: Attached scsi disk sdb

  sd 1:0:0:1: Attached scsi generic sg1 type 0

  I now have sda and sdb, each with 53687 MB in size (results for your setup may be different.

• Running fdisk:fdisk -l
  

  Disk /dev/xvda: 32.2 GB, 32212254720 bytes
  255 heads, 63 sectors/track, 3916 cylinders
  Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot Start End Blocks Id System
  /dev/xvda1 * 1 13 104391 83 Linux
  /dev/xvda2 14 3916 31350847+ 8e Linux LVM

  Disk /dev/sda: 53.6 GB, 53687091200 bytes
  64 heads, 32 sectors/track, 51200 cylinders
  Units = cylinders of 2048 * 512 = 1048576 bytes

  Disk /dev/sda doesn’t contain a valid partition table

  Disk /dev/sdb: 53.6 GB, 53687091200 bytes
  64 heads, 32 sectors/track, 51200 cylinders
  Units = cylinders of 2048 * 512 = 1048576 bytes

  Disk /dev/sdb doesn’t contain a valid partition table

Now do the same for node02. Once the disks are seen by both guests, you can then start setting up a two-node cluster. I’ve used this configuration to test a two-node Oracle 10gR2 RAC setup with shared ASM storage and OCFS2 on a 64-bit system.

F. Conclusion

This kind of setup will help you to learn the basics of clustering without the need of acquiring additional hardware. In no way can this setup be used in a “live” environment. Once you have familiarized yourself with the concept of how a cluster is prepared, you can apply the concept when building real, physical setups that you need for your organization. I hope you’ll find this useful.

G. Further Readings

• RHEL5 Virtualization Manual
• http://iscsitarget.sourceforge.net
• Oracle RAC HowTO

Pre-Configuration Requirements

1. Assign hostname node01 to primary node with IP address 172.16.4.80 to eth0.
2. Assign hostname node02 to slave node with IP address 172.16.4.81.

Note: on node01

uname -n

must return node01.

On node02

uname -n

must return node02.

172.16.4.82 is the virtual IP address that will be used for our Apache webserver (i.e., Apache will listen on that address).

Configuration

1. Download and install the heartbeat package. In our case we are using CentOS so we will install heartbeat with yum:

yum install heartbeat

or download these packages:

heartbeat-2.08
heartbeat-pils-2.08
heartbeat-stonith-2.08

2. Now we have to configure heartbeat on our two node cluster. We will deal with three files. These are:

authkeys
ha.cf
haresources

3. Now moving to our configuration. But there is one more thing to do, that is to copy these files to the /etc/ha.d directory. In our case we copy these files as given below:

cp /usr/share/doc/heartbeat-2.1.2/authkeys /etc/ha.d/
cp /usr/share/doc/heartbeat-2.1.2/ha.cf /etc/ha.d/
cp /usr/share/doc/heartbeat-2.1.2/haresources /etc/ha.d/

4. Now let’s start configuring heartbeat. First we will deal with the authkeys file, we will use authentication method 2 (sha1). For this we will make changes in the authkeys file as below.

vi /etc/ha.d/authkeys

Then add the following lines:

auth 2
2 sha1 test-ha

Change the permission of the authkeys file:

chmod 600 /etc/ha.d/authkeys

5. Moving to our second file (ha.cf) which is the most important. So edit the ha.cf file with vi:

vi /etc/ha.d/ha.cf

Add the following lines in the ha.cf file:

logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
initdead 120
bcast eth0
udpport 694
auto_failback on
node node01
node node02

Note: node01 and node02 is the output generated by

uname -n

6. The final piece of work in our configuration is to edit the haresources file. This file contains the information about resources which we want to highly enable. In our case we want the webserver (httpd) highly available:

vi /etc/ha.d/haresources

Add the following line:

node01 172.16.4.82 httpd

7. Copy the /etc/ha.d/ directory from node01 to node02:

scp -r /etc/ha.d/ root@node02:/etc/

8. As we want httpd highly enabled let’s start configuring httpd:

vi /etc/httpd/conf/httpd.conf

Add this line in httpd.conf:

Listen 172.16.4.82:80

9. Copy the /etc/httpd/conf/httpd.conf file to node02:

scp /etc/httpd/conf/httpd.conf root@node02:/etc/httpd/conf/

10. Create the file index.html on both nodes (node01 & node02):

On node01:

echo "node01 apache test server" > /var/www/html/index.html

On node02:

echo "node02 apache test server" > /var/www/html/index.html

11. Now start heartbeat on the primary node01 and slave node02:

/etc/init.d/heartbeat start

12. Open web-browser and type in the URL:

http://172.16.4.82

It will show node01 apache test server.

13. Now stop the hearbeat daemon on node01:

/etc/init.d/heartbeat stop

In your browser type in the URL http://172.16.4.82 and press enter.

It will show node02 apache test server.

14. We don’t need to create a virtual network interface and assign an IP address (172.16.4.82) to it. Heartbeat will do this for you, and start the service (httpd) itself. So don’t worry about this.

Don’t use the IP addresses 172.16.4.80 and 172.16.4.81 for services. These addresses are used by heartbeat for communication between node01 and node02. When any of them will be used for services/resources, it will disturb hearbeat and will not work. Be carefull!!!

In this document I use Debian Etch 4.0 for all nodes. Therefore the setup might differ a bit for other distributions. The two data nodes were x64 to use all of the 8GB RAM. Servers were compiled from source so you should be able to make it running on any platform. The MySQL version I use in this setup is 5.1.24-rc. It’s a release candidate, but I wanted to use 5.1 to take advantage of Memory-Disk Based tables.

Beginning with MySQL 5.1.6, it is possible to store the non-indexed columns of NDB tables on disk, rather than in RAM as with previous versions of MySQL Cluster.[More here]

This howto is just a scratch to make it running, for many of you I am suggesting to read some off docs from MySQL page to be prepared to how manage the whole cluster and always know whats going on.

This document comes without warranty of any kind! Bare in mind you need to make tests and prepare your databases before using it in production mode.

1 My Servers

I will use the following Debian servers that are all in the same network (10.0.1.x in this example):

• mysql-mngt.example.com:10.0.1.30 [ MySQL cluster management server ]
• lb1.example.com: 10.0.1.31 [ Load Balancer 1 ]
• lb2.example.com: 10.0.1.32 [ Load Balancer 2 ]
• mysql-data1.example.com: 10.0.1.33 [ MySQL cluster node 1 ]
• mysql-data2.example.com: 10.0.1.34 [ MySQL cluster node 2 ]

In addition to that we need a virtual IP address : 10.0.1.10. It will be assigned to the MySQL cluster by the load balancer so that applications have a single IP address to access the cluster.

Although we want to have two MySQL cluster nodes in our MySQL cluster, we still need a third node, the MySQL cluster management server, for mainly one reason: if one of the two MySQL cluster nodes fails, and the management server is not running, then the data on the two cluster nodes will become inconsistent (“split brain”). We also need it for configuring the MySQL cluster.

2 MySQL data nodes + 1 cluster management server + 2 Load Balancers = 5

Here is my hardware configuration:

MySQL Data : DELL R300 Intel(R) Quad Core Xeon(R) CPU X3353 @ 2.66GHz, 2x SAS 146GB Drives (Raid 1), 8GB RAM

MySQL LoadBalancer : DELL R200 Intel(R) Xeon(R) CPU 3065 @ 2.33GHz, 2x SATA 250 GB Drives (Raid 1), 1GB RAM

MySQL Management : DELL R200 Intel(R) Celeron(R) CPU 430 @ 1.80GHz, 1x SATA 160 GB Drives (Raid 1), 1GB RAM

As the MySQL cluster management server does not use many resources, you can put additional load balancer on this machine or you can use it for monitoring the whole cluster by Nagios or Cacti.

2 Set Up The MySQL Cluster Management Server

First we have to download MySQL 5.1.24 (the sources version) and install the cluster management server (ndb_mgmd) and the cluster management client (ndb_mgm – it can be used to monitor what’s going on in the cluster). The following steps are carried out on mysql-mngt.example.com (10.0.1.30):

mysql-mngt.example.com:

cd /usr/src
wget http://mysql.mirrors.pair.com/Downloads/MySQL-5.1/mysql-5.1.24-rc.tar.gz
tar xvzf mysql-5.1.24-rc.tar.gz
#Lets add proper user and group
groupadd mysql
useradd -g mysql mysql
./configure --prefix=/usr/local/mysql --enable-community-features \
--with-mysqld-user=mysql --with-mysqlmanager --with-plugins=ndbcluster
make
make install

This way we are wasting about 124MB space since we do not need all the actual MySQL files, but believe me it’s way easier to make any cleanup/upgrade in just one dir /usr/local/mysql instead of searching for all files in /usr/bin and so on. After compiling we have two directories we are interested in /usr/local/mysql/bin and /usr/local/mysql/libexec [last one contains the ndb management exec].

Just to make life easier let’s add this below to your PATH environment, to do so we have to edit file /root/.bash_profile:

mysql-mngt.example.com:

echo "PATH=$PATH:/usr/local/mysql/bin:/usr/local/mysql/libexec" >>/root/.bash_profile
echo "export PATH" >>/root/.bash_profile

Next, we must create the cluster configuration file, /usr/local/mysql/var/mysql-cluster/config.ini: [Hints here]

mysql-mngt.example.com:

mkdir /usr/local/mysql/var/mysql-cluster
cd /usr/local/mysql/var/mysql-cluster
vi config.ini

[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=80M # How much memory to allocate for data storage
IndexMemory=18M # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the “world” database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[TCP DEFAULT]
# Section for the cluster management node
[NDB_MGMD]
# IP address of the management node (this system)
HostName=10.0.1.30
# Section for the storage nodes
[NDBD]
# IP address of the first storage node
HostName=10.0.1.33
DataDir=/usr/local/mysql/var/mysql-cluster
BackupDataDir=/usr/local/mysql/var/mysql-cluster/backup
DataMemory=2048M
[NDBD]
# IP address of the second storage node
HostName=10.0.1.34
DataDir=/usr/local/mysql/var/mysql-cluster
BackupDataDir=/usr/local/mysql/var/mysql-cluster/backup
DataMemory=2048M

# one [MYSQLD] per storage node
[MYSQLD]
[MYSQLD]

Please replace the IP addresses in the file appropriately to your setup.

Then we proceed to DataSQL nodes to make necessary dirs and files setup.

mysql-mngt.example.com:

ndb_mgmd -f /usr/local/mysql/var/mysql-cluster/config.ini

It makes sense to automatically start the management server at system boot time, so we create a very simple init script and the appropriate startup links:

mysql-mngt.example.com:

echo '/usr/local/mysql/libexec/ndb_mgmd -f /usr/local/mysql/var/mysql-cluster/config.ini' > /etc/init.d/ndb_mgmd
chmod 755 /etc/init.d/ndb_mgmd
update-rc.d ndb_mgmd defaults

3 Set Up The MySQL Cluster Nodes (Storage Nodes)

Now we install mysql-5.1.14-rc on both mysql-data1.example.com and mysql-data2.example.com:

mysql-data1.example.com / mysql-data2.example.com:

cd /usr/src
wget http://mysql.mirrors.pair.com/Downloads/MySQL-5.1/mysql-5.1.24-rc.tar.gz
tar xvzf mysql-5.1.24-rc.tar.gz
groupadd mysql
useradd -g mysql mysql
cd /usr/src/mysql-5.1.24-rc
./configure --prefix=/usr/local/mysql --enable-community-features --with-mysqld-user=mysql --with-plugins=ndbcluster
make
make install
/usr/src/mysql-5.1.24-rc/scripts/mysql_install_db --user=mysql
cd /usr/local/mysql
chown -R root:mysql .
chown -R mysql.mysql /usr/local/mysql/var

cd /usr/src/mysql-5.1.24-rc
cp support-files/mysql.server /etc/init.d/
chmod 755 /etc/init.d/mysql.server
cd /etc/init.d
update-rc.d mysql.server defaults

Then we create the MySQL configuration file /etc/my.cnf on both nodes:

mysql-data1.example.com / mysql-data2.example.com:

vi /etc/my.cnf

[client]
port=3306
socket=/tmp/mysql.sock
[mysqld]
ndbcluster
# IP address of the cluster management node
ndb-connectstring=10.0.1.30
default-storage-engine=NDBCLUSTER

#Those are for future tuning
#max_connections=341
#query_cache_size=16M
#thread_concurrency = 4
[mysql_cluster]
# IP address of the cluster management node
ndb-connectstring=10.0.1.30

Make sure to fill in the correct IP address of the MySQL cluster management server.

Lets add PATH env. to data nodes also:

mysql-data1.example.com / mysql-data2.example.com:

echo "PATH=$PATH:/usr/local/mysql/bin:/usr/local/mysql/libexec" >>/root/.bash_profile
echo "export PATH" >>/root/.bash_profile

Next we create the data and backup directories and start the MySQL server on both cluster nodes:

mysql-data1.example.com / mysql-data2.example.com:

mkdir /usr/local/mysql/var/mysql-cluster
mkdir /usr/local/mysql/var/mysql-cluster/backup
cd /var/lib/mysql-cluster
ndbd --initial
/etc/init.d/mysql.server start

(Please note: we have to run ndbd –initial only when we start MySQL for the first time, and if /usr/local/mysql/mysql-cluster/config.ini on mysql-mngt.example.com changes.)

Now is a good time to set a password for the MySQL root user:

mysql-data1.example.com / mysql-data2.example.com:

mysqladmin -u root password yourrootsqlpassword

We want to start the cluster nodes at boot time, so we create an ndbd init script and the appropriate system startup links:

mysql-data1.example.com / mysql-data2.example.com:

echo '/usr/local/mysql/libexec/ndbd' > /etc/init.d/ndbd
chmod 755 /etc/init.d/ndbd
update-rc.d ndbd defaults

4 Test The MySQL Cluster

Our MySQL cluster configuration is already finished, now it’s time to test it. On the cluster management server (loadb1.example.com), run the cluster management client ndb_mgm to check if the cluster nodes are connected:

mysql-mngt.example.com:

ndb_mgm

You should see this:

– NDB Cluster — Management Client –
ndb_mgm>

Now type show; at the command prompt:

show;

The output should be like this:

ndb_mgm> show;
Connected to Management Server at: localhost:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=2 @10.0.1.33 (Version: 5.1.24, Nodegroup: 0, Master)
id=3 @10.0.1.34 (Version: 5.1.24, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @10.0.1.30 (Version: 5.1.24)

[mysqld(API)] 2 node(s)
id=4 @10.0.1.34 (Version: 5.1.24)
id=5 @10.0.1.33 (Version: 5.1.24)

ndb_mgm>

If you see that your nodes are connected, then everything’s ok!

Type

quit;

to leave the ndb_mgm client console.

Now we have to create a test database with a test table and some data on mysql-data1.example.com:

mysql-data1.example.com:

mysql -u root -p
CREATE DATABASE mysqlclustertest;
USE mysqlclustertest;
CREATE TABLE testtable (I INT) ENGINE=NDBCLUSTER;
INSERT INTO testtable () VALUES (1);
SELECT * FROM testtable;
quit;

(Have a look at the CREATE statement: We must use ENGINE=NDBCLUSTER for all database tables that we want to get clustered! If you use another engine, then clustering will not work!)

The result of the SELECT statement should be:

mysql> SELECT * FROM testtable;
+——+
| I |
+——+
| 1 |
+——+
1 row in set (0.03 sec)

Now we have to create the same database on sql2.example.com (yes, we still have to create it, but afterwards testtable and its data should be replicated to mysql-data2.example.com because testtable uses ENGINE=NDBCLUSTER):

mysql-data2.example.com:

mysql -u root -p
CREATE DATABASE mysqlclustertest;
USE mysqlclustertest;
SELECT * FROM testtable;

The SELECT statement should deliver you the same result as before on mysql-data1.example.com: [The CREATE statement should fail due to NDBCLUSTER Engine]

mysql> SELECT * FROM testtable;
+——+
| I |
+——+
| 1 |
+——+
1 row in set (0.04 sec)

So the data was replicated from mysql-data1.example.com to mysql-data2.example.com. Now we insert another row into testtable:

mysql-data2.example.com:

INSERT INTO testtable () VALUES (2);
quit;

Now let’s go back to mysql-data1.example.com and check if we see the new row there:

mysql-data1.example.com:

mysql -u root -p
USE mysqlclustertest;
SELECT * FROM testtable;
quit;

You should see something like this:

mysql> SELECT * FROM testtable;
+——+
| I |
+——+
| 1 |
| 2 |
+——+
2 rows in set (0.05 sec)

So both MySQL cluster nodes always have the same data!

Now let’s see what happens if we stop node 1 (mysql-data1.example.com): Run

mysql-data1.example.com:

killall ndbd

and check with

ps aux | grep ndbd | grep -iv grep

that all ndbd processes have terminated. If you still see ndbd processes, run another

killall ndbd

until all ndbd processes are gone.

Now let’s check the cluster status on our management server (mysql-mngt.example.com):

mysql-mngt.example.com:

ndb_mgm

On the ndb_mgm console, issue

show;

and you should see this:

ndb_mgm> show;
Connected to Management Server at: localhost:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=2 (not connected, accepting connect from 10.0.1.33)
id=3 @10.0.1.34 (Version: 5.1.24, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @10.0.1.30 (Version: 5.1.24)

[mysqld(API)] 2 node(s)
id=4 @10.0.1.34 (Version: 5.1.24)
id=5 @10.0.1.33 (Version: 5.1.24)

ndb_mgm>

You see, mysql-data1.example.com is not connected anymore.

Type

quit;

to leave the ndb_mgm console.

Let’s check mysql-data2.example.com:

mysql-data2.example.com:

mysql -u root -p
USE mysqlclustertest;
SELECT * FROM testtable;
quit;

The result of the SELECT query should still be

mysql> SELECT * FROM testtable;
+——+
| I |
+——+
| 1 |
| 2 |
+——+
2 rows in set (0.17 sec)

Ok, all tests went fine, so let’s start our mysql-data1.example.com node again:

mysql-data1.example.com:

ndbd

5 How To Restart The Cluster

Now let’s asume you want to restart the MySQL cluster, for example because you have changed /usr/local/mysql/var/mysql-cluster/config.ini on mysql-mngt.example.com or for some other reason. To do this, you use the ndb_mgm cluster management client on mysql-mngt.example.com:

mysql-mngt.example.com:

ndb_mgm

On the ndb_mgm console, you type

shutdown;

You will then see something like this:

ndb_mgm> shutdown;
Node 3: Cluster shutdown initiated
Node 2: Node shutdown completed.
2 NDB Cluster node(s) have shutdown.
NDB Cluster management server shutdown.
ndb_mgm>

This means that the cluster data nodes mysql-data1.example.com and mysql-data2.example.com and also the cluster management server have shut down.

Run

quit;

to leave the ndb_mgm console.

To start the cluster management server, do this on mysql-mngt.example.com:

mysql-mngt.example.com:

ndb_mgmd -f /usr/local/mysql/mysql-cluster/config.ini

and on mysql-data1.example.com and mysql-data2.example.com you can run

mysql-data1.example.com / mysql-data2.example.com:

ndbd

or, if you have changed /usr/local/mysql/var/mysql-cluster/config.ini on mysql-mngt.example.com:

ndbd --initial

Afterwards, you can check on mysql-mngt.example.com if the cluster has restarted:

mysql-mngt.example.com:

ndb_mgm

On the ndb_mgm console, type

show;

to see the current status of the cluster. It might take a few seconds after a restart until all nodes are reported as connected.

Type

quit;

to leave the ndb_mgm console.

6 Configure The Load Balancers

Our MySQL cluster is finished now, and you could start using it now. However, we don’t have a single IP address that we can use to access the cluster, which means you must configure your applications in a way that a part of it uses the MySQL cluster node 1 (mysql-data1.example.com), and the rest uses the other node (mysql-data2.example.com). Of course, all your applications could just use one node, but what’s the point then in having a cluster if you do not split up the load between the cluster nodes? Another problem is, what happens if one of the cluster nodes fails? Then the applications that use this cluster node cannot work anymore.

The solution is to have a load balancer in front of the MySQL cluster which (as its name suggests) balances the load between the MySQL cluster nodes. The load blanacer configures a virtual IP address that is shared between the cluster nodes, and all your applications use this virtual IP address to access the cluster. If one of the nodes fails, then your applications will still work, because the load balancer redirects the requests to the working node.

Now in this scenario the load balancer becomes the bottleneck. What happens if the load balancer fails? Therefore we will configure two load balancers (mysql-lb1.example.com and mysql-b2.example.com) in an active/passive setup, which means we have one active load balancer, and the other one is a hot-standby and becomes active if the active one fails. Both load balancers use heartbeat to check if the other load balancer is still alive, and both load balancers also use ldirectord, the actual load balancer the splits up the load onto the cluster nodes. heartbeat and ldirectord are provided by the Ultra Monkey package that we will install.

It is important that mysql-lb1.example.com and mysql-lb2.example.com have support for IPVS (IP Virtual Server) in their kernels. IPVS implements transport-layer load balancing inside the Linux kernel.

6.1 Install Ultra Monkey

Ok, let’s start: first we enable IPVS on mysql-lb1.example.com and mysql-lb2.example.com:

mysql-lb1.example.com / mysql-lb2.example.com:

modprobe ip_vs_dh
modprobe ip_vs_ftp
modprobe ip_vs
modprobe ip_vs_lblc
modprobe ip_vs_lblcr
modprobe ip_vs_lc
modprobe ip_vs_nq
modprobe ip_vs_rr
modprobe ip_vs_sed
modprobe ip_vs_sh
modprobe ip_vs_wlc
modprobe ip_vs_wrr

In order to load the IPVS kernel modules at boot time, we list the modules in /etc/modules:

mysql-lb1.example.com / mysql-lb2.example.com:

vi /etc/modules

ip_vs_dh
ip_vs_ftp
ip_vs
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_nq
ip_vs_rr
ip_vs_sed
ip_vs_sh
ip_vs_wlc
ip_vs_wrr

Now we edit /etc/apt/sources.list and add the Ultra Monkey repositories (don’t remove the other repositories), and then we install Ultra Monkey:

mysql-lb1.example.com / mysql-lb2.example.com:

vi /etc/apt/sources.list

deb http://www.ultramonkey.org/download/3/ sarge main
deb-src http://www.ultramonkey.org/download/3 sarge main

apt-get update
apt-get install ultramonkey libdbi-perl libdbd-mysql-perl libmysqlclient14-dev

Now Ultra Monkey is being installed. If you see this warning:

¦ libsensors3 not functional ¦
¦ ¦
¦ It appears that your kernel is not compiled with sensors support. As a ¦
¦ result, libsensors3 will not be functional on your system. ¦
¦ ¦
¦ If you want to enable it, have a look at “I2C Hardware Sensors Chip ¦
¦ support” in your kernel configuration. ¦

you can ignore it.

Answer the following questions:

Do you want to automatically load IPVS rules on boot?
<– No

Select a daemon method.
<– none

The libdbd-mysql-perl package we’ve just installed does not work with MySQL 5 (we use MySQL 5 on our MySQL cluster…), so we install the newest DBD::mysql Perl package:

mysql-lb1.example.com / mysql-lb2.example.com:

cd /tmp
wget http://search.cpan.org/CPAN/authors/id/C/CA/CAPTTOFU/DBD-mysql-3.0002.tar.gz
tar xvfz DBD-mysql-3.0002.tar.gz
cd DBD-mysql-3.0002
perl Makefile.PL
make
make install

We must enable packet forwarding:

mysql-lb1.example.com / mysql-lb2.example.com:

vi /etc/sysctl.conf

# Enables packet forwarding
net.ipv4.ip_forward = 1

sysctl -p

6.2 Configure heartbeat

Next we configure heartbeat by creating three files (all three files must be identical on mysql-lb1.example.com and mysql-lb2.example.com):

mysql-lb1.example.com / mysql-lb2.example.com:

vi /etc/ha.d/ha.cf

logfacility local0
bcast eth0
mcast eth0 225.0.0.1 694 1 0
auto_failback off
node mysql-lb1
node mysql-lb2
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster

Please note: you must list the node names (in this case mysql-lb1 and lmysql-lb2) as shown by

uname -n

IP addresses does not work here, it is also good idea to add proper entries on both load balancers(mysql-lb1 and lmysql-lb2) in

vi /etc/hosts

127.0.0.1 localhost
10.0.1.31 mysql-lb1.example.com mysql-lb1
10.0.1.32 mysql-lb2.example.com mysql-lb2

Now let’s change here your Virtual MySQL IP:

vi /etc/ha.d/haresources

loadb1 \
ldirectord::ldirectord.cf \
LVSSyncDaemonSwap::master \
IPaddr2::10.0.1.10/24/eth0/10.0.1.255

You must list one of the load balancer node names (here: mysql-lb1) and list the virtual IP address (10.0.1.10) together with the correct netmask (24) and broadcast address (10.0.1.255). If you are unsure about the correct settings, http://www.subnetmask.info/ might help you.

vi /etc/ha.d/authkeys

auth 3
3 md5 somerandomstring

somerandomstring is a password which the two heartbeat daemons on loadb1 and loadb2 use to authenticate against each other. Use your own string here. You have the choice between three authentication mechanisms. I use md5 as it is the most secure one.

/etc/ha.d/authkeys should be readable by root only, therefore we do this:

mysql-lb1.example.com / mysql-lb2.example.com:

chmod 600 /etc/ha.d/authkeys

6.3 Configure ldirectord

Now we create the configuration file for ldirectord, the load balancer:

mysql-lb1.example.com / mysql-lb2.example.com:

vi /etc/ha.d/ldirectord.cf

# Global Directives
checktimeout=10
checkinterval=2
autoreload=no
logfile=”local0″
quiescent=yes
virtual = 10.0.1.10:3306
service = mysql
real = 10.0.1.33:3306 gate
real = 10.0.1.34:3306 gate
checktype = negotiate
login = “ldirector”
passwd = “ldirectorpassword”
database = “ldirectordb”
request = “SELECT * FROM connectioncheck”
scheduler = wrr

Please fill in the correct virtual IP address (10.0.1.10) and the correct IP addresses of your MySQL cluster nodes (10.0.1.33 and 10.0.1.34). 3306 is the port that MySQL runs on by default. We also specify a MySQL user (ldirector) and password (ldirectorpassword), a database (ldirectordb) and an SQL query. ldirectord uses this information to make test requests to the MySQL cluster nodes to check if they are still available. We are going to create the ldirector database with the ldirector user in the next step.

Now we create the necessary system startup links for heartbeat and remove those of ldirectord (bacause ldirectord will be started by heartbeat):

mysql-lb1.example.com / mysql-lb2.example.com:

update-rc.d -f heartbeat remove
update-rc.d heartbeat start 75 2 3 4 5 . stop 05 0 1 6 .
update-rc.d -f ldirectord remove

6.4 Create A Database Called ldirector

Next we create the ldirector database on our MySQL cluster nodes mysql-data1.example.com and mysql-data2.example.com. This database will be used by our load balancers to check the availability of the MySQL cluster nodes.

mysql-data1.example.com:

mysql -u root -p
GRANT ALL ON ldirectordb.* TO 'ldirector'@'%' IDENTIFIED BY 'ldirectorpassword';
FLUSH PRIVILEGES;
CREATE DATABASE ldirectordb;
USE ldirectordb;
CREATE TABLE connectioncheck (I INT) ENGINE=NDBCLUSTER;
INSERT INTO connectioncheck () VALUES (1);
quit;

mysql-data2.example.com:

mysql -u root -p
GRANT ALL ON ldirectordb.* TO 'ldirector'@'%' IDENTIFIED BY 'ldirectorpassword';
FLUSH PRIVILEGES;
CREATE DATABASE ldirectordb;
quit;

6.5 Prepare The MySQL Cluster Nodes For Load Balancing

Finally we must configure our MySQL cluster nodes mysql-data1.example.com and mysql-data2.example.com to accept requests on the virtual IP address 192.168.0.105.

mysql-data1.example.com / mysql-data2.example.com:

apt-get install iproute

Add the following to /etc/sysctl.conf:

mysql-data1.example.com / mysql-data2.example.com:

vi /etc/sysctl.conf

# Enable configuration of arp_ignore option
net.ipv4.conf.all.arp_ignore = 1

# When an arp request is received on eth0, only respond if that address is
# configured on eth0. In particular, do not respond if the address is
# configured on lo
net.ipv4.conf.eth0.arp_ignore = 1

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_ignore = 1

# Enable configuration of arp_announce option
net.ipv4.conf.all.arp_announce = 2

# When making an ARP request sent through eth0 Always use an address that
# is configured on eth0 as the source address of the ARP request. If this
# is not set, and packets are being sent out eth0 for an address that is on
# lo, and an arp request is required, then the address on lo will be used.
# As the source IP address of arp requests is entered into the ARP cache on
# the destination, it has the effect of announcing this address. This is
# not desirable in this case as adresses on lo on the real-servers should
# be announced only by the linux-director.
net.ipv4.conf.eth0.arp_announce = 2

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_announce = 2

sysctl -p

Add this section for the virtual IP address to /etc/network/interfaces:

mysql-data1.example.com / mysql-data2.example.com:

vi /etc/network/interfaces

auto lo:0
iface lo:0 inet static
address 10.0.1.10
netmask 255.255.255.255
pre-up sysctl -p > /dev/null

ifup lo:0

7 Start The Load Balancer And Do Some Testing

Now we can start our two load balancers for the first time:

mysql-lb1.example.com / mysql-lb2.example.com:

/etc/init.d/ldirectord stop
/etc/init.d/heartbeat start

If you don’t see errors, you should now reboot both load balancers.If you do see errors go to the end of this tutorial I might know whats the problem [Chapter 8]

mysql-lb1.example.com / mysql-lb2.example.com:

shutdown -r now

After the reboot we can check if both load balancers work as expected :

mysql-lb1.example.com / mysql-lb2.example.com:

ip addr sh eth0

The active load balancer should list the virtual IP address (10.0.1.10):

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:16:3e:45:fc:f8 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.31/24 brd 192.168.0.255 scope global eth0
inet 10.0.1.10/24 brd 192.168.0.255 scope global secondary eth0

The hot-standby should show this:

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:16:3e:16:c1:4e brd ff:ff:ff:ff:ff:ff
inet 10.0.1.32/24 brd 192.168.0.255 scope global eth0

mysql-lb1.example.com / mysql-lb2.example.com:

ldirectord ldirectord.cf status

Output on the active load balancer:

ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 1603

Output on the hot-standby:

ldirectord is stopped for /etc/ha.d/ldirectord.cf

mysql-lb1.example.com / mysql-lb2.example.com:

ipvsadm -L -n

Output on the active load balancer:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.1.10:3306 wrr
-> 10.0.1.33:3306 Route 1 0 0
-> 10.0.1.34:3306 Route 1 0 0

Output on the hot-standby:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn

mysql-lb1.example.com / mysql-lb2.example.com:

/etc/ha.d/resource.d/LVSSyncDaemonSwap master status

Output on the active load balancer:

master running
(ipvs_syncmaster pid: 1766)

Output on the hot-standby:

master stopped
(ipvs_syncbackup pid: 1440)

If your tests went fine, you can now try to access the MySQL database from a totally different server in the same network (10.0.1.x) using the virtual IP address 10.0.1.10:

mysql -h 10.0.1.10 -u ldirector -p

(Please note: your MySQL client must at least be of version 4.1; older versions do not work with MySQL 5.)

You can now switch off one of the MySQL cluster nodes for test purposes; you should then still be able to connect to the MySQL database.

8 Annotations and hints

Errors with ldirectord:

/etc/init.d/ldirectord stop
/etc/init.d/heartbeat start
Stopping ldirectord Error [] reading file /etc/ha.d/ldirectord.cf at line 8: >Unknown command service=mysql

This error is due to formatting, it took me some time to figure it out.
The serivce= mysql is probably connected to /etc/services which is correct.
If you have libmysqlclient15 or 14 installed, fixing right formatting will do the trick like here:

vi /etc/heartbeat/ldirectord.cf

# Global Directives
checktimeout=10
checkinterval=2
autoreload=no
logfile=”local0″
quiescent=yes
#after virtual=… make TABS (or double spaces) in every line
virtual = 10.0.1.10:3306
service = mysql
real = 10.0.1.33:3306 gate
real = 10.0.1.34:3306 gate
checktype = negotiate
login = “ldirector”
passwd = “ldirector”
database = “ldirectordb”
request = “SELECT * FROM connectioncheck”
scheduler = wrr

And that should be it.

There are some important things to keep in mind when running a MySQL cluster:

1.If you have your databases and you want to use them in MySQL cluster please read this as it will guide you through how to change ENGINE=MyISAM to NDBCLUSTER.

2. Adding user to mysql database and changing GRANTS must be done on all data nodes since mysql database is MyISAM/InoDB. You can of course convert it’s engine.

3.All data is stored in RAM! as a default, but Therefore you need lots of RAM on your cluster nodes. The formula how much RAM you need on each node goes like this:

(SizeofDatabase × NumberOfReplicas × 1.1 ) / NumberOfDataNodes

So if you have a database that is 1 GB of size, you would need 1.1 GB RAM on each node!

4.The cluster management node listens on port 1186, and anyone can connect. So that’s definitely not secure, and therefore you should run your cluster in an isolated private network! Since it would be harder to manage it may be a good idea to make changes to /etc/hosts.deny or prepare iptables based firewall.

5.It’s a good idea to have a look at the MySQL Cluster FAQ: http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-faq.html and also at the MySQL Cluster documentation: http://dev.mysql.com/doc/refman/5.1/en/ndbcluster.html

Links

• MySQL: http://www.mysql.com/
• MySQL Cluster documentation: http://dev.mysql.com/doc/refman/5.1/en/ndbcluster.html
• MySQL Cluster FAQ: http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-faq.html
• Ultra Monkey: http://www.ultramonkey.org/
• The High-Availability Linux Project: http://www.linux-ha.org/
• MySQL 5.1 Cluster Replication http://blog.dbadojo.com/2007/08/mysql-51-ndb-cluster-replication-on-ec2.html
• How to stress test my MySQL Cluster 5.1 http://blogs.techrepublic.com.com/howdoi/?p=133