Category: ‘Haproxy’

Haproxy Load balancing

May 24, 2011 Posted by admin

I recently had the pleasure playing with keepalived, haproxy and stunnel.
If your not familiar with haproxy it’s a layer 7 reverse proxy load balancer that runs in user space.
Now I have had the pleasure of playing with a few other open source load balancing solutions namely keepalived and linuxvirtualsever.
Linux virtualserver is a kernel module the run in kernel space and keepalived is just userspace software that controls and does health checks.
One of the reasons I went with Haproxy is because it’s eas to setup.
You can find a binary package with almost all linux distros.
One of the biggest challenges that I founds was finding a solution for apache logging and scripts or programs that needed to record the remote host ips. There were lots of howto’s out there that address some of the challenges but not all.

Install Haproxy and keepalived on your loadbalancers

[root@haproxy01 ~] yum install haproxy.x86_64 -y
[root@haproxy01 ~] yum install keepalived.x86_64 -y

[root@haproxy02 ~] yum install haproxy.x86_64 -y
[root@haproxy02 ~] yum install keealived.x86_64 -y

Next we install stunnel we have to get the version that is supported with the patches on the haproxy site.

[root@haproxy01 ~] wget http://haproxy.1wt.eu/download/patches/stunnel-4.32-xforwarded-for.diff
[root@haproxy01 ~] wget http://mirrors.zerg.biz/stunnel/obsolete/4.x/stunnel-4.34.tar.gz

[root@haproxy02 ~] wget http://haproxy.1wt.eu/download/patches/stunnel-4.32-xforwarded-for.diff
[root@haproxy02 ~] wget http://mirrors.zerg.biz/stunnel/obsolete/4.x/stunnel-4.34.tar.gz

Next we need to patch and build stunnel

[root@haproxy02 ~] tar -zxvf stunnel-4.32.tar.gz
[root@haproxy02 ~] cd stunnel-4.32
[root@haproxy02 ~]patch -p1 < ../stunnel-4.32-xforwarded-for.diff
[root@haproxy02 ~]./configure –prefix=/usr
[root@haproxy02 ~]make && make install

[root@haproxy02 ~] tar -zxvf stunnel-4.32.tar.gz
[root@haproxy02 ~] cd stunnel-4.32
[root@haproxy02 ~]patch -p1 < ../stunnel-4.32-xforwarded-for.diff
[root@haproxy02 ~]./configure –prefix=/usr
[root@haproxy02 ~]make && make install

Next we configure stunnel. You can configure multiple stunnel instances depending on the number of SSL enable sites you may have. Each SSL enabled site will need a unique ip address. You will need to configure stunnel on both haproxy servers.

sslVersion = all
options = NO_SSLv2
setuid = root
setgid = root
pid = /var/run/stunnel.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
output = /var/log/stunnel.log

[www.linuxdriven.us]
cert = /etc/stunnel/www.linuxdriven.us/server.crt
key = /etc/stunnel/www.linuxdriven.us/server.key
accept = 192.168.1.21:443
connect = 192.168.1.21:80
xforwardedfor = yes
TIMEOUTclose = 0

wget -O /etc/init.d/stunnel http://www.gaztronics.net/rc/stunnel.txt
chmod 755 /etc/init.d/stunnel
chkconfig stunnel on

Next we need to configure keepalived. We used this only to initlize the ip address and to monitor the haproxy process.

[root@haproxy01 ~] cat /etc/keepalived/keepalived.conf

global_defs {
notification_email {
sysadmin@linuxdriven.us
}
notification_email_from keepalived@haproxy01.linuxdriven.us
smtp_server 192.168.1.84
smtp_connect_timeout 30
}

vrrp_script chk_haproxy {
script “killall -0 haproxy” # widely used idiom
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}

vrrp_instance VI_1 {
interface eth0
state MASTER # or “BACKUP” on backup
priority 101 # 101 on master, 100 on backup
virtual_router_id 51

smtp_alert # Activate SMTP notifications

authentication {
auth_type AH
auth_pass supersecret
}

virtual_ipaddress {
192.168.1.21
192.168.1.22
192.168.1.23
192.168.1.24
192.168.1.25
}

track_script {
chk_haproxy
}
}

We need to configure the 2nd haproxy server. Key things to make note of on the 2nd server is the priority and the state. On the 2nd server set the state to BACKUP and the priority to 100.

[root@haproxy02 ~] cat /etc/keepalived/keepalived.conf
global_defs {
notification_email {
sysadmin@linuxdriven.us
}
notification_email_from keepalived@haproxy02.linuxdriven.us
smtp_server 192.168.1.84
smtp_connect_timeout 30
}

vrrp_script chk_haproxy {
script “killall -0 haproxy” # widely used idiom
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}

vrrp_instance VI_1 {
interface eth0
state MASTER # or “BACKUP” on backup
priority 101 # 101 on master, 100 on backup
virtual_router_id 51

smtp_alert # Activate SMTP notifications

authentication {
auth_type AH
auth_pass supersecret
}

virtual_ipaddress {
192.168.1.21
192.168.1.22
192.168.1.23
192.168.1.24
192.168.1.25
}

track_script {
chk_haproxy
}
}

Next we configure Haproxy. One of the things were going to do here is capture the HTTPS traffic with stunnel and redirect it to port 80.

[root@haproxy01 ~]cat /etc/haproxy/haproxy.cfg
global
daemon
maxconn 10000
nbproc 2
log 127.0.0.1 syslog

defaults
log global
clitimeout 60000
srvtimeout 30000
contimeout 4000
retries 3
option redispatch
option httpclose
option abortonclose
listen domain_cluster_http 192.168.1.21:80
mode http
balance roundrobin
cookie SERVERID insert nocache
option forwardfor except 192.168.1.17
reqadd X-Forwarded-Proto:\ http
server server1 192.168.1.40:80 cookie s1 weight 1 maxconn 5000 check
server server2 192.168.1.41:80 cookie s2 weight 1 maxconn 5000 check

listen domain_cluster_https 192.168.1.21:81
mode http
balance roundrobin
cookie SERVERID insert nocache
option forwardfor except 192.168.1.17
reqadd X-Forwarded-Proto:\ https
server server1 192.168.1.40:80 cookie s1 weight 1 maxconn 5000 check
server server2 192.168.1.41:80 cookie s2 weight 1 maxconn 5000 check

Don’t forget to to the same on the 2nd server. Next were going to configure Apache with the mod_rpaf2 this with change the X-Forwarded header to the real client’s ip

[root@haproxy01 ~] wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz
[root@haproxy01 ~] tar -zxvf mod_rpaf-06.tar.taz
[root@haproxy01 ~] cd mod_rpaf-0.6
[root@haproxy01 ~] mod_rpaf-0.6]# apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
/usr/lib/apr-1/build/libtool –silent –mode=compile gcc -prefer-pic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -fno-strict-aliasing -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -pthread -I/usr/include/httpd -I/usr/include/apr-1 -I/usr/include/apr-1 -c -o mod_rpaf-2.0.lo mod_rpaf-2.0.c && touch mod_rpaf-2.0.slo
mod_rpaf-2.0.c: In function ‘rpaf_cleanup’:
mod_rpaf-2.0.c:151: warning: implicit declaration of function ‘inet_addr’
mod_rpaf-2.0.c: In function ‘change_remote_ip’:
mod_rpaf-2.0.c:169: warning: suggest parentheses around assignment used as truth value
mod_rpaf-2.0.c:190: warning: suggest parentheses around assignment used as truth value
mod_rpaf-2.0.c:195: warning: suggest parentheses around assignment used as truth value
/usr/lib/apr-1/build/libtool –silent –mode=link gcc -o mod_rpaf-2.0.la -rpath /usr/lib/httpd/modules -module -avoid-version mod_rpaf-2.0.lo
/usr/lib/httpd/build/instdso.sh SH_LIBTOOL=’/usr/lib/apr-1/build/libtool’ mod_rpaf-2.0.la /usr/lib/httpd/modules
/usr/lib/apr-1/build/libtool –mode=install cp mod_rpaf-2.0.la /usr/lib/httpd/modules/
cp .libs/mod_rpaf-2.0.so /usr/lib/httpd/modules/mod_rpaf-2.0.so
cp .libs/mod_rpaf-2.0.lai /usr/lib/httpd/modules/mod_rpaf-2.0.la
cp .libs/mod_rpaf-2.0.a /usr/lib/httpd/modules/mod_rpaf-2.0.a
chmod 644 /usr/lib/httpd/modules/mod_rpaf-2.0.a
ranlib /usr/lib/httpd/modules/mod_rpaf-2.0.a
PATH=”$PATH:/sbin” ldconfig -n /usr/lib/httpd/modules
chmod 755 /usr/lib/httpd/modules/mod_rpaf-2.0.so

vi /etc/httpd/conf/httpd.conf

LoadModule rpaf_module modules/mod_rpaf-2.0.so
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 192.168.1.17
RPAFheader X-Forwarded-For

All that’s left to do is start up the services in the following order

on both loadbalancers start keepalived
then startup haproxy and stunnel