HAProxy

[Ref: OpenBSD 5.7, 5.8, HAProxy 1.5.14 Configuration Manual ]

Why another document on configuring HAProxy? Because like so many, I also scan the Internet for HOWTOs instead of reading the project documentation.

I had a particular goal in mind when I first installed HaProxy, and since I'd got most of it working using something else Enginx how difficult could it be to rehash it in HaProxy? Why do I need to read documentation, why can't i just get a recipe and have it up and running in minutes instead of days?

It really isn't that difficult to go from

  • basic/simple HaProxy configuration to a
  • moderately difficult configuration, to
  • your desired configuration.

After many missteps I find that new deployments work better following simple steps like the above. Get the basic environment up and running to make sure all the bits and pieces are actually working as expected before debugging the more difficult 'beautiful' design I had in mind 8-)

The Project documentation has a nice Section 2 configuring HAProxy

2.0. Configuring HAProxy
2.1. Configuration file format 2.2. Time format
2.3. Examples

It didn't make sense to me when I first read it, but I know you could do better (then you can come back here to be confused.)

HAProxy's configuration process involves 3 major sources of parameters :

  - the arguments from the command-line, which always take precedence
  - the "global" section, which sets process-wide parameters
  - the proxies sections which can take form of "defaults", "listen",
    "frontend" and "backend".

Of course, after having a working system the documentation makes a whole lot more sense?

If you're trying HAProxy for the first time, I recommend you go through at least the examples in the above reference.

Packages

The basic installation for OpenBSD is through the ports/package system.

Once up and installed, and we're ready to test our installation the process I generally use re-emphasises the first configuration process:

HAProxy's configuration process involves ... :

  - the arguments from the command-line, which always take precedence
  • When verifying a configuration change or new updates I normally run 'haproxy' in console debug mode with the following commands.
1 $ sudo haproxy -c -f /etc/haproxy/haproxy.conf
  • Command-line:
    • -c checks the config file and exits.
    • -f specify the configuration file
1 $ sudo haproxy -d -f /etc/haproxy/haproxy.conf
  • Command-line:
    • -d Start in foreground with debugging mode enabled.
    • -f specify the configuration file

The above spits out a lot of debug information useful for visualising what the proxy is doing.

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
....

Basic Example

A simple adaptation of the first example from the documentation highlights how simple (flexible?) HAProxy can be.

Below is a simple configuration for an HTTP proxy listening port 80 on all interfaces and forwarding requests to two backend servers.

File: /etc/haproxy/haproxy.conf

 1     global
 2         daemon
 3         maxconn 256
 4 
 5     defaults
 6         mode http
 7         timeout connect 5000ms
 8         timeout client 50000ms
 9         timeout server 50000ms
10 
11     listen http-in
12         bind *:80
13         server websvr1 192.168.0.5:80 maxconn 32
14         server websvr2 192.168.0.6:80 maxconn 32

Presuming that 192.168.0.5 and 192.168.0.6 are web servers with a simple web page "This is HTTP Server websvrX", then you could expect to have a result such as:

$ sudo haproxy -c -f /etc/haproxy/haproxy.conf
$ sudo haproxy -f /etc/haproxy/haproxy.conf
$ while true; do curl http://localhost; sleep 1; done
This is HTTP server websvr1 (192.168.0.5).
This is HTTP server websvr2 (192.168.0.6).
This is HTTP server websvr1 (192.168.0.5).
This is HTTP server websvr2 (192.168.0.6).
...

TO help you familiarise with the syntax, and using the manual we'll walk through the directives in the sample configuration.

global

The general settings are configured in the (line 1) "global" and (line 5) "defaults" section, and the real work is configured in the (line 11)"listen" section labelled "http-in"

The 'global' parameters are categorised into three separate groups.

  • Process management and security,
  • Performance tuning, and
  • Debugging
Process Management and Security

For Process Management and Security we specify daemon

2         daemon
daemon
Makes the process fork into background. This is the recommended mode of
operation. It is equivalent to the command line "-D" argument. It can be
disabled by the command line "-db" argument.

Performance Tuning

For Performance tuning we set the maxconn rate

3         maxconn 256
maxconn <number>
Sets the maximum per-process number of concurrent connections to <number>. It
is equivalent to the command-line argument "-n". Proxies will stop accepting
connections when this limit is reached. The "ulimit-n" parameter is
automatically adjusted according to this value. See also "ulimit-n". Note:
the "select" poller cannot reliably use more than 1024 file descriptors on
some platforms. If your platform only supports select and reports "select
FAILED" on startup, you need to reduce maxconn until it works (slightly
below 500 in general).
Debugging

There is no debugging specified in the configuration file, and we've shown above and below that we can specify this on the command-line.

(you can look those up at: debug or quiet)

Proxies

The Proxy configuration
can be located in a set of sections :

  • defaults <name>
  • frontend <name>
  • backend <name>
  • listen <name>

Most examples you'll find involve the 'frontend' and 'backend', but in our simplified example, the required services are provided in a 'listen' section.

Defaults

5     defaults
6         mode http
7         timeout connect 5000ms
8         timeout client 50000ms
9         timeout server 50000ms
Right now, two major proxy modes are supported : "tcp", also known as layer 4,
and "http", also known as layer 7. In layer 4 mode, HAProxy simply forwards
bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the
protocol, and can interact with it by allowing, blocking, switching, adding,
modifying, or removing arbitrary contents in requests or responses, based on
arbitrary criteria.

For this example's defaults, we are going to be dealing with http (layer 7) traffic, with some preset timeouts related to http traffic.

Listen

A "listen" section defines a complete proxy with its frontend and backend
parts combined in one section. It is generally useful for TCP-only traffic.
11     listen http-in
12         bind *:80
13         server websvr1 192.168.0.5:80 maxconn 32
14         server websvr2 192.168.0.6:80 maxconn 32
bind [<address>]:<port_range> [, ...] [param*]
bind /<path> [, ...] [param*]

Define one or several listening addresses and/or ports in a frontend.

HAProxy will listen (bind) on all interfaces (".") at port 80. All traffic for that port will be load balanced to two web servers:

  • websvr1 which is at host: 192.168.0.5 port 80, and
  • websvr2 which is at host: 192.168.0.6 port 80

Below we show how we can start the HaProxy server to review the above configuration.

Command-line Interface

Two ports/packages are important for using HAProxy on OpenBSD are:

Socat isn't really required on OpenBSD, as Netcat nc(1) is in 'base' and supports '-U' UNIX-domain socket connections. But you will find the manual and majority of online references using socat.

Parse the Configuration File

The basic command-line option is to parse/check the configuration file.

$ sudo haproxy -c -f /etc/haproxy/haproxy.conf

Run in the foreground

Command-line options:

-d    Start in foreground with debugging mode enabled. When the proxy
      runs in this mode, it dumps every connections, disconnections,
      timestamps, and HTTP headers to stdout. ...
$ sudo haproxy -d -f /etc/haproxy/haproxy.conf
Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
[WARNING] ..
[ALERT] .. Starting frontend absdefg: cannot bind socket [0.0.0.0:80]
[ALERT] .. Starting frontend abcdefghij: cannot bind socket [0.0.0.0:443]

The errors are well categorised, [ALERTS] highlight sections that will definitely cause problems with your installation (such as in the above example, it is not going to work as you expect.) [WARNING]s are likely to cause problems and you should probably investigate and make a determination whether you need to fix or it can be safely ignored.

Soft Reconfiguration

[Ref: HAProxy hot-reconfiguration, HAProxy reloading your config with minimal service impact]

HAProxy supports 'soft-reconfiguration,' a mechanism for pausing or disabling haproxy without effecting existing connections.

-sf 
      Send FINISH signal to the pids in pidlist after startup. The   
      processes which receive the signal will wait for all sessions
      to finish before exiting. This option must be specified last,
      followed by any number of PIDs. Technically speaking, SIGTTOU
      and SIGUSR1 are sent.

-st 
      Send TERMINATE signal to pids in pidlist after startup. The
      processes which receive this signal will wait immediately
      terminate, closing all active sessions. This option must be
      specified last, followed by any number of PIDs. Technically
      speaking, SIGTTOU and SIGTERM are sent.
$ sudo haproxy -f /etc/haproxy/haproxy.conf -sf
$ sudo haproxy -f /etc/haproxy/haproxy.conf -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.conf)

Statistics on the Command-line

[Ref: Unix Socket commands, HAProxy stats from the command line]

The command-line, live query is based on HAProxy's 'stats' features which are configured using something like the below (more details later)

1 global
2         stats socket /var/run/haproxy/admin.sock mode 600 level admin
3         stats timeout 2m

[ Ref: stats socket]

stats socket [<address:port>|<path>] [param*]
Binds a UNIX socket to <path> or a TCPv4/v6 address to <address:port>.
Connections to this socket will return various statistics outputs and even
allow some commands to be issued to change some runtime settings. Please
consult section 9.2 "Unix Socket commands" for more details.

All parameters supported by "bind" lines are supported, for instance to
restrict access to some users or their access rights. Please consult
section 5.1 for more information.

stats timeout <timeout, in milliseconds>
The default timeout on the stats socket is set to 10 seconds. It is possible
to change this value with "stats timeout". The value must be passed in
milliseconds, or be suffixed by a time unit among { us, ms, s, m, h, d }.

To get a list of available options, send a 'blank' to the socket, such as:

$ echo "" | sudo nc -U /var/run/haproxy/admin.sock
Unknown command. Please enter one of the following commands only :
  clear counters : clear max statistics counters (add 'all' for all counters)
  clear table    : remove an entry from a table
  help           : this message
  prompt         : toggle interactive mode with prompt
  quit           : disconnect
  show info      : report information about the running process
  show pools     : report information about the memory pools usage
  show stat      : report counters for each proxy and server
  show errors    : report last request and response errors for each proxy
  show sess [id] : report the list of current sessions or dump this session
  show table [id]: report table usage stats or dump this table's contents
  get weight     : report a server's current weight
  set weight     : change a server's weight
  set server     : change a server's state or weight
  set table [id] : update or create a table entry's data
  set timeout    : change a timeout setting
  set maxconn    : change a maxconn setting
  set rate-limit : change a rate limiting value
  disable        : put a server or frontend in maintenance mode
  enable         : re-enable a server or frontend which is in maintenance mode
  shutdown       : kill a session or a frontend (eg:to release listening ports)
  show acl [id]  : report avalaible acls or dump an acl's contents
  get acl        : reports the patterns matching a sample for an ACL
  add acl        : add acl entry
  del acl        : delete acl entry
  clear acl  : clear the content of this acl
  show map [id]  : report avalaible maps or dump a map's contents
  get map        : reports the keys and values matching a sample for a map
  set map        : modify map entry
  add map        : add map entry
  del map        : delete map entry
  clear map  : clear the content of this map
  set ssl  : set statement for ssl
$ echo "show info" | sudo nc -U /var/run/haproxy/admin.sock
Name: HAProxy
Version: 1.5.14
Release_date: ...
...

Using 'socat' is similar

$ echo "show errors" | sudo socat /var/run/haproxy/admin.sock

Global

[Ref: Global parameters]

Parameters in the "global" section are process-wide and often OS-specific. They
are generally set once for all and do not need being changed once correct. Some
of them have command-line equivalents.

As mentioned earlier, HAProxy's global section can be logically grouped into three groups:

  • Process Management and Security
  • Performance tuning
  • Debugging
 1 global
 2         log 127.0.0.1   local0 info
 3         log 127.0.0.1   local1 notice
 4         maxconn 50000
 5         chroot /var/haproxy
 6         user _haproxy
 7         group _haproxy
 8         daemon
 9         pidfile /var/run/haproxy.pid
10         stats socket /var/run/haproxy/admin.sock mode 660 level admin
11         stats timeout 30s

We have covered some of the configuration settings earlier, and the more interesting parts from our production environment are the "log" specifications, and the "statistics" directives.

Logging

[Ref: log]

When we enable logging using log <address> <facility> [max level] we are sending startup, exit notifications.

log <address> [len <length>] <facility> [max level [min level]]
Adds a global syslog server. Up to two global servers can be defined. They
will receive logs for startups and exits, as well as all logs from proxies
configured with "log global".

By default, configuring a log address sends all events to the log host. We can separate log entries from HAProxy with a configuration such as in the above.

2         log 127.0.0.1   local0 info
3         log 127.0.0.1   local1 notice
  • line 2 - send logs to address 127.0.0.1 syslog facility local0 and severity level info
  • line 3 - send logs to address 127.0.0.1 syslog facility local1 and severity level notice

Statistics

 9         stats socket /var/run/haproxy/admin.sock mode 660 level admin
10         stats timeout 30s

As shown in the earliest example, you can specify the statistical view in the Global section. There are options, one described further below.

Proxies

[Ref: Proxies]

Right now, two major proxy modes are supported : "tcp", also known as layer 4,
and "http", also known as layer 7. In layer 4 mode, HAProxy simply forwards
bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the
protocol, and can interact with it by allowing, blocking, switching, adding,
modifying, or removing arbitrary contents in requests or responses, based on
arbitrary criteria.
All proxy names must be formed from upper and lower case letters, digits,
'-' (dash), '_' (underscore) , '.' (dot) and ':' (colon). ACL names are
case-sensitive, which means that "www" and "WWW" are two different proxies.

Defaults

[Ref: Proxies]

A "defaults" section sets default parameters for all other sections following
its declaration. Those default parameters are reset by the next "defaults"
section. See below for the list of parameters which can be set in a "defaults"
section. The name is optional but its use is encouraged for better readability.
13 defaults
14         log global # use the log definition from the 'global' section
15         mode http
16         balance roundrobin
17 
18         timeout http-request 50s
19         timeout connect 5000s
20         timeout client 50000s
21         timeout server 50000s

frontend

[Ref: Proxies]

A "frontend" section describes a set of listening sockets accepting client
connections.

The naming convention we find on the Internet is of the format: * ft_servicename_protocolport*

  • The text "ft_"
  • A name for the service + "" (e.g. mywebsite)
  • The protocol (e.g. http) and the port number (e.g. http80)
23 # Virtual IP use hdr_* and req_sni_* to switch depending on domain name
24 frontend ft_vip_http80
25         bind *:80
26         mode http
27         log global
28 
29         option httpclose    # add "Connection:close" header if it is missing
30         option forwardfor   # insert x-forwarded-for header so that app servers can see both proxy and
31         tcp-request inspect-delay 5s
32 
33         acl lb_nomoa_com_www_80 hdr_dom(host) -i nomoa.com
34         acl lb_nomoa_com_www_80 hdr_dom(host) -i www.nomoa.com
35 
36         use_backend bk_nomoa_com_www_80  if lb_nomoa_com_www_80
37 
38 frontend ft_vip_tcp443
39         bind *:443
40         mode tcp
41         log global
42 
43         tcp-request inspect-delay 5s
44         tcp-request content accept if { req_ssl_hello_type 1 }
45         
46         acl lb_nomoa_com_www_443 req_ssl_sni -i nomoa.com
47         acl lb_nomoa_com_www_443 req_ssl_sni -i www.nomoa.com
48 
49         use_backend bk_nomoa_com_www_443  if lb_nomoa_com_www_443

Back End

[Ref: Proxies]

A "backend" section describes a set of servers to which the proxy will connect
to forward incoming connections.
55 #
56 # BACK ENDS bk_
57 #
58 backend bk_nomoa_com_www_80
59         mode http
60         option httplog
61         balance roundrobin
62 
63         server svr1-nomoa.com-80 192.168.0.5:80 check
64         server svr2-nomoa.com-80 192.168.0.6:80 check
65         server svr3-nomoa.com-80 192.168.0.7:80 check backup
66 
67 backend bk_nomoa_com_www_443
68         mode tcp
69         option tcplog
70         balance roundrobin
71 
72         option httpchk HEAD / HTTP/1.1\r\nHost:www.nomoa.com
73         option ssl-hello-chk
74 
75         stick-table type binary len 32 size 30k expire 30m
76 
77         acl clienthello req_ssl_hello_type 1
78         acl serverhello rep_ssl_hello_type 2
79 
80         tcp-request inspect-delay 5s
81         tcp-request content accept if clienthello
82         tcp-response content accept if serverhello
83         
84         stick on payload_lv(43,1) if clienthello
85         stick store-response payload_lv(43,1) if serverhello
86 
87         server svr1-nomoa.com-443 192.168.0.5:443 check
88         server svr2-nomoa.com-443 192.168.0.6:443 check
89         server svr3-nomoa.com-443 192.168.0.7:443 check backup

Listen

[Ref: Proxies]

A "listen" section defines a complete proxy with its frontend and backend
parts combined in one section. It is generally useful for TCP-only traffic.
91 #
92 # STATS
93 #
94 
95 listen HAProxy-Statistics *:1936
96         mode http
97         stats enable
98         stats refresh 20s
99         stats auth username:mypassword

The example 'listen' section we have, highlights the general feature, but is better described below.

Statistics

[Ref: Proxies]

[Ref: Statistics and monitoring]

It is possible to query HAProxy about its status. The most commonly used
mechanism is the HTTP statistics page. This page also exposes an alternative
CSV output format for monitoring tools. The same format is provided on the
Unix socket.
91 #
92 # STATS
93 #
94 
95 listen HAProxy-Statistics *:1936
96         mode http
97         stats enable
98         stats refresh 20s
99         stats auth username:mypassword

Logging

[Ref: Logging

Configuring Syslog

[Ref: syslog.conf(5)]

You should have something like the below in your configuration file:

Format

File extract: /etc/syslog.conf

1 !haproxy
2 local0.*            /var/log/haproxy/info.log
3 local1.*            /var/log/haproxy/notice.log

The format is explained in the manpage, with some of the relevant extracts below:

Line 1: Blocks of rules(?) are separated with a tag line beginning with an exclamation mark.

1 !haproxy

manpage extracts: syslog.conf(5)

Each block of lines is separated from the previous block by a tag. 
The tag is a line beginning with !prog and each block will be 
associated with calls to syslog from that specific program. When a 
message matches multiple blocks, the action of each matching block 
is taken. If no tag is specified at the beginning of the file, every 
line is checked for a match and acted upon (at least until a tag is 
found).
2 local0.*            /var/log/haproxy/info.log
3 local1.*            /var/log/haproxy/notice.log
  • local0 and local1 are the facilities
  • '*' says to use all levels
The selectors are encoded as a facility, a period ('.'), and a 
level, with no intervening whitespace. Both the facility and the 
level are case insensitive.

The action field is /var/log/haproxy/info.log and /var/log/haproxy.notice.log

The action field of each line specifies the action to be taken when 
the selector field selects a message. There are six forms:

• A pathname (beginning with a leading slash). Selected messages are 
  appended to the file.

Note: the 'air-gap' between the 'selector' and 'action' are separated by 'tabs' not invisible blank spaces.

Tab Stops

You must use 'tabs' between the 'selector' field and the 'action' field. You can guarantee yourself a disfunctioning logging environmnet if you ignore this and blindly presume the 'whitespaces' are correct (even if it is correct for /etc/newsyslog.conf)

manpage extract: syslog.conf(5)

The selector field is separated from the action field by one or more tab characters

The simplest way to view whether you used 'tabs' or spaces, is to open the file in 'vim' and expose tabs using the commands: Ref: http://vi.stackexchange.com/questions/422/displaying-tabs-as-characters

1 :set list
2 :set listchars=tab:>-

For those with "Unicode" enabled in VIM and their console:

  • :set list
  • :set listchars=eol:⏎,tab:↦→,trail:␠,nbsp:⎵
  • :set list listchars=tab:>-,trail:.,extends:> " Enter the middle-dot by pressing Ctrl-k then .M
  • :set list listchars=tab:\|_,trail:· " Enter the right-angle-quote by pressing Ctrl-k then >>
  • :set list listchars=tab:»·,trail:· " Enter the Pilcrow mark by pressing Ctrl-k then PI
  • :set list listchars=tab:>-,eol:¶ " The command :dig displays other digraphs you can use.

If you put these two lines in your .vimrc, tabs will be shown as A6; for the start position and → through the rest of the tab.

  • For eol, use U+23CE RETURN SYMBOL ⏎
  • For trail, use U+2420 SYMBOL FOR SPACE ␠
  • For nbsp, use U+23B5 BOTTOM SQUARE BRACKET ⎵
  • For tab, use U+21A6 Rightwards Arrow from Bar ↦ U+2192 Rightwards Arrow →

These characters are distinctive enough that they rarely appear literally in document text. To type these special characters into your vimrc, type Ctrl-V+u23CE while in Insert Mode, as explained in :help utf-8-typing.

References

 1 global
 2         log 127.0.0.1   local0 info
 3         log 127.0.0.1   local1 notice
 4         maxconn 50000
 5         chroot /var/haproxy
 6         user _haproxy
 7         group _haproxy
 8         daemon
 9         pidfile /var/run/haproxy.pid
10         stats socket /var/run/haproxy/admin.sock mode 660 level admin
11         stats timeout 30s
12 
13 defaults
14         log global # use the log definition from the 'global' section
15         mode http
16         balance roundrobin
17 
18         timeout http-request 50s
19         timeout connect 5000s
20         timeout client 50000s
21         timeout server 50000s
22 
23 #
24 # FRONT ENDS: ft_
25 #
26 
27 
28 # Virtual IP use hdr_* and req_sni_* to switch depending on domain name
29 frontend ft_vip_http80
30         bind *:80
31         mode http
32         log global
33 
34         option httpclose    # add "Connection:close" header if it is missing
35         option forwardfor   # insert x-forwarded-for header so that app servers can see both proxy and
36         tcp-request inspect-delay 5s
37 
38         acl lb_nomoa_com_www_80 hdr_dom(host) -i nomoa.com
39         acl lb_nomoa_com_www_80 hdr_dom(host) -i www.nomoa.com
40 
41         use_backend bk_nomoa_com_www_80  if lb_nomoa_com_www_80
42 
43 frontend ft_vip_tcp443
44         bind *:443
45         mode tcp
46         log global
47 
48         tcp-request inspect-delay 5s
49         tcp-request content accept if { req_ssl_hello_type 1 }
50         
51         acl lb_nomoa_com_www_443 req_ssl_sni -i nomoa.com
52         acl lb_nomoa_com_www_443 req_ssl_sni -i www.nomoa.com
53 
54         use_backend bk_nomoa_com_www_443  if lb_nomoa_com_www_443
55 #
56 # BACK ENDS bk_
57 #
58 backend bk_nomoa_com_www_80
59         mode http
60         option httplog
61         balance roundrobin
62 
63         server svr1-nomoa.com-80 192.168.0.5:80 check
64         server svr2-nomoa.com-80 192.168.0.6:80 check
65         server svr3-nomoa.com-80 192.168.0.7:80 check backup
66 
67 backend bk_nomoa_com_www_443
68         mode tcp
69         option tcplog
70         balance roundrobin
71 
72         option httpchk HEAD / HTTP/1.1\r\nHost:www.nomoa.com
73         option ssl-hello-chk
74 
75         stick-table type binary len 32 size 30k expire 30m
76 
77         acl clienthello req_ssl_hello_type 1
78         acl serverhello rep_ssl_hello_type 2
79 
80         tcp-request inspect-delay 5s
81         tcp-request content accept if clienthello
82         tcp-response content accept if serverhello
83         
84         stick on payload_lv(43,1) if clienthello
85         stick store-response payload_lv(43,1) if serverhello
86 
87         server svr1-nomoa.com-443 192.168.0.5:443 check
88         server svr2-nomoa.com-443 192.168.0.6:443 check
89         server svr3-nomoa.com-443 192.168.0.7:443 check backup
90 
91 #
92 # STATS
93 #
94 
95 listen HAProxy-Statistics *:1936
96         mode http
97         stats enable
98         stats refresh 20s
99         stats auth username:mypassword