Linux

I've been using Linux/Unix for many years. I've always had a strong interest in technology in general and computing specifically.

These are my opinions. Opinions are like noses, everyone has one, and they all smell.

Enjoy your visit.
July 2017
M T W T F S S
« Dec    
 12
3456789
10111213141516
17181920212223
24252627282930
31  

Remote Logging Apache with syslog-ng

Project

We have been recently working on a project to configure a remote logging server for a cluster of web servers running behind a load balancer. In the standard configuration, each server logs locally. To analyze statistics we have to download the logs from each of the servers and then do the analysis on a local workstation. Since multiple servers in a load balanced cluster provides the identical content for a given virtual host, the log files from each server have to be downloaded and combined to do log analysis for a single site. As you can imagine this is tedious and time consuming.

The goal of this project was to get all the virtual hosts logging to a single log server, with each virtual host writing their files to a single log. This will allow the administrator to download from a single host or do the log analysis on the log server. Because the log server is in another network and the logging will be over the internet, it would be preferable that the traffic be encrypted.

Apache Logging

Apache has built in logging facilities and doesn’t depend on something like syslog. However, in order to do remote logging it appeared at first reading that we needed to change the apache configuration to use some kind of syslog. We chose  syslog-ng, because of additional features that are not in the old syslogd.

To make apache log with syslog-ng we created the following entries in /opt/syslog-ng/etc/syslog-ng.conf.

#######
# sources #
#######

source s_apache {
                 unix-stream("/var/log/httpd/apache_log.socket"
                 max-connections(512)
                 keep-alive(yes));
};

################
# destinations #
################

destination d_custom_access {
                            file("/var/log/httpd/custom_access.log");
};

destination d_custom_error {
                            file("/var/log/httpd/custom_error.log");
};

###########
# filters #
###########

filter f_apache_access {
                       message("GET|POST");
};
filter f_apache_error {
                       message("error");
};

log {
     source(s_apache);
     filter(f_apache_access);
     destination(d_custom_access);
};

log {
     source(s_apache);
     filter(f_apache_error);
     destination(d_custom_error);
};

#######
# end #
#######

When we restart syslog-ng the socket apache_log appears in /var/log/httpd/. Now we’ll make apache write it’s logs to the socket, where syslog-ng was configured to listen.

We edit the default apache logging configuration and change,

ErrorLog logs/error_log

to

ErrorLog "|/usr/bin/logger -t 'apache'  -u /var/log/httpd/apache_log.socket"

and change,

CustomLog logs/access_log combined

to

CustomLog "|/usr/bin/logger -t 'apache'  -u /var/log/httpd/apache_log.socket" combined

Where this is located is slightly different from one system to another. If the system we are working on has one large configuration file for apache, it will probably be in the main configuration file. Some  systems break the configuration into many smaller parts. On the system we’re working on, there is a separate logs.conf file. We have to consult our local system documentation to find where this is on the system we’re administering.

This configuration change causes apache to use the logger program to write to the socket /var/log/httpd/apache_log.socket. For testing, I changed the default configuration, then commented out the logging lines on a virtual host, which makes logging use the default configuration. After restarting apache, logs start to appear in /var/log/httpd/custom_access.log and /var/log/httpd/custom_error.log, as I expected. If our site is slow, this can take a little time, because logs won’t get written until there is something to log.

SSH Tunnel

So now we have successfully configured apache to use syslog-ng in place of it’s native logging service, to write to logs on the local file system. Next step is to get it to log to a remote server, over an encrypted channel.

One option is to use an ssh tunnel between the server and the logging host to encrypt the traffic. We can create a special user logman and generate keys, without any passphrase. This user will open the tunnel on a port higher than 1024. Lower numbered ports require root access. For security we prefer not to use root access, especially with a key pair that isn’t protected by a passphrase. Here are the steps I used to create this user and set it up to create and ssh tunnel.

# adduser logman
# su - logman
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/logman/.ssh/id_rsa):
Created directory '/home/logman/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/logman/.ssh/id_rsa.
Your public key has been saved in /home/logman/.ssh/id_rsa.pub.
The key fingerprint is:
23:9f:e4:59:94:59:32:f8:22:8c:16:37:a4:6a:ed:bf

Next we configure an ssh-tunnel from the apache server to the logging server using port 2514. We can start the tunnel from init, by creating an entry in the /etc/inittab file.

log1:345:respawn:/usr/bin/ssh -i /home/logman/.ssh/id_dsa -nNTx -R 2514:logging.server.com:2514 logman@logging.server.com > /dev/null 2>&1

We can check to see if the tunnel came up as expected, with netstat.

# netstat -tunap | grep 2514
tcp        0      0 127.0.0.1:2514              0.0.0.0:*                   LISTEN      12019/sshd: logman
tcp        0      0 ::1:2514                    :::*                        LISTEN      12019/sshd: logman

Now we configure /opt/syslog-ng/etc/syslog-ng.conf on the logging server to log these incoming connections with the following lines:

# source from incoming ssh connection on apache
#######
# sources #
#######

# message from apache
source s_apache {
                             tcp(ip(127.0.0.1)
                            port(2514)
                           max-connections(300));
};
##########
# destinations #
##########
# destinations from remote servers

destination d_apache_custom_access {
                                                               file ("/var/log/httpd/custom_acess.log);
};
destination d_apache_custom_error {
                                                           file ("/var/log/httpd/custom_error.log");
};

######
# filters #
######
# apache messages from remote servers

filter f_apache_access {
                                      message("GET|POST");
};

filter f_apache_error {
                                    message("error");
};

#####
# logs #
#####

# custom logs for apache
log {
       source(s_apache);
       filter(f_apache_access);
       destination(d_apache_custom_access);
};

log {
       source(s_apache);
        filter(f_apache_error);
       destination(d_apache_custom_error);
};

#####
# end #
#####

We check for the files custom_error.log and custom_access.log on the logging server, but they are not created until a log is generated.

On this project,  I chose to test a low traffic virtual host , so it wouldn’t interfere with high volume data. What I didn’t know at the time, was that I was also working on the secondary web server of the load balanced pair. The little traffic that was going to this domain was hitting the primary server and logging there. I expected the files to be created as soon as I restarted syslog-ng on the logging server. That didn’t happen and I surmised that it wasn’t passing data down the tunnel. Later I learned that the files only get created on the logging server, when data gets passed, not before. So the ssh tunnel may have been working correctly, I didn’t script tests to hit the server to force it to log. I chose the ssh tunnel option first because I figured it would be fastest and easiest to set up. You might want so see if it works for you.

The newer versions of syslog-ng are capable of doing TLS encryption. However the older versions installed with the package manager on the servers didn’t support TLS. So I had to replace the packages provided in the repository and download and install the newer version from the syslog-ng web site. If you decide to try setting up remote logging, make sure you have syslog-ng version 3.0 or later. I would recommend using the same version on all servers.

TCP Transport

Before we get TLS working we should configure logging to use an unencrypted tcp transport. We don’t need a filter since we’re sending to a remote logging server. We can filter on the logging server.

##########
# source #
##########

source s_apache {
                 unix-stream("/var/log/httpd/apache_log.socket"
                 max-connections(512)
                 keep-alive(yes));
};

################
# destinations #
################

destination d_messages {
                        file("/var/log/messages");
};

destination d_syslog_apache {
                             syslog("logger.example.com"
                             port(6514)
                             transport("tcp"));
};

#######
# log #
#######

log {
     source(s_apache);
     destination(d_syslog_apache);
};
#######
# end #
#######

A note about the hostname. If the hostname cannot be resolved, perhaps because it’s not in DNS,
then you must use the IP address in the configuration, above. I experimented with entries in /etc/hosts,
however, for some reason only hostnames in DNS are recognized by syslog-ng. To avoid problems make sure the hostname is resolvable.

At this point we’re still using the socket apache_log.socket to listen to apache log messages. Only now our destination has changed to d_syslog_apache which uses the IETF syslog protocol to talk to our
remote server, logger.example.com on port 6514 using the tcp transport. As before, a low volume
web server can take some time to write logs on the remote server. You might be able to use some
web server testing tool to simulate a load on the server and cause logs to be written immediately. Or you might just right a quickie shell script to wget the web page.

SSL Certificate

In order to configure TLS we need SSL certificates. In order to generate those TLS certificates
we need to create our own Certificate Authority to sign the certificates. So we start by creating
a certificate authority.

When we create a Certificate Authority we set the expiration for the CA at 20 years. This way
we won’t have to renew any certificates any time soon. You might choose to expire the CA sooner.
Remember though if the Certificate Authority expires then all the certificates have to be revoked
and then you have to create new ones. If you only have a few, this probably isn’t a problem. If you
use a lot of keys, this could be an inconvenience.

# openssl req -new -x509 -extensions v3_ca -keyout private/cakey.pem \
-out cacert.pem -days 7300

Then we create a certificate request.

# openssl req -new -nodes -out req_logger.pem

This creates two files req_logger.pem and privkey.pem.

Next we need to sign the certificate

# openssl ca -out logger_cert.pem -config openssl.cnf -infiles logger_req.pem
Using configuration from openssl.cnf
Enter pass phrase for ./private/CAkey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Apr  2 00:47:34 2010 GMT
            Not After : Apr  1 00:47:34 2012 GMT
        Subject:
            countryName               = US
            stateOrProvinceName       = California
            organizationName          = LinuxGeek
            organizationalUnitName    = logger
            commonName                = Neil Schneider
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                FB:7B:B1:97:2C:97:E7:FF:32:6F:4D:E4:5A:E4:66:DE:3C:EB:31:56
            X509v3 Authority Key Identifier:
                keyid:06:16:0A:10:5B:B6:4F:68:E2:7E:CD:25:5B:FC:34:F1:E7:37:B3:7B

Certificate is to be certified until Apr  1 00:47:34 2012 GMT (730 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

TLS Transport

Now that we have a ssl certificates to use for encrypting our sessions,  we need to configure syslog-ng to use them. We need certificates for the server placed on each client machine. We’ll create a directory /opt/syslog-ng/etc/ca.d on each machine that will be logging to the server, and copy the key syslog-ng-server.crt into that directory. Then we’ll find the hash for that cert.

# openssl x509 -noout -hash -in syslog-ng-server.crt
98a089d7

Then we’ll create a symbolic link using that hash

ln -s syslog-ng-server.crt 98a089d7

Then we need to create a destination statement specifying the directory where the cert is, using either tcp transport of the IETF syslog transport.

tcp

destination d_apache_tls {
                         tcp("logger.network.com"
                         port(6515)
                         tls( ca_dir("/opt/syslog-ng/etc/ca.d")));
};

syslog

destination d_syslog_apache_tls {
                                 syslog("logger.network.com"
                                 port(6515)
                                 tls(ca_dir("/opt/syslog-ng/etc/ca.d")));
};
#######
# end #
#######

Warning

The encrypted connection between the server and the client fails if the Common Name or the subject_alt_name parameter of the server certificate does not contain the hostname or the IP address (as resolved from the syslog-ng clients and relays) of the server.

Now we need to copy the certificates to the server logger.example.com. We’ll create two directories and put the certificate in the directory /opt/syslog-ng/etc/cert.d/ and the matching key under /opt/syslog-ng/etc/key.d/.

Next we need to create appropriate source statement in /opt/syslog-ng/etc/syslog-ng.conf that uses the tls source driver.

tcp

source s_tls_app1 {
                   tcp(ip(0.0.0.0)
                   port (6514)
                   tls( key_file("/opt/syslog-ng/etc/key.d/syslog-ng-server.key")
                   cert_file("/opt/syslog-ng/etc/cert.d/syslog-ng-server.crt)
                   peer_verify(optional-untrusted));
};

syslog

###########
# sources #
###########

source s_tls_syslog { syslog(ip(0.0.0.0) port(6514)
                                 tls( key_file("/opt/syslog-ng/etc/key.d/syslog-ng-server.key")
                                 cert_file("/opt/syslog-ng/etc/cert.d/syslog-ng-server.crt")) );
};
#######
# end #
#######

The option tls( peer_verify(optional-untrusted) is added so the server does not try to authenticate the client. It’s possible to configure TLS so the server also authenticates the clients. This requires that each client has a certificate and that the server has a copy of the certificate from the CA, as well as certificates for each client. If you have a large cluster of machines this can get complex to manage, quickly.

After discovering that the logs were being written to logger.example.com in syslog format I went looking for other answers. I found a couple of pages that helped me find the final solution.

Final Solution

Peter Höltzl has an excellent page “How to collect apache logs by syslog-ng” which was helpful in getting me closer to the final solution. He has some excellent tips on setting up directories in the logging path, so that the logs get sorted. Peter also includes some great tips on options to create directories on the fly and set permissions. He essentially used the same methods for getting messages out of apache, that we read on other sites. The only difference was that he created his socket manually using the mknod command. The page helped move us along toward the conclusion but it was missing some key pieces, we needed.

The page on the University of Iowa site, Using syslog-ng to collect remote Apache web server logs, really was the key to the final solution. It showed us that we can log from the regular apache files and use the flags(no-parse) option to keep log format the same across the connection. No need to create a special socket, in fact, no need to make any changes to the logging method configured in apache. Just read the log file as if it were a socket. Essentially doing tail -f to the log file and sending it over the tcp connection to the remote log server.

The page includes a method to parse and insert the logs into a database. We aren’t logging to a database, but do want to separate the logs for each virtual host.

Since it would be very time consuming to write a log file for each virtual server by hand we wrote a perl script to create them. We created a file with a list of virtual hosts and the two ports we would use for logging. Each line in the file looked similar to this;

example.com|6414|6415

We opened the file in the perl script put the data into an array  and then gave names to each element in the array. The two numbers are the ports for the two streams, access and error. We used the same file for building the syslog-ng configuration files on both the web server/clients and the logging server. The final config file for each domain looks similar to this. There is one config file named with the domain name, for each virtual domain. So the file for example.com would be  named example.com. We use an include statement at the top of the syslog-ng.conf file to reference these files..

include "/opt/syslog-ng/etc/client.d"

This is the second line in the configuration, right after the @version: line.  It tells syslog-ng to include all files in that directory. We could have listed each file in an include line, but that would be a bit tedious with 100+ virtual hosts, the directory method seems more straightforward.

We could also have written out a single file with all the virtual hosts listed in it. We felt this would have been more of a maintenance headache, since it would require hand editing, to add or delete virtual hosts. With a single file named after each individual domain, creating new files is simple and deleting discontinued virtual hosts simply requires us to move or remove the configuration file from the directory.

The configuration file example.com for a single virtual host example.com on the syslog-ng client side would look like this:

################################################################################
#Configuration for example.com
################################################################################

##########
#sources #
##########

source s-example.com-access {
                            file ("/var/log/httpd//access.log"
                            flags(no-parse));
};
# source for error logs
source s-example.com-error {
                           file ("/var/log/httpd//error.log"
                           flags(no-parse));
};

################
# destinations #
################

destination d-example.com-access {
                                 tcp("logger.example.com"
                                 port(6415));
};
destination d-example.com-error {
                                tcp("logger.example.com"
                                port(6416));
};

#######
# log #
#######

log {
    source(s-example.com-access);
    destination(d-example.com-access);
};

log {
     source(s-example.com-error);
     destination(d-example.com-error);
};
#######
# end #
#######

On the log server side the configuration file for the same virtual host looks like this.

################################################################################
# Configuration for example.com
################################################################################

##########
#sources #
##########

source s-example.com-access {
                            tcp(ip(0.0.0.0) port(6415)
                            flags(no-parse) );
};

source s-example.com-error {
                           tcp(ip(0.0.0.0) port(6416)
                           flags(no-parse) );
};

################
# destinations #
################

destination d-example.com-access {
                                 file("/var/log/httpd/$R_YEAR/$R_MONTH/example.com/$R_DAY-access.log"
                                 template("$MSGONLY\n")
                                 template-escape(no)
                                 owner("root")
                                 group("adm")
                                 perm(0640)
                                 create_dirs(yes)
                                 dir_owner("root")
                                 dir_group("adm") );
                                 };
destination d-example.com-error {
                                 file(\"/var/log/httpd/$R_YEAR/$R_MONTH/example.com/$R_DAY-error.log"
                                 template("$MSGONLY\n")
                                 template-escape(no)
                                 owner("root")
                                 group("adm")
                                 perm(0640)
                                 create_dirs(yes)
                                 dir_owner("root")
                                 dir_group("adm") );
                                 };

#######
# log #
#######

log {
    source(s-example.com-access);
    destination(d-example.com-access);
};

log {
    source(s-example.com-error);
    destination(d-example.com-error);
};

#######
# end #
#######

The destination objects contain some extra macros that help sort the logs.

$R_YEAR
designates the Year number, in this configuration it will create a new directory for each year
$R_MONTH
designates the Month number, again configured to create a new directory for each month
$R_DAY
designates the Day of the Month

So in our setup the logs are sorted /var/log/httpd/YEAR/MONTH/VIRTUAL_DOMAIN and then the logs are Day_Of_Month-access.log and Day_Of_Month-error.log. The access.log comes over unaltered to the log server on port 6414 and the error log arrives on port 6415.

To add the TLS layer on  top, we generate the keys for the server. Then we add the tls later to both the client and server configuration.

On the client side we add the tls configuration options for the transport.

################################################################################
#Configuration for example.com
################################################################################

##########
#sources #
##########

source s-example.com-access {
                             file ("/var/log/httpd//access.log"
                             flags(no-parse));
};

# source for error logs
source s-example.com-error {
                            file ("/var/log/httpd//error.log"
                            flags(no-parse));
};

################
# destinations #
################

destination d-example.com-access {
                                  tcp("logger.example.com"
                                  port(6415)
                                  tls(ca_dir("/opt/syslog-ng/etc/syslog-ng/ca.d")
                                  key_file("/opt/syslog-ng/etc/syslog-ng/key.d/client.key")
                                  cert_file("/opt/syslog-ng/etc/syslog-ng/cert.d/client_cert.pem"));
};
destination d-example.com-error {
                                  tcp("logger.example.com"
                                  port(6416));
                                  tls(ca_dir("/opt/syslog-ng/etc/syslog-ng/ca.d")
                                  key_file("/opt/syslog-ng/etc/syslog-ng/key.d/client.key")
                                  cert_file("/opt/syslog-ng/etc/syslog-ng/cert.d/client_cert.pem")
};

#######
# log #
#######

log { source(s-example.com-access);
 destination(d-example.com-access);
};

log { source(s-example.com-error);
 destination(d-example.com-error);
};
#######
# end #
#######
On the log server side the configuration file for the same virtual host configured using tls look like this.
################################################################################
# Configuration for example.com
################################################################################

##########
#sources #
##########

source s-example.com-access {
                             tcp(ip(0.0.0.0) port(6415)
                             flags(no-parse)
                             tls( key_file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")
                             cert_file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert")
                             ca_dir("/opt/syslog-ng/etc/syslog-ng/ca.d")) ););
};

source s-example.com-error {
                            tcp(ip(0.0.0.0) port(6416)
                            flags(no-parse)
                            tls( key_file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")
                            cert_file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert")
                            ca_dir("/opt/syslog-ng/etc/syslog-ng/ca.d")) ););
};

################
# destinations #
################

destination d-example.com-access {
                    file("/var/log/httpd/$R_YEAR/$R_MONTH/example.com/$R_DAY-access.log"
                    template("$MSGONLY\n")
                    template-escape(no)
                    owner("root")
                    group("adm")
                    perm(0640)
                    create_dirs(yes)
                    dir_owner("root")
                    dir_group("adm") );
};
destination d-example.com-error {
                    file(\"/var/log/httpd/$R_YEAR/$R_MONTH/example.com/$R_DAY-error.log"
                    template("$MSGONLY\n")
                    template-escape(no)
                    owner("root")
                    group("adm")
                    perm(0640)
                    create_dirs(yes)
                    dir_owner("root")
                    dir_group("adm") );
};

#######
# log #
#######

log {
    source(s-example.com-access);
    destination(d-example.com-access);
};

log {
    source(s-example.com-error);
    destination(d-example.com-error);
};
#######
# end #
#######

If we generated our keys correctly and the hosts can resolve the remote hostnames, when we restart sylog-ng it should come up  with TLS encryption enabled.

The one drawback to using encryption with this many connections is that it might slow the server down. Every connection has to be encrypted. Each server in this cluster had 119 virtual hosts. There are two streams, access and error, for each host. That’s 238 streams and there are two servers in the cluster. The log server isn’t doing much else, so it might not have a problem decrypting 476 streams. But the web servers contain some dynamic content, so the additional load of encrypting 238 streams could create a substantial load on the server.  And  the higher the load, the more logs and the more encryption.

Errors were appearing in the logs related to syslog-ng. So I went looking for what the errors meant and how to make them go away.

Apr 17 22:56:58 logger.example.com syslog-ng[3199]: Number of allowed concu
rrent connections reached, rejecting connection; client='AF_INET(192.168.0.26:396
54)', local='AF_INET(0.0.0.0:6727)', max='10'

This error is from tcp. I google searched “increase tcp buffers kernel 2.6.18″ and was lead to this TCP Tuning Guide . Using these instructions I added the following lines to /etc/sysctl.conf

kernel.shmmax = 4294967295

kernel.shmall = 268435456

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1

net.core.netdev_max_backlog = 2500

Then ran sysctl -p and got a bunch of lines containing error: “Operation not permitted” still trying to figure out what this error is coming from.

So now all I see in the logs is things like:

Apr 20 19:14:00 logger.example.com syslog-ng[1792]: Syslog connection accepted; fd='478', client='AF_INET(192.168.0.26:50238)', local='AF_INET(0.0.0.0:6537)

To deal with the following error coming from syslog-ng I found I had to add a parameter to the config file.

Apr 18 02:39:01 logger.example.com -- MARK --
ERR(root): makeconnection: cannot create socket: No buffer space available
: fl=0x0, mode=20666: CHR: dev=0/57, ino=23909, nlink=1, u/gid=0/0, size=0
: fl=0x1, mode=20666: CHR: dev=0/57, ino=23909, nlink=1, u/gid=0/0, size=0
: fl=0x1, mode=20666: CHR: dev=0/57, ino=23909, nlink=1, u/gid=0/0, size=0
: fl=0x2, mode=100660: dev=0/50, ino=104136752, nlink=1, u/gid=51/51, size=965
: fl=0x0, mode=100660: dev=0/50, ino=104136748, nlink=1, u/gid=51/51, size=251
: fl=0x2, mode=140777: SOCK localhost->[[UNIX: /dev/log]]

I found the answer to this on The Rule of Tech web site. I simply changed the max-connections paramter in /opt/syslog-ng/etc/syslog-ng.conf from this:

unix-stream(“/dev/log”);

to this:

unix-stream(“/dev/log” max-connections(600));

I’m watching the logs now, to see if all the errors have been solved. I restarted the syslog-ng processes on the client side, to see if anything changes. After restarting syslog-ng on each of the syslog-ng client machines, I now see 238 connections from each. That’s 119 virtual hosts, with one access log stream and one error log stream. Those are the right numbers. We’ll see if this is the last of the errors.

Share