Log Management can be tough

For the umpteenth time one of our squid boxes went down due to the logs filling up all available disk space. As we have 3 sites, plus a special client access network, we have FOUR squid boxes that have been problematic for a long time.

The first major problem we had (it just seemed way too slow) was identified after trawling the logs to find the problem was always there, we need to fixup the number of available file descriptors.

grep WARNING /var/squid/logs/cache.log
YYYY/MM/DD HH:MM:SS| WARNING: Your cache is running out of filedescriptors
In this particular case, we had a lot of error messsages with the same text "Your cache is running out of filedescriptors." How many file descriptors do I have for my cashing service/daemon? Using your shell, you can determine the number of file descriptors by something like the below:
  • change the default shell to something that will give you a shell (my default shell for the _squid account is /sbin/nologin
  • use ulimit to display the process limits
This will show you what your login class settings give to your squid daemon.
# usermod -s /bin/ksh _squid
# su _squid
$ ulimit -a
time(cpu-seconds)     unlimited
file(blocks)          unlimited
coredump(blocks)      unlimited
data(kbytes)          2097152
stack(kbytes)         8192
lockedmem(kbytes)     296673
memory(kbytes)        887008
nofiles(descripters)  128
processes             1310
$ exit
# usermod -s /sbin/nologin _squid
More simply
$ ulimit -n
128
As in the earlier example, you can also use *squidclient* to view what the running *squid* process sees as available file descriptors. This view will also let you watch over time as your file descriptors deplete etc.
# squidclient mgr:info | grep -i file
File descriptor usage for squid:
       Maximum number of file descriptors:
       Largest file desc currently in use:
       Number of file desc currently in use:
       Files queued for open:
       Available number of file descriptors:
       Reserved number of file descriptors:
       Store Disk files open:
The solution? Increase the number of file descriptors available to your daemon and a clean way of doing this is the login class as described here

But where is the green sheep? What is there to fix our current persistent log over grown problems?

Insanity

I just love that saying oft attributed to Einstein defining Insanity.

Doing the same thing over and over, expecting a different result.

It's crazy that part of our solution was to just delete the old logs, and restart squid service.

Method replacing the madness

The problem we have with our Squid Caching Proxies, which is quite different from all our other services, is that we just don't have much disk space (strange how that should make one think more.)

We can't just replicate our log management strategy used on other boxes. The Squid access logs really do grow quickly on these boxes. It isn't good enough to collect a year's worth of logs, rotating them at the end of each month.

Squid's default log rotation process, just 'moves' the file, without compression. So, last month's 8GB log file, is 8GB permanently lost on the HDD. We need to compress those old log files (do we really need them?)

Looking at our log analysis ... we don't need to keep that length of archiving on disk.

Moving on?

[Ref: newsyslog(8)]

There are various ways of using newsyslog(8) to administer your squid logs, one being below

File extract: /etc/newsyslog.conf

#log file name              owner:group  mode count size when  flags options
/var/squid/logs/access.log _squid:_squid 640   12    *   $M1D0 ZB    /var/squid/logs/squid.pid SIGUSR1
/var/squid/logs/cache.log  _squid:_squid 640   12    *   $M1D0 ZB    /var/squid/logs/squid.pid SIGUSR1
/var/squid/logs/store.log  _squid:_squid 640   12    *   $M1D0 ZB    /var/squid/logs/squid.pid SIGUSR1

Confirm the location of the log files and squid.pid (the above works for some of my hosts.)

The above configuration works on keeping 12 (count) previous logs of unlimited (* size) and runs the archive/compress on the 1st day of the month ($M1D0) This strategy works for our servers with more disk space, but is going to be problematic for some of us with smaller disks.

Note: squid 3.1 and later(?) changed default creation for cache_store_log OFF.

flags
Your strategy should consider the tools you use for analysing the log files. If you can get your logs off the server on a daily basis, then you can probably go for a more frequent 'when' such as $D0 daily at midnight, same as @T00.

Examples from the manpage newsyslog(8) include:

  • $D23 every day at 23:00 pm
  • $W0D01 weekly on Sunday at 01:00 am
Options

PID and SIGUSR1

Newsyslog will send SIGUSR1 to squid, which in this case instructs squid to close, reopen the log file.

squid.conf - logfile_rotate

With the above configuration, we need to tell squid that we're using newsyslog using the $!bing("logfile\_rotate","squid.conf logfile_rotate to 0 newsyslog") value of 0

File extract: /etc/squid/squid.conf

logfile_rotate 0
Creative Commons License © 2010-2014 Samiuela LV Taufa.
This work is licensed under a Creative Commons Attribution 3.0 Unported License.