Custom OSX log file rotation / archiving

This is an updated/edited version of an article I posted years ago on World’s Worst Software.

Custom Log File Rotation In OSX

It’s not easy to figure out how to configure OSX’s internal periodic tasks. I googled around a bit for info on integrating your own log rotation scheme into OSX’s existing scheduled tasks, but I could not find any great documentation on the matter, so here’s how I solved the problem.

How OSX’s log rotation works

OSX (10.4+) automatically rotates your log files for you on a weekly basis. It first rotates older logs, then it gzips your current log file up, and then it creates a new log file for your system to continue logging to.

The log files are gzipped into a file like logfile.0.gz where “logfile” is the name of the logfile in question. For example my apache access log would be zipped into access_log.0.gz.

The log files are rotated by incrementing the number in the filename, so logfile.0.gz is renamed to logfile.1.gz before a new logfile.0.gz is made. This incrementing rotation continues for 4 or 5 steps. So eventually you are losing whatever is in logfile.5.gz because it won’t be renamed to logfile.6.gz.

OSX also DELETES your archived log files on a daily basis. The daily scheduled task looks for log files that are have not been modified in X many days and deletes them. This is problematic because you’re losing logs you may not want to lose.

For example, on my system I simply edited the weekly log rotation scheme to timestamp each new logfile.gz file into something like logfile.2007-11-15.gz so I wouldn’t overwrite older gzipped logs, but the daily deletion task started removing older gzipped logfiles.

Now that we understand how the log rotation is working, it’s time for us to dig in and figure out how we can hook into OSX’s scheduled tasks and get our own log rotation scheme going.

Integrating with OSX’s scheduled tasks

OSX (10.4+) uses its proprietary launchd and launchctl programs to run scheduled tasks at selected times.

When your system boots up launchd reads through the files in your /System/Library/LaunchDaemons directory and discovers which tasks to run at boot and which tasks to run at certain times.

We could build our own something.plist file that’d call our own log rotation script and put that file in /System/Library/LaunchDaemons, but for my purposes I wanted to make sure my log rotation script was run as a part of the OS X weekly system task that was already doing log rotation.

OSX’s weekly system task is specified in the following file:

/System/Library/LaunchDaemons/com.apple.periodic-weekly.plist

That file looks like this:

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"  
  3.    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  4. <plist version="1.0">
  5. <dict>
  6.         <key>Label</key>
  7.         <string>com.apple.periodic-weekly</string>
  8.         <key>ProgramArguments</key>
  9.         <array>
  10.                 <string>/usr/sbin/periodic</string>
  11.                 <string>weekly</string>
  12.         </array>
  13.         <key>LowPriorityIO</key>
  14.         <true/>
  15.         <key>Nice</key>
  16.         <integer>1</integer>
  17.         <key>StartCalendarInterval</key>
  18.         <dict>
  19.                 <key>Hour</key>
  20.                 <integer>3</integer>
  21.                 <key>Minute</key>
  22.                 <integer>15</integer>
  23.                 <key>Weekday</key>
  24.                 <integer>6</integer>
  25.         </dict>
  26. </dict>
  27. </plist>

The file basically dictates that launchd should run the command /usr/sbin/periodic weekly at 3:15 in the morning on the 6th day of the week.

So, the next question is, what’s the periodic program? Well, if we run man periodic in the terminal, we see that there are scripts in the /etc/periodic directory that are run by the program depending on the argument passed in.

There are several directories under /etc/periodic such as /etc/periodic/daily and /etc/periodic/weekly. When we run periodic weekly the periodic program looks in the /etc/periodic/weekly directory and executes any scripts it finds there.

On my system, the only file in the weekly directory is a file called 500.weekly. That file contains a few tasks that are run as part of the weekly task, including the log rotation.

WARNING: DO NOT EDIT YOUR /etc/periodic/weekly/500.weekly FILE! YOUR CHANGES WILL NOT STAY!

Initially I edited my 500.weekly file, removing the OS X default log rotation logic and replacing it with a callout to a log rotation script of my own. Testing the change worked just fine, but OSX eventually regenerated the 500.weekly file and my changes were lost.

You should not edit the 500.weekly file, but instead should edit the file mentioned below.

There is an interesting section near the bottom of the 500.weekly file:

  1. if [ -f /etc/weekly.local ]; then
  2.     echo ""
  3.     echo "Running weekly.local:"
  4.     sh /etc/weekly.local
  5. fi

That code says to run the /etc/weekly.local file if that file exists. That file did not exist on my OSX system, so I created it and edited it to call my own script like so:

  1. #!/bin/bash
  2. /scripts/osxrotatelogs.sh /var/log /archive/logs/locallogs/

Now OSX is going to run it’s internal log rotation scheme every week and gzip each of my log files to something like logfile.0.gz, and then it will run /archive/scripts/osxrotatelogs.sh before the weekly task ends.

Using osxrotatelogs.sh on your own machine

For my log rotation scheme I wanted to do two things:

#1: Timestamp the log file and gzip the file into something like logfile.2007-11-15 and logfile.2007-11-15.gz.

This way my log files have easily sortable information in my filename, and I can unzip a bunch of the logs into one directory without worrying about overwriting previously unzipped logs.

#2: Copy the timestamped gzip file to a directory where OSX’s daily scheduled task wouldn’t delete old log files.

I wrote the osxrotatelogs.sh script to solve both of those problems.

You can use the script on your own osx machine if you’d like. Simply download the osxrotatelogs.sh file to some place on your computer, then execute chmod +x osxrotatelogs.sh in your terminal to make the script executable. Then edit your /etc/weekly.local file to call the osxrotatelogs.sh script.

Download osxrotatelogs.sh from github.