Backup logs to AWS S3 using logrotate and awscli

Photo Credit: datatrend.com

Backup logs to AWS S3 using logrotate and awscli

Panggi Libersa J.A bio photo By Panggi Libersa J.A Comment

Motivation

As developer, logging feature is super useful for debugging and in production it is also can be used for tracking users and security mitigation as well.

But the problem comes when the size of log is increasing and make your disk full that will affect the performance of your web applications.

In this article i will share the tips using two apps:

  • logrotate: designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files.
  • awscli: Universal Command Line Interface for Amazon Web Services.

Installation

awscli

Prerequisites: Python 2.6.3 or later

$ curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
$ sudo python get-pip.py
$ sudo pip install awscli

logrotate

logrotate is preinstalled with major GNU/Linux distribution, so i assume your production server might have one installed already.

Configuration

First thing first, lets make sure that our awscli can upload files to AWS S3.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: ap-southeast-1
Default output format [None]: json

Lets try to upload some files recursively using cp command:

$ aws s3 cp myfolder s3://mybucket/myfolder --recursive 
upload: myfolder/file1.txt to s3://mybucket/myfolder/file1.txt 
upload: myfolder/subfolder/file1.txt to s3://mybucket/myfolder/subfolder/file1.txt 
...

Ok, it works, now let’s continue with logrotate. There’s a nice tutorial by Rackspace and i recommend you to see it.

I’m going to use nginx logrotate sample generated by nginx during installation with package manager:

$ vim /etc/logrotate.d/nginx

/var/log/nginx/*log {
    create 0644 nginx nginx
    daily
    rotate 1
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
        /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
    endscript
}

So, this configuration will run daily, will keep 1 log and remove the older one, compress the nonactive logs and will kill nginx after log rotation done.

Now, how to combine the two? you can run Cron to upload to s3 daily or put your awscli command to postrotate.

Conclusion

So you got the idea why it is great to combine logrotate and awscli, you will save disk space at your production server and your AWS S3 volumes by not uploading the same log files again and again.

Tips

  • For debugging purpose, you check logrotate.status file to see which files rotated: cat /var/lib/logrotate.status and run logrotate in a debug mode to see the problem if your logrotate config is not running: /usr/sbin/logrotate -d /etc/logrotate.conf
  • Create folders with timestamp to make it easy to search, the command is like this: $ aws s3 cp /var/log s3://mybucket/web-server-01/$(date +%Y%m%d)/varlog/$(date +%Y%m%d%H%M%S) –recursive