Raspberry Pi as a Splunk Universal Forwarder to Store-and-Foward Logs

I am a fan of Splunk, so I run it at home. The problem is, I don’t want it running all the time, but I always want to collect logs.

Why don’t I just leave Splunk running? I run it as a virtual machine and it consumes memory and cpu, which I often need for other VMs. Also, the system that hosts my VMs consumes a fair amount of power, and as the owner of a partially solar-powered home, I try to conserve power consumption. So I turn that system off when I’m not using it.

Therefore, I need a low power way to continuously gather logs from any systems that are running, store the logs, and forward them to splunk when I choose to run it.

The Raspberry Pi is perfect for this sort of store-and-forward scenario. The architecture is pretty simple. I run syslog-ng on the pi to collect syslogs from all the systems and save them to disk. The pi also runs the Splunk Universal Forwarder to forward those logs to Splunk when Splunk is running. The Universal Forwarder keeps track of what it has forwarded and what it hasn’t, and only forwards what it needs. So, I can leave the pi up, running (but not consuming much power), and gathering logs, and only fire up my Splunk system when I need to. When I run Splunk, the Universal Forwarder sends along all the logs it gathered while Splunk was powered off and continues to forward logs until I power Splunk off again.

One of the key things I learned early on is the importance of keeping the logs on a separate drive so if they fill the drive, they don’t consume the operating system’s disk (which will crash the pi). I use a USB flash drive, and moved the full /var/log directory to it because any one of several logs could fill the drive.

Here is the overall process:

Build the core splunk system (the indexer & search head) and confirm it works ok. I’m not going to cover that here.
Set your core splunk system to receive traffic from forwarders
Install Raspbian on the Raspberry Pi (from raspberrypi.org)
Using raspi-config, set the disk to use the full SD card, set the hostname, and set the timezone
Set your router to give a fixed ip to the raspberry pi.
Set up a USB stick to be the /var/log directory
1. format the USB stick to ext3
2. move the /var/log dir to the USB stick (instructions here are for moving /var, but the procedure is the same).
  Note, in the step to edit the fstab file, I used the following:
  UUID=uuid /var/log ext3 defaults 0 1
  (use blkid to determine the UUID)
Install syslog-ng (sudo apt-get install syslog-ng)
Configure a system to send syslogs to this system using udp port 514.
Configure syslog-ng
1. edit syslog-ng.conf (sudo vi /etc/syslog-ng/syslog-ng.conf)
2. add the following lines to the appropriate sections to set syslog-ng to listen for syslogs on udp port 514 and save them to /var/log/udp514.log (or whatever you want to call your log file. syslog-ng can do a lot more if you wish, including create unique log files for every log that comes in)
```
 # source for syslog 514 traffic
 source s_udp514 { udp(port(514)); };
 # destination for udp 514 syslogs
 destination d_udp514 { file("/var/log/udp514.log"); };
 # All udp514 logs
 log { source(s_udp514); destination(d_udp514); };
```
3. restart syslog-ng (sudo /etc/init.d/syslog-ng restart)
4. watch that log to see it is getting data (tail -f /var/log/udp514.log)
Install the Splunk Universal Forwarder
1. download the forwarder
2. install the forwarder (sudo tar xvzf forwarder-for-linux-arm-raspberry-pi_10.tgz -C /opt)
3. configure splunk to run with user id splunk & start splunk
  sudo useradd splunk
  sudo groupadd splunk (the group may be created already)
  sudo chown splunk:splunk /opt/splunkforwarder/
  sudo -H -u splunk /opt/splunkforwarder/bin/splunk start
4. configure to run at boot
  sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
5. make sure any logs you wish to forward are readable by splunk. Since logs are typically read-writefor the owner and read-only for the group, you can change the group. You may choose to do a single file, eg
  sudo chgrp splunk /var/log/udp514.log
  or all the log files, eg
  sudo -R chgrp splunk /var/log/

Reboot your pi and confirm splunk is running using the right id (splunk)
sudo reboot -r now
ps -ef | grep splunk
You should get a result similar to this, showing that splunk, not root, is running splunk (the first column is the user):

splunk    2188     1 24 00:21 ?        00:00:08 splunkd -p 8089 start
splunk    2189  2188  0 00:21 ?        00:00:00 [splunkd pid=2188] splunkd -p 8089 start [process-runner]
pi        2262  2247  0 00:22 pts/0    00:00:00 grep --color=auto splunk

Configure the Universal Forwarder
1. since splunk now runs as the splunk id, change to that id and change to the splunk directory
  sudo su – splunk
  cd /opt/splunkforwarder/bin
2. set the admin password to something unique (the default is “changeme”)
  ./splunk edit user admin -password <new password> -role admin -auth admin:changeme
3. set the forwarder to forward (use your new password)
  ./splunk add forward-server <host>:<port> -auth admin:<password>
4. set what to monitor and forward
  ./splunk add monitor /var/log/
Log into your Splunk instance and check out your logs!
1. If you have problems, check out this troubleshooting page: Troubleshooting Forwarding
2. Validate the approach works by shutting down your core Splunk. Notice that the next time you power it up, after a little delay logs will start filling in from during your outage.
Configure your logs to rotate
1. Edit /etc/logrotate.conf and add the following (this will rotate when the size hits 1G, the new log file it creates will be owned by splunk/splunk with 740 permissions, and we’ll keep up to 10 files)
```
/var/log/udp514.log {
    size 1g 
    create 740 splunk splunk
    rotate 10
}
```
2. Note that Splunk won’t read the rotated logs, so it probably makes sense to zip them and keep fewer copies.

Since first writing this, I found these excellent instructions, complete with screenshots, for setting up a universal forwarder. They are great, although they don’t have the Raspberry Pi specific details, or how to run splunk as another user.

Leave a comment Cancel reply