Aaron's Blog

Tech, tinkering, and occasionally, a banjo tune

Monitoring My Kegerator With Sensu

It's been a long time coming, but I've finally got the KegPi mostly working and being monitored with Sensu! This post is going to go over the whole kit and caboodle and is going to build on the posts of creating the kegerator and getting Sensu to run on a Raspberry Pi. The deep and dark part of this post will cover:

  • The code to measure the temp/contact sensors (Sensu Plugin)
  • Writing Sensu checks that will incorporate that code
  • Exporting the data from the temp sensor to Graphite and InfluxDB for seeing temperature trends

Writing the Plugins

If you haven't had a chance to read the previous post on using Sensu to monitor Raspberry Pis, I'll give you a short blurb about Sensu. It's an open-source monitoring framework that allows you to monitor everything from fleets of servers and networking equipment to applications, and even yes, Raspberry Pis. It also has a highly decoupled architecture and relies on plugins to extend its functionality.

Currently there aren't any plugins inside of the Sensu ecosystem that exist for monitoring components that would be hooked up to a Raspberry Pi. So, I decided to write a few!

WARNING: I don't write a whole lot of Ruby professionally, and it's likely pretty UGLY code, but what the hell. It works, so let's just go with it! Also, your mileage may vary if you decide to put something like this together and use my code.

To get started writing a plugin, Sensu provides some skeletons for writing code in various languages:

Before I'd started working at Sensu, the KegPi project was something that I tried to do from the ground up. I realized that the monitoring portion wasn't something that I wanted to write from scratch and tools (like Sensu) already existed, hence why I wrote the plugin for it.

One quick note--by design, this plugin and the checks are designed to be fairly generic. What that means practically is that even though I'm using this primarily for a Raspberry Pi hooked up to a kegerator, you can use these checks in any sort of context. Have a Raspi that's monitoring the temp of a chicken coop? You can use this plugin. Want to build security system for your kid's school locker and use Sensu to handle the alerts? You can use this plugin to do it!

So let's take a look at a couple of the checks that are part of the plugin.

Status Checks

Contact Sensor Check

The contact sensor check is, again, designed to be generic and able to be used in a variety of circumstances. Here's the code:

require 'sensu-plugin/check/cli'
require 'rpi_gpio'

# Starting check class
class CheckContactSensor < Sensu::Plugin::Check::CLI
  option  :pinnum,
          short: '-p PINNUM',
          long: '--pin-num PINNUM',
          description: 'Sets the pin number the contact sensor is attached to',
          proc: proc(&:to_i),
          required: true

  option  :boardnum,
          short: '-b BOARDNUM',
          long: '--board-num BOARDNUM',
          description: 'Sets the board numbering type to either :bcm or :board',
          default: :bcm

  def initialize
    super
    RPi::GPIO.set_numbering config[:boardnum]
    RPi::GPIO.setup config[:pinnum], as: 'input', pull: 'up'
  end

  def run
    if RPi::GPIO.high? config[:pinnum]
      puts 'Contact sensor is open!'
      exit(2)
    else
      puts 'Contact sensor is closed.'
      exit(0)
    end
  end
end

We've got an option to set our pin and our board numbering options (those coming from Clockvapor's ruby port of rpi.gpio). So when we write our check (see below), we can add these options to the check.

Temp Sensor Check

Working with the temp in Python examples was a bit of a challenge and much of that code I was able to translate into Ruby for the purpose of this particular check. Let's take a look:

require 'sensu-plugin/check/cli'
require 'rpi_gpio'

# Starting check class
class CheckTempSensor < Sensu::Plugin::Check::CLI
  option  :fahrenheit,
          short: '-F',
          long: '--fahrenheit',
          description: 'Return temperature in Fahrenheit'

  option  :celsius,
          short: '-C',
          long: '--celsius',
          description: 'Return temperature in Celsius'

  option  :tcrit,
          short: '-c TEMP',
          long: '--critical',
          proc: proc(&:to_i),
          description: 'Critical if TEMP greater than set value'

  option  :twarn,
          short: '-w TEMP',
          long: '--warn',
          proc: proc(&:to_i),
          description: 'Warning if TEMP greater than set value'

  # Set up variables
  def initialize
    super
    system('modprobe w1-gpio')
    system('modprobe w1-therm')
    @basedir = '/sys/bus/w1/devices/'.freeze
    @device_folder = Dir.glob(@basedir + '28*')[0]
    @device_file = @device_folder + '/w1_slave'
  end

  def read_temp
    lines = File.read(@device_file)
    while lines[0][-4..-2] != 'YES'
      sleep 0.2
      lines = File.readlines(@device_file)
    end

    equals_pos = lines[1].index('t=')
    return if equals_pos == -1

    lines[1][equals_pos + 2..-1].chomp.to_f / 1000.0
  end

  def temp_to_fahrenheit
    read_temp * 9.0 / 5.0 + 32.0
  end

  def temp_status
    critmsg = 'Temp is critical'
    warnmsg = 'Temp is abnormal'
    okmsg = 'Temp is OK'
    if config[:celsius] && read_temp > config[:tcrit]
      puts critmsg
      exit(2)
    elsif config[:celsius] && read_temp.between?(config[:tcrit], config[:twarn])
      puts warnmsg
      exit(1)
    elsif config[:fahrenheit] && temp_to_fahrenheit > config[:tcrit]
      puts critmsg
      exit(2)
    elsif config[:fahrenheit] && temp_to_fahrenheit.between?(config[:tcrit], config[:twarn])
      puts warnmsg
      exit(1)
    else
      puts okmsg
      exit(0)
    end
  end

  def run
    if config[:fahrenheit]
      puts 'Current Temp: ' + temp_to_fahrenheit.round(2).to_s + ' Fahrenheit'
    else
      puts 'Current Temp: ' + read_temp.round(2).to_s + ' Celsius'
    end
    temp_status
  end
end

Yikes! Yeah, that's a lot, but it boils down to a few key options you can include in a check:

  • Celsius
  • Fahrenheit
  • Warning temp
  • Critical temp

Again, this check can be super-generic, and it's got the option for Celsius, so what's not to love?

Alright, so one last check to take a look at as an example...metrics!

Temp Sensor Metric Check

So this particular check is SUPER useful if you want to see your temp trends over time. It looks like this:

  # Set up variables
  def initialize
    super
    system('modprobe w1-gpio')
    system('modprobe w1-therm')
    @basedir = '/sys/bus/w1/devices/'.freeze
    @device_folder = Dir.glob(@basedir + '28*')[0]
    @device_file = @device_folder + '/w1_slave'
  end

  def read_temp
    lines = File.read(@device_file)
    while lines[0][-4..-2] != 'YES'
      sleep 0.2
      lines = File.readlines(@device_file)
    end

    equals_pos = lines[1].index('t=')
    return if equals_pos == -1

    lines[1][equals_pos + 2..-1].chomp.to_f / 1000.0
  end

  def temp_to_fahrenheit
    read_temp * 9.0 / 5.0 + 32.0
  end

  def run
    timestamp = Time.now.to_i

    if config[:celsius]
      ok output "#{config[:scheme]}", read_temp.round(2)
    else
      ok output "#{config[:scheme]}", temp_to_fahrenheit.round(2)
    end
  end
end

When we write a metric check in Sensu, this won't be a "status" type check (think "up" or "down", or "OH CRAP THIS THRESHHOLD'S BEEN HIT"), but more of a "here's the data" type check. Cool, so now that that's all out of the way, let's talk about the cool part, writing our checks.

Writing the Checks

Sensu's check definitions are all written in JSON (trust me, I know the pains of trailing or missing commas). So when we write a check for the KegPi, we'll have to write this all in JSON and it's SUPER CRAZY IMPORTANT that you have the check scripts living on your Pi. To do that, just do the following:

git clone https://github.com/asachs01/sensu-plugins-rpi-sensors.git

Cool, done? Awesome.

So let's take a look at three checks that correspond to the scripts I shared:

check-temp-sensor

{
	"checks": {
		"check-temp-sensor": {
			"command": "/home/pi/Documents/sensu-plugins-rpi-sensors/bin/check-temp-sensor.rb -F -w 45 -c 60",
			"interval": 15,
			"subscribers": ["kegpi"],
			"handlers": ["email", "slack"],
			"occurrences": 3,
			"refresh": 600
		}
	}
}

check-contact-sensor

{
	"checks": {
		"check-contact-sensor": {
			"command": "sudo /home/pi/Documents/scratchpad_kegpi/bin/check-contact-sensor.rb -p 22",
			"interval": 10,
			"occurrences": 8,
			"refresh": 120,
			"subscribers": ["kegpi"],
			"contacts": ["aaron"],
			"handlers": ["slack"]
		}
	}
}

metrics-temp-sensor

{
	"checks": {
		"metrics-temp-sensor": {
			"command": "/home/pi/Documents/sensu-plugins-rpi-sensors/bin/metrics-temp-sensor.rb -F",
			"type": "metric",
			"interval": 15,
			"metrics-temp": "<URL TO GRAPHITE GRAPH>",
			"subscribers": ["kegpi"],
			"handlers": ["graphite"]
		}
	}
}

Those are what the checks look like on a practical level. You'll notice the contact attribute is present in these checks. That's because I'm running a Sensu Enterprise deployment and taking advantage of contact routing...even though I'm the only one monitoring the kegerator.

So you can get a good idea of what these checks might look like in a dashboard (Uchiwa, or Sensu Enterpise), let's take a look at the checks as they're currently running:

temp-sensor-1

contact-sensor-1

metric-temp-1

So there you have it--some working checks that are at this point in time, checking the contact and temperature sensors attached ot the Pi.

Sending the Data to Other Sources

Before I wrap this up, let's do a quick discussion about how you can send the data to other sources and get some cool graphs out of the deal. Personally, I have the data being sent to two sources: an InfluxDB instance and a Graphite instance. The reason for exporting to two data sources simply comes down to the fact that I can perform a quick bit of magic in my check and embed my Graphite graph into my check for a quick visualization of the temperature. Also, I much prefer the graphs I can get out of Influx, but alas, those aren't embeddable.

influx-kegpi

So how does the data make it to the Graphite and InfluxDB instances? Enter our handlers. You'll notice in the checks defined above, I have my influxdb and graphite handlers specified. Just like everything else in Sensu, this is done via JSON configuration files:

Our Graphite configuration;
/etc/sensu/conf.d/graphite.json

{
  "handlers": {
    "graphite": {
      "type": "tcp",
      "mutator": "only_check_output",
      "timeout": 30,
      "socket": {
        "host": "graphite.company.tld",
        "port": 2003
      }
    }
  }
}

Our InfluxDB configuration (both the Influx configuration and requisite handler):

/etc/sensu/conf.d/influxdb.json

{
  "influxdb": {
      "host": "192.168.1.5",
      "port": "8086",
      "database": "sensu",
      "username": "user",
      "password": "password"
  }
}

/etc/sensu/conf.d/influx-tcp.json

{
  "handlers": {
     "influx-tcp": {
       "type": "pipe",
       "command": "/opt/sensu/embedded/bin/metrics-influxdb.rb"
     }
   }
}

With those files added, we're able to then send the check data from the KegPi to Graphite and InfluxDB respectively.

Wrapping Up

Whew! This post has been a bit of a long one, and I thank you for sticking through the read until this point. If you have any questions about any aspect of this, don't hesitate to leave a comment down below.

Thanks for reading!

How I Homebrew

I've met quite a few homebrewers over the years and have learned tons from them. One of the things I always like asking about it their setup and getting to know what they use on brew day.

I figured it's time for me to do a HIB (How I Homebrew) post and let you know what I started brewing on, and where I've progressed.

For Starters

When I first started brewing, I started on a hand-me-down Mr. Beer kit that had solidified LME with it. I went to Walmart and picked up a cheap 3 gallon aluminum kettle, and brewed my first beer (an ESB).

It wasn't long before I realized that I wantd to brew larger batches and to do that, I'd need to upgrade. I upgraded my fermentation vessel (the Mr. Beer LBK) first. This was great, but I was essentially making extract in my undersized brew kettle, which I upgraded next. This began a long process of creating the piecemeal setup that I currently have.

Currently Using...

This is the fun part for me. I'll skip over all the sequential upgrades, and get to the good stuff. Here's what I'm currently using, and am quite happy with:

  • 2 x 8gal Tamale Steamers (boil kettle & HLT)
  • Igloo mash tun
  • Some copper tubing from Home Depot (immersion chiller)
  • 2x PVC buckets (primary fermenters)
  • 2x 6gal glass carboys (secondary fermenters)
  • Bayou Classic w/ 6" burner

But wait! There's more!

I've got some incoming upgrades, namely a 10" burner for the Bayou Classic stand, and a Jaded Brewing Hydra! I'm hoping these will shorten my brew day. Not that I want them to be short, but I'm hoping for them to be a bit more efficient. Anywho, that's the current setup!

Monitoring Raspberry Pi's with Sensu

Back in August, I took a Customer Success role at Sensu. I'd been familiar with Sensu through my various bits of tinkering and dabbling with monitoring in the past, and was happy to take the role and work with the Sensu team officially.

Now that I've been working with Sensu for a few months (and am officially on paternity leave for the next several weeks), I decided it was time to revisit the idea of using Sensu to monitor my Kegpi's various sensors. I'd hit some blocks previously, but came back to it with some fresh eyes. Lo and behold, I've got it working! Let's walk through how I've got it running.

The Setup

Here are the prerequisites for getting Sensu to work on your Raspberry Pi:

  • Ruby > 2.0
  • Rubygems
    sudo apt-get install ruby ruby-dev (this should install rubygems by default)
  • Sensu gem
    sudo gem install sensu
  • A working Sensu server (you can snag one quick by downloading Vagrant, cloning my "Sensu Up and Running" repo, and doing a vagrant up)

Configuration

Before you get started, ensure you've added the sensu user:

  • useradd sensu

Sensu

By default, the Sensu gem installs three services: sensu-client, sensu-server, and sensu-api. These form the core components of any Sensu installation, but we're ultimately about getting a client up and running on the Raspi. Before we go too much futher into configuring Sensu, I recommend that if you're not familiar with Sensu as a monitoring framework, you should take a moment and watch some of Sensu's free training videos.

Presuming you've already installed the gem, the commands to manage the Sensu client are located in /usr/local/bin. The configuration files, however, are located in /etc/sensu/conf.d. You'll likely have to create that directory:

sudo mkdir -p /etc/sensu/conf.d

From there, we'll add three different configuration files, as per the Sensu Documentation:

  • client.json
  • rabbitmq.json

Here's what they'll look like:

client.json:

{
 "client": {
   "environment": "development",
   "subscriptions": [
     "dev"
   ]
 }
}

rabbitmq.json

{                    
  "rabbitmq": {      
    "host": "127.0.0.1",                   
    "port": 5672,    
    "vhost": "/sensu",                     
    "user": "sensu", 
    "password": "secret",                  
    "heartbeat": 30, 
    "prefetch": 50   
  }                  
} 

Finish off by ensuring the permissions for all the files in /etc/sensu/ are owned by sensu:

  • chown sensu. /etc/sensu/

NOTE: The credentials listed above should be changed. They're merely examples pulled straight from the Sensu documentation and should NOT be used as is on any system you care about. You'll also need to chage the IP addresses for Rabbit to the IP of the system(s) you have those running on. If you're using the Sensu Up and Running Vagrant box, the IP addresses will be the same.

Systemd

If you're used to a system using systemd for service management, you'll be familiar with the service files located in /etc/systemd/system. Sensu includes service files for each of the components, but for them to work on a Raspberry Pi, they need to be slightly modified. You'll need to create the service file like so:

sudo vim /etc/systemd/system/sensu-client.service

With the following content:

[Unit]               
Description=sensu client                   

[Service]            
User=sensu           
Group=sensu          
ExecStart=/usr/local/bin/sensu-client                                    
KillMode=process     
Restart=on-failure   
RestartSec=1min      

[Install]            
WantedBy=multi-user.target

From there, start the service:
sudo systemctl start sensu-client

And set it to start at boot:
sudo systemctl enable sensu-client

Which, if you're using the Sensu Up and Running repo I mentioned earlier, should get you something that looks like this:

kegpi-sensu

Next Steps

So what do you do after getting Sensu working on your Raspi? Well, it's up to you! In my case, I'll use it to monitor the temperature, whether or not the door is open, and remaining liquid in the kegs for my kegerator.

In your case, you may want to monitor some system-level activity, like load, available RAM, or something of the like. You could also use it to say, monitor a web app you're running on the Raspi, or, since Sensu's plugins can be written in any programming language, use it to monitor any number of sensors you can attach to the Pi. Again, if you haven't already, take a look over Sensu's training videos to get a better idea of how Sensu works and what you can do with it.

Moving to Kegging my Homebrew

I've posted previously about homebrewing and have been brewing for almost three years now. Back in April, I took to destroying a perfectly good mini fridge to turn it into a kegerator.

IMG_20170403_105320

As any homebrewer can tell you, bottling is a pain in the ass. The delabling (if you're reusing bottles), the santization, and the bottle carbonization makes for a rather long and laborious process. Because I'd finally had it with bottling, I decided that kegging would be a wiser choice since:

  1. Less time would be spent on bottling
  2. Carbonization would be better controlled
  3. For most ales, production time would be cut by a week

Thankfully, my assumptions have proven to be true. So let's walk through what it took to make the kegerator.

The Build

I started off with the following hardware:

I won't go through the whole process of assembling, since the post over at missionarybrewer (which my build was largely based on) covers the steps more than sufficiently. I will mention a few gotchas for the particular fridge I used:

  • There are coolant lines running over in the upper rear left corner. DON'T drill back there
  • The 4.3 cu ft size is just a hair too small for both kegs to fit comfortably. I ended up using a multitool to carve out the front of the door, as well as the molding for holding the shelves so that the kegs would fit.

Now, some pics of the kegerator:

IMG_20170403_162917

IMG_20170403_193658

IMG_20170403_193702

IMG_20170403_193658-1

IMG_20170403_193706

IMG_20170403_193711

IMG_20170403_231824

If you've got any questions, feel free to reach out and let me know!

Moving (Back) To Linux

I'm a bit of a productivity nerd, I'll admit. Over the last couple of years, I've cultivated a workflow that revolved around a lot of OS X tools that I LOVED:

  • Alfred
  • TextExpander
  • iTerm2

These tools were my bread and butter. My professional life, if you will. However, I found myself in a bit of an odd spot in recent months--my MacBook Air was proving to be underpowered for running VM's, and doing any sort of virtualized anything (which I wanted to be able to do for some training and other projects). While I wanted a shiny new MacBook Pro, I found it to be overpriced, and something that I couldn't justify spending the kind of money on, despite that all of my tools were built exclusively for the MBP platform.

So I decided to move back to Linux. At Rackspace, I'd had an Optiplex workstation that ran Fedora, and it worked OK, if not for the fact that every update screwed up my display drivers. I honestly wasn't very keen to be back on Linux. The last thing I wanted to do was be in a situation where I'd have to cobble some sort of solution together, and it would require more time and effort that I have at the moment.

Serendipitously, one of my former coworkers and her husband turned me onto Korora. To be fair, there's not a whole lot more that Korora has over Fedora. Thus far, it's proven to have some prettier defaults. Not much more than what Fedora would have, but it beats taking the time out of my day to hunt around and change things.

To my delight, the experience has been surprisingly Mac-like (even to the point of competing with ElementaryOS). To my further delight, it has dark themes out of the box (swoon), which makes working at night much less stressful on the eyes.

The biggest challenge in moving back to a Linux distro as a daily driver has been finding a replacement for Alfred & TextExpander. The terminal challenge was easily solved with either terminator or tmux (now being a bit wiser, I'm using tmux with GREAT success). However, finding something that had similar functionality to TextExpander and Alfred was a bit difficult at the onset.

I'd tried Albert, an Alfred-like replacement for Linux, but to no avail. While Albert's got a decent start, it lacks quite a bit of Alfred's functionality. For example, Alfred has built-in clipboard management (now using Clipit), built-in snippets (which Autokey seems to do a decent job of), and workflows, which I haven't quite figured out a replacement for and probably won't at this point since I wasn't using them all that much anyways.

At the time of writing this post, I'm back up to speed in terms of where I was with my Mac. I've managed to recreate my level of productivity and automation with the following:

  • Alfred --> Korora menu button/Clipit
  • TextExpander --> Autokey
  • iTerm2 --> Terminator/tmux

Next on my productivity list is tackling my dotfiles. More to follow!

Cheers,

Aaron