Aaron's Blog

Tech, tinkering, and occasionally, a banjo tune

Monitoring Raspberry Pi's with Sensu

Back in August, I took a Customer Success role at Sensu. I'd been familiar with Sensu through my various bits of tinkering and dabbling with monitoring in the past, and was happy to take the role and work with the Sensu team officially.

Now that I've been working with Sensu for a few months (and am officially on paternity leave for the next several weeks), I decided it was time to revisit the idea of using Sensu to monitor my Kegpi's various sensors. I'd hit some blocks previously, but came back to it with some fresh eyes. Lo and behold, I've got it working! Let's walk through how I've got it running.

The Setup

Here are the prerequisites for getting Sensu to work on your Raspberry Pi:

  • Ruby > 2.0
  • Rubygems
    sudo apt-get install ruby (this should install rubygems by default)
  • Sensu gem
    sudo gem install sensu
  • A working Sensu server (you can snag one quick by downloading Vagrant, cloning my "Sensu Up and Running" repo, and doing a vagrant up)

Configuration

Before you get started, ensure you've added the sensu user:

  • useradd sensu

Sensu

By default, the Sensu gem installs three services: sensu-client, sensu-server, and sensu-api. These form the core components of any Sensu installation, but we're ultimately about getting a client up and running on the Raspi. Before we go too much futher into configuring Sensu, I recommend that if you're not familiar with Sensu as a monitoring framework, you should take a moment and watch some of Sensu's free training videos.

Presuming you've already installed the gem, the commands to manage the Sensu client are located in /usr/local/bin. The configuration files, however, are located in /etc/sensu/conf.d. You'll likely have to create that directory:

sudo mkdir -p /etc/sensu/conf.d

From there, we'll add three different configuration files, as per the Sensu Documentation:

  • client.json
  • rabbitmq.json

Here's what they'll look like:

client.json:

{
 "client": {
   "environment": "development",
   "subscriptions": [
     "dev"
   ]
 }
}

rabbitmq.json

{                    
  "rabbitmq": {      
    "host": "127.0.0.1",                   
    "port": 5672,    
    "vhost": "/sensu",                     
    "user": "sensu", 
    "password": "secret",                  
    "heartbeat": 30, 
    "prefetch": 50   
  }                  
} 

Finish off by ensuring the permissions for all the files in /etc/sensu/ are owned by sensu:

  • chown sensu. /etc/sensu/

NOTE: The credentials listed above should be changed. They're merely examples pulled straight from the Sensu documentation and should NOT be used as is on any system you care about. You'll also need to chage the IP addresses for Rabbit to the IP of the system(s) you have those running on. If you're using the Sensu Up and Running Vagrant box, the IP addresses will be the same.

Systemd

If you're used to a system using systemd for service management, you'll be familiar with the service files located in /etc/systemd/system. Sensu includes service files for each of the components, but for them to work on a Raspberry Pi, they need to be slightly modified. You'll need to create the service file like so:

sudo vim /etc/systemd/system/sensu-client.service

With the following content:

[Unit]               
Description=sensu client                   

[Service]            
User=sensu           
Group=sensu          
ExecStart=/usr/local/bin/sensu-client                                    
KillMode=process     
Restart=on-failure   
RestartSec=1min      

[Install]            
WantedBy=multi-user.target

From there, start the service:
sudo systemctl start sensu-client

And set it to start at boot:
sudo systemctl enable sensu-client

Which, if you're using the Sensu Up and Running repo I mentioned earlier, should get you something that looks like this:

kegpi-sensu

Next Steps

So what do you do after getting Sensu working on your Raspi? Well, it's up to you! In my case, I'll use it to monitor the temperature, whether or not the door is open, and remaining liquid in the kegs for my kegerator.

In your case, you may want to monitor some system-level activity, like load, available RAM, or something of the like. You could also use it to say, monitor a web app you're running on the Raspi, or, since Sensu's plugins can be written in any programming language, use it to monitor any number of sensors you can attach to the Pi. Again, if you haven't already, take a look over Sensu's training videos to get a better idea of how Sensu works and what you can do with it.

Moving to Kegging my Homebrew

I've posted previously about homebrewing and have been brewing for almost three years now. Back in April, I took to destroying a perfectly good mini fridge to turn it into a kegerator.

IMG_20170403_105320

As any homebrewer can tell you, bottling is a pain in the ass. The delabling (if you're reusing bottles), the santization, and the bottle carbonization makes for a rather long and laborious process. Because I'd finally had it with bottling, I decided that kegging would be a wiser choice since:

  1. Less time would be spent on bottling
  2. Carbonization would be better controlled
  3. For most ales, production time would be cut by a week

Thankfully, my assumptions have proven to be true. So let's walk through what it took to make the kegerator.

The Build

I started off with the following hardware:

I won't go through the whole process of assembling, since the post over at missionarybrewer (which my build was largely based on) covers the steps more than sufficiently. I will mention a few gotchas for the particular fridge I used:

  • There are coolant lines running over in the upper rear left corner. DON'T drill back there
  • The 4.3 cu ft size is just a hair too small for both kegs to fit comfortably. I ended up using a multitool to carve out the front of the door, as well as the molding for holding the shelves so that the kegs would fit.

Now, some pics of the kegerator:

IMG_20170403_162917

IMG_20170403_193658

IMG_20170403_193702

IMG_20170403_193658-1

IMG_20170403_193706

IMG_20170403_193711

IMG_20170403_231824

If you've got any questions, feel free to reach out and let me know!

Moving (Back) To Linux

I'm a bit of a productivity nerd, I'll admit. Over the last couple of years, I've cultivated a workflow that revolved around a lot of OS X tools that I LOVED:

  • Alfred
  • TextExpander
  • iTerm2

These tools were my bread and butter. My professional life, if you will. However, I found myself in a bit of an odd spot in recent months--my MacBook Air was proving to be underpowered for running VM's, and doing any sort of virtualized anything (which I wanted to be able to do for some training and other projects). While I wanted a shiny new MacBook Pro, I found it to be overpriced, and something that I couldn't justify spending the kind of money on, despite that all of my tools were built exclusively for the MBP platform.

So I decided to move back to Linux. At Rackspace, I'd had an Optiplex workstation that ran Fedora, and it worked OK, if not for the fact that every update screwed up my display drivers. I honestly wasn't very keen to be back on Linux. The last thing I wanted to do was be in a situation where I'd have to cobble some sort of solution together, and it would require more time and effort that I have at the moment.

Serendipitously, one of my former coworkers and her husband turned me onto Korora. To be fair, there's not a whole lot more that Korora has over Fedora. Thus far, it's proven to have some prettier defaults. Not much more than what Fedora would have, but it beats taking the time out of my day to hunt around and change things.

To my delight, the experience has been surprisingly Mac-like (even to the point of competing with ElementaryOS). To my further delight, it has dark themes out of the box (swoon), which makes working at night much less stressful on the eyes.

The biggest challenge in moving back to a Linux distro as a daily driver has been finding a replacement for Alfred & TextExpander. The terminal challenge was easily solved with either terminator or tmux (now being a bit wiser, I'm using tmux with GREAT success). However, finding something that had similar functionality to TextExpander and Alfred was a bit difficult at the onset.

I'd tried Albert, an Alfred-like replacement for Linux, but to no avail. While Albert's got a decent start, it lacks quite a bit of Alfred's functionality. For example, Alfred has built-in clipboard management (now using Clipit), built-in snippets (which Autokey seems to do a decent job of), and workflows, which I haven't quite figured out a replacement for and probably won't at this point since I wasn't using them all that much anyways.

At the time of writing this post, I'm back up to speed in terms of where I was with my Mac. I've managed to recreate my level of productivity and automation with the following:

  • Alfred --> Korora menu button/Clipit
  • TextExpander --> Autokey
  • iTerm2 --> Terminator/tmux

Next on my productivity list is tackling my dotfiles. More to follow!

Cheers,

Aaron

Thinking Spaces (A Support Driven Challenge Post)

It's week 3 in the Support Driven Writing Challenge, and this week I'm posting about my thinking space. Let's start out by establishing that I'm an introvert. I may be a bit of an extroverted introvert, but an introvert nonetheless. So naturally, being around people, or in social situations does not allow me to think easily. I'll confess that the larger the group of people I'm around, the more taxing it is for me to sort through my thoughts. It's like people are a mental jamming signal for me. Now, this excludes people that I'm very comfortable around. For example, a small group of friends can make sorting out my thoughts easier, as they know me, know how I think, and can assist in the sorting.

For me, there are a couple of situations where thoughts get sorted out. I'll start with the most...unconventional one: the shower.

There's something about a shower that allows me to sort out my thoughts better than any other space. And I guess that there's a bit of proof that a shower certainly lends itself to that sort of thing. It's in the shower that I've realized what I've been doing incorrectly when writing a script or program, had revelations about why some personal relationships work the way they do, and have come up with designs for some shave brushes.

The second space for me to work things out is on the porch. A great example of this is the recent post/video I made of me playing around on the banjo. Porch time usually requires a good pipe, and some bourbon (Basil Hayden is my go to, if you're wondering). There's something magical about using music to work out my thoughts, and the pipe/bourbon certainly don't hurt either.

The last thinking space for me is at a good coffee shop, usually with a friend to talk things out. There's something about having someone else to bounce ideas off of over a cup of coffee that I find refreshing. I remember that I used to do this weekly with my friend and mentor, Mike. We'd meet every Tuesday at Old City Java in Knoxville, Tennessee for mentoring sessions. We'd also bounce ideas off of each other, and I found those times to be the most rewarding when I needed to sort things out, whether it be about family, work, or life in general.

I do want to add an observation here--the importance of being disconnected. I've found that being constantly on a device seems to add to my mental pile of work. There's also some research that lends some validity to this observation. Whenever I'm in any of those thinking spaces (and especially in the shower), I've found that it's crucial to be disconnected...even if I don't do a great job of disconnecting. I've also found that the lack of disconnecting and allowing myself to be bored is a major contributor to getting burnt out. If I'm letting that mental pile of work continue to grow, then I'm not doing anything to get out from under it. There's also a bit of a rant here on being present, learning to switch off, and the value of actually engaging with real live humans here too, but I think I'll save that for another post.

How I Work

A long time ago, in a galaxy far away, I thought I was productive. I used a whiteboard (still do, by the way) for my todo lists, was running a Windows laptop for work, and that worked. When I moved to Texas and started working at Rackspace, things changed...drastically. First, my workload changed, and it changed how I had to approach my workload. I went from a workload that was so full of technical snowflakes (read 'Unique and one-off issues') and remote support (me logging into customers' laptops and desktops), that any sort of automation was useless.

When I started at Rackspace, I found that my workload was largely repeat issues (low disk on a server, same Apache misconfigurations, same user creation specifications). This led to me using quite a few tools that were focused on 1.) accurately and quickly diagnosing, or solving those issues, and 2.) using tools that allowed me to make templated responses (a la TextExpander and/or aText). While I tend to shy away from response templates, a consistent queue of 80-100 tickets and a focus on response times necessitated working quickly, and utilizing tools that allowed me to do so.

That said, I still carry that need and drive to work quickly and efficiently, and my work setup and the software I use (I feel), reflect that need. My setup also enables me to be mobile and pack light (sans the desktop setup), since working remotely is definitely contributing to my former coffee shop rat habits.

So let's get to the good stuff:

Gear and Software

My Gear

  • MacBook Air (early 2015)
  • Custom desktop dual booting Fedora and Windows
  • 2x Dell 2715Q monitors
  • Logitech Z533 desktop sound system

When I'm On the Move

Mac OSX Tools/Sofware

Other Apps

My Workspace

  • Hand-built iron pipe desk (I'll show it off in the week 5 post for the Support Driven writing challenge)
  • ...Starbucks*

Other Tools

So now that those are out of the way, let me get to my favorite tools. The ones that are indispensable.

My Favorite Tools

Favorite OSX Tool Software:

BetterTouchTool: It was either this, or Atom. I know, who knew a text editor could be awesome. Well, it is, and you should totally use it (or SublimeText), if you're not already. But anywho, the real star: BetterTouchTool. I'm a sucker for window snapping. You know, having a split screen? I use this DAILY. Not only does it do window snapping, but it also creates hotkeys/macros for doing snapping, other window positioning preferences, and various and sundry other functions. To be honest, I'm likely underutilizing it, but it's awesome. Go snag it.

Favorite Non-Digital Tool:

Gridit XL Organizer: Let me say, it's not easy having a favorite out of the tools/gear that I have. The Code & Quill notebook almost won out here. But here's why the Gridit wins: It took the chaos that was my backpack, and brought it all into order. It holds all the cables I have (and let's face it, good cable management is next to godliness) and keeps them tangle-free and organized. It's without a doubt the best $15 (at the time) I've spent on gear.

Favorite WebApp:

Brain.fm: Hands down, Brain.fm is worth the subscription for me to stay sane and focus. I don't do music with lyrics when I need to focus--I end up jamming out in the middle of coffee shops and get weird looks. I'm stuck on this one song in the "Intense Focus Mode" that I just end up on a roll every single time I listen to the track. I've used it when I'm relaxing too (e.g., reading a book, or writing this post, for example). I can't get enough of Brain.fm. Not only is the app great, but their support is fantastic, and responsive. Definitely worth the subscription.

As a note, there are some tools that I've tried like Todoist, Evernote, Magpie, and Wunderlist, just to name some of the ones I formerly used. I've found that fore me, they just haven't stuck. Instead, I stick to my Code & Quill notebook for notetaking, and have started using Slack's native 'remind' function for task management (they start off on the whiteboard so i remember them first, and then make their way to Slack). This seems to have streamlined a lot of my task management and has done away with the task/notetaking tool overload that I've found myself in lately.

Well, there you have it! This, ladies and gents, is how I work.

* So Starbucks requires a bit of explanation and defense (as most people who know me well find it a bit appalling that even a mild coffee snob like myself would spend his/her time at a Starbucks). Let me start by saying, I've found that most local coffee shops in SA do one of two things well: amenities or coffee. There's really not a place here that does both well. Most of the local coffee shops focus on the latter, and rightfully so. However, there's a bit of a knock-on effect: there's a ton of people (most of them students, or remote workers like myself) all pounding the poor little consumer-grade wireless network to death. So I have a choice: Great coffee and crap internet, which as most workers know, it's kind of needed to do our jobs, or mediocre coffee and internet comparable to what I have at my house. I opt for the latter, since I tend to do the coffee pretty decently as a former barista.