Hey All!
/* Ubar long post warning! */
So if you’ve ever managed any servers or application you know how much of a pain in the ass logging is and how important it is to keep excellent logs for debugging.
We recently started breaking out infrastructure out at Stitch from a single fat server to a more robust, horizontally scalable architecture. (I’ll talk more about these changes in the future) With that comes even more headaches with log management because events can take place atomically on any server so tracking down the entirety of a request can prove to be most annoying.
As such, I’ve started kicking the tires on a few SaaS logging apps:
However two of these really stood out to me, PaperTrail and Loggly.
Honestly, I had never even heard of either of any these apps until I went to our True University and heard loggly’s CEO talking about their app’s “Big Data” problems. That was the catalyst for finding a logging system for all of our gear.
I guess I’ll start with the bad news first, I wasn’t a fan of LogEntries. Their UI just didn’t have the right feel for me and the way they classify their hosts seemed odd in my book. Their support was helpful with my questions and they were able to get TLS log shipping added to the app as per my request. Their pricing was also pretty darn good, I like that you don’t have to pay much if you don’t use much. However, I could never get that working properly. In the end I just got lazy and decided to move forward with PaperTrail and Loggly since getting those up and running with TLS log shipping proved to be quite easy.
All of these services offer an agent that you can install on your boxes to tail logs and ship them securely, but I don’t really want to induce any sort of vendor lock-in for Stitch. So I’d rather use the more standardized syslog system that already comes with all Linux machines. Down the line if I feel we need to take ownership of our logging this can be easily done by just pointing the syslog daemons to our own in-house logging server.
Along that thought pattern, all of these services have some sort of RESTful API that you can send data to from your application whether it’s a Python app or a PHP app. This seemed pretty neat at first but then I fell back on the vendor lock-in dilemma so I ended up just using PHP’s built in support for syslog and offloading all of the log routing to the rsyslog daemon.
Now, on to our round finalists…..PaperTrail and Loggly! Both off these services offer their own set of pro’s and con’s and it kind of sucks because my ideal would be a hybrid of the two…..PaperTrail has a feature that Loggly needs and vice-versa.
Quick Summary:
If all you want is a quick summary of PaperTrail and Loggly in my eyes, you can stop here. Otherwise continue for a fairly in-depth review!
| PaperTrail | Loggly |
| "tail -f" like functionality | Yes | No |
| Pretty Graphs | Yes (through a 3rd party service Librato) | Yes, natively |
| Realtime Graph History | No | Yes |
| JSON Parsing | No | Yes-ish |
| AWS Log Archiving | Yes | Yes |
| Support | Better | Good |
| Search Functionality | Pretty Win... | Also Pretty Win... |
| Price | More expensive | Less expensive |
Nitty Gritty:
Tail -f
So right off the bat the feature that is pretty much holding up my decision from using Loggly is a “tail -f” functionality. That is….the ability to auto-follow logs as they come in. I like to view logs as they update, makes me feel like I’m watching the matrix and it gives me a quick overview of how things are moving along (or not). So for my uses this functionality is quite essential. Tailing functionality seems to be a pretty popular request over at the Loggly forums but more than a year after the question was first asked (and 6 months after their “planned” release) nothing has been done about this. This is kind of sad, but being at a startup I know how finite resources really put a damper on some things. (PaperTrail: +2, Loggly: +0)
Graphs
Next we have graphing. Graphs are good, no great…they allow you to quickly visualize information and form trends on your data. Both of these applications have the ability to graph specific search results, however, Loggly has this functionality built in and it’s wonderfully integrated. PaperTrail on the other hand outsources the graphing work to Librato Metrics. Librato is a ridiculously cheap charting application for pretty much any kind of data you want to throw at it. Problem is, I don’t want to have to go to 20 different sites to look at my data, if it’s all in once place that is mo betta! Another issue with PaperTrails usage of Librato is that you only get graphs for searches you have defined. So you won’t get any historical graph data unless you’ve defined a particular search and set it up with Librato. While this doesn’t seem like a big deal on the surface, I often find myself thinking of things that I’d like to graph on the fly, (Hey I wonder what X vs Y looks like) rather than thinking of allllll the possible graphing combinations up front. Loggly really fosters this because it uses prior logged data to create it’s graphs. So if I think of a query to graph something off the top of my head it will show me some historical trends right away rather than me having to wait. Pretty kewl…. (PaperTrail: +0, Loggly: +1)
Since this is turning into a wall of text, here are some of the graph examples for the same dataset between Librato and Loggly’s built in graphing:
Librato
Loggly
JSON Parsing
Up next is JSON parsing, Loggly goes to some length to explain the benefits of sending in JSON formatted logs. I instantly saw the value in this for surgically precise log filtering. (We currently log a lot of things in a SQL database, so mimicking SQL queries with a low barrier to entry is a super win for us) Unfortunately Loggly doesn’t support JSON inputs when using TLS log shipping. I can’t imagine there to be any technical reasons why this can’t be done, but I assume it’s a lack of resources based on the generic “We’ll add it to the TODO with no ETA” response I got. So as awesome as the JSON formatted logs are, I can’t even use them because we ship all of our logs securely using TLS. Alas, I am already shipping our logs JSON encoded/formatted so when either of these services decides to get this working I’ll be good to go! (PaperTrail: +0, Loggly: +0)
Support
First off, I want to say that as an addicted engineer I spend many many hours working, learning and tinkering with stuff. So I tend to stay up to the wee hours of the night doing things and breaking them. I fully exercised the support teams for both PaperTrail and Loggly, likely to the point of where I was getting to be annoying. (If not that I’m definitely on a first name basis with the guys over in the PaperTrail chat room!) Both services had pretty good support, turn around times on support tickets were under 24 hours consistently so that’s a plus. However, the guys at PaperTrail must be addicted engineers just as I am. Very late in the evening on the 4th of July (when we’re all supposed to be off apparently) I jumped into the PaperTrail chat room to see if anyone was around for a few questions that I had while trying to get the syslogging to work. Much to my surprise there was a live person there sitting around being bored. (I believe this was Eric…but it might have been Troy, I don’t recall) All of my questions were answered right away with links to the KB articles that I couldn’t find apparently. Awesome! I’ve since been back in the chat room for basic usage questions and someone is always there to field them (do you guys even sleep!?). Now don’t get the wrong impression here, Loggly also has a chat room on IRC. I idle on IRC alllll day so I know how easy it can be to miss messages using that medium. I sent a few queries Loggly’s way via IRC and they didn’t get noticed right away like they were for PaperTrail (PaperTrail actually says hello to me before I even finish typing my message….I think they may really be bots) however one of their representatives did see my questions and helped clear up my questions right then and there….pretty cool! I think PaperTrail takes this round, but only by a small margin. (PaperTrail: +1, Loggly: +0)
Pricing
Last but not least is the infamous price. While both services are pretty cheap, Loggly bests PaperTrail in this category. PaperTrail bills on monthly log volume, Loggly bills on daily log volume, and both bill on log index (search-ability) retention. The common denominator between both services for us is to be able to search logs 2 weeks back. Taking our average daily log throughput from Loggly we do about 50MB of logs a day (not much, but that’s where we’re at). With Loggly we can do a plan with 200MB/day and 2 weeks of retention for $8/mo. Extrapolating that data usage out a month we get about 1.46GB/mo. (If we do a straight up comparison with the 200MB/day limit on Loggly that works out to 5.85GB/mo) With PaperTrail’s pricing, at our current volume we’re charged $22/mo for 2GB of logging and 2 weeks of retention. (If you do the straight across comparison PaperTrail would be $35/mo for 4GB/mo of logging…the closest amount their calculator has) Now some (many?) will complain that I’m just being too cheap, but if look at this in terms of percentages PaperTrail is 175% more expensive. Being at a startup costs and burn rate mean a lot, so the less I have to spend on our infrastructure the more money we can put toward acquiring more customers. (PaperTrail: +0, Loggly: +2)
Phew! If you’ve made it this far, thanks for staying! For those of you who have been keeping score here is the result:
So if you haven’t guessed I’m still on the fence about which to choose, once I make up my mind I’ll update this post to outline what we ended up going with. If you have any questions/concerns/ideas feel free to post them in the comments!
Until next time….Later!
