Lately, I've been finding myself building more and more complex environments in VMWare, simulating complex scenarios that have networking requirements above and beyond a mere LAN Segment. For example, as an SE, suppose I want to more closely mirror a customer environment, when simulating a deployment change. If you want to maintain the IPs of [...]
If you've added an input, savedsearch, or any manner of other config through the ui, you might have a lot of trouble finding the configuration file later. If you're on Unix, the easiest way is to do a search #/opt/splunk/etc/: egrep -R MySearchName . ./users/jsmith/launcher/local/savedsearches.conf:[MySearchName] If you're on Windows.. [...]
A piece of advice for those starting out in Splunk: Always Rename Your Fields It's slightly more accurate to say "always rename your fields after aggregation functions (e.g., stats, timechart)" but that lacks the zing. Why is this so important? The biggest reason is that it will just consistently make your life easier. If your search is as [...]
Getting comfortable with timechart is key to getting comfortable with Splunk. It's the core of your first reports in the system. Take a look at the below posts for a shortcut to going from basic to advanced with timechart: Basic Syntax Advanced Statistics Analyzing Trends (Comparing Summarized Hours, Days, or Etc.) Using Eval Within Time [...]
Timechart and stats are very similar in many ways. They have access to the same (mostly) functions, and they both do aggregation. The biggest difference lies with how Splunk thinks you'll use them. Timechart is much more user friendly. You can run a | timechart span=1d sum(MB) by series and it will create take each series and create a column name [...]
Taking the game up another level, the ability to use eval statements within Timechart increases flexibility immensely. Suppose you have a health page where you can look at one server out of twenty, or one partner out of seven, or any other scenario where you're comparing one fungible element to the average of others. Without using eval in timechart [...]
One of the most useful theories to get when using timechart is generalizing data to a certain level of granularity, and then tracking changes over time. A good example of this is looking at the hourly amount by Splunk, but viewed on a per day basis. Splunk is going to toss many events per hour. You need to first find the hourly sum for every hour [...]
In the last post (Overview of Timechart Syntax in Splunk), we covered the basics of Splunk's timechart Syntax. This post will cover some of the more advanced statistical functions of timechart. The statistical functions are really the core of timechart (along with the core of Stats -- see Timechart Versus Stats). Check out the full list of statistics [...]
Timechart is one of the most essential functions in Splunk. Unfortunately for those starting out, getting access to its many wonders is a bit hard to do. The official documentation (timechart, functions available to timechart) ?is great once you've got your feet firmly in the water -- this blog post (and the links at the bottom) are for people who [...]
The Problem A pet peeve of mine in Splunk is its tendency to summarize data too much when putting it in a Timechart. It makes for smoother flowing (and I suppose more executive-friendly) charts. For example, with one of my data sources, I have data points every ten minutes (result of a script). If I graph that over the course of the last 24 hours [...]
The Problem I have been building a Splunk ?App to monitor customer usage of a product. In particularly, we're replacing some very ugly and generally inaccurate Zenoss graphs with Splunk precision. Zenoss, for those who haven't interacted with it, is an RRD based monitoring tool. That means it has accuracy issues and has some interesting quirks [...]
Why?: The problems I wanted to solve were controlling timechart bin selection, and providing transparent index switching in Splunk. The solution relied heavily on Sideview Utils, created by Nick Mealy of Sideview, LLC. Example App: I could probably whip one up pretty easily, but I don't want to spend the time unless there's a demand. Let me know [...]
At the recent San Francisco Splunk Meetup, there was a brief joking exchange about how the secret to using Summary Indexing was to ignore the summary index commands (sistats, etc.). This brought up a question about realistically, how we one should use summary indexing, so I decided to create an explanation of how I use it in my environment. There [...]
If you are anything like me, the poor drilldown support in Splunk drives you up the wall. Splunk's major strength is the creativity it affords you -- given a number of command line tools, you can implement all manner of crazy reports. And for most problems, you there's another (usually better) way of solving your problem. But bizarrely, the drilldown [...]
This is as much for my documentation as for anyone else spending ages searching through Google to find a working solution. I wanted to run Splunk on my personal site, naturally, but I also didn't want Splunk listening to the world on port 8000. It took me a lot of experimentation to find a working solution, but here it is: [...]
Probably the hardest working component of my Splunk CDR project is the Perl script that converts CCM logs to Splunk logs. In addition to just doing the base conversion of CCM CSV format logs to Splunk-readable logs, this script adds a significant amount of additional detail to the log files to make reporting easier. Its core functionalities are as [...]
I'm engaging in a project to move our Cisco Unified Call Manager (aka CUCM, here known as CCM) Call Detail Report (CDR) logs into an internally managed system. I've elected to use Splunk for this task because of its flexibility and power in processing log entries. ?The goal is to make a Splunk app that is generic enough to be used by others, and [...]