Creating Indexed Fields in Splunk to Identify Heavy Forwarders

Do you use Heavy Forwarders in your organization? Perhaps you have one installed on your syslog server, or on a dozen syslog servers? Chances are that your host field is already being used to identify which host generated any particular event, which is exactly what it was designed to do. But, what if you need to identify where that data is coming from? That's where indexed fields can help out.

I like to call this indexed field, "splunk_forwarder" because it's not one of the fields Splunk uses by default (e.g. splunk_server), and it's easy to remember.

First, we'll create a fields.conf file on our search head(s) to tell Splunk about our indexed field:

[splunk_forwarder]
INDEXED = true

Next, we'll add an inputs.conf file to our heavy forwarder that creates the new field along with its value:

[default]
_meta = splunk_forwarder::myforwarderhostname

This configuration will create a new indexed field called, "splunk_forwarder" and will set its value to whatever you put after the double colons. In this case, it will be assigned a value of "myforwarderhostname". I typically use the hostname of the heavy forwarder, but you could also use the IP address, FQDN, etc.

Finally, restart Splunk on your heavy forwarder and search head(s). Any new data that gets indexed will automatically have your new splunk_forwarder field!

Now, you can run cool searches like this one to quickly see which forwarders are sending what data to Splunk:

| tstats count  where splunk_forwarder=* index=* by splunk_forwarder sourcetype index | stats values(index) as index values(sourcetype) as sourcetype sum(count) as count by splunk_forwarder ​