Just a few notes on settings that everyone should be thinking about when creating custom sourcetypes or technology add-ons in Splunk...

Data Parsing
Do you have these configurations in props.conf?
SHOULD_LINEMERGE =
LINE_BREAKER  =
MAX_TIMESTAMP_LOOKAHEAD = 
TIME_PREFIX =
TIME_FORMAT = 
TRUNCATE =

More Data Parsing...
ANNOTATE_PUNCT = false (if you don't need the punct field)
TZ = (if it's not part of the timestamp in your data)
CHARSET = UTF-8 (usually)
NO_BINARY_CHECK = true
KV_MODE = 

Check out Splunk's documentation on props.conf for help with these settings.

Field Extractions
Are you extracting fields for your users at data on-boarding? You should be! Splunk tends to grow organically and if your data isn't well-groomed when you bring it on, it may never be. Setup your users for success by identifying the fields they need and getting them extracted when you on-board their data. 

Be sure to use either EXTRACT in props.conf or a REPORT in props.conf and corresponding REGEX/FORMAT in transforms.conf.

For CIM compliance, use this as a guide: http://docs.splunk.com/Documentation/CIM/4.12.0/User/Howtousethesereferencetables
 
Or, consider using the Splunk Add-on Builder
​
A word on community-built/3rd party apps and addons....

• For COTS products, be sure to check splunkbase.com and github.com for any community built technology-addons. Sometimes you have to modify them a bit to get them working with your data, but they can potentially save you some serious time.
• Don't be afraid of customizing community-built addons, or ripping out pieces you don't want (like eventgen.conf, indexes.conf, KV stores you don't plan to use, etc.).
• Finally, be sure to test anything you download for Splunk in a development environment before installing it in your production environment.  Setting up a test environment can be as simple as spinning up Splunk on your laptop, setting up a vagrant host to run Splunk, or even using a docker image of Splunk. All you need is a sample of your data and a test environment to see if everything will work right.
• Once you have installed a TA in a test environment, be sure to check for startup errors (or simply run splunk cmd btool check. This will tell you if there are any syntax errors in the config files. You can also search the _internal index on your test instance for log_level=ERROR with something like the following: data_source="*" OR data_host="*" OR data_sourcetype="*" You can replace the "*" with the source, host, or sourcetype that you are trying to ingest. This can uncover problems with data parsing, event breaking, etc. that you might not otherwise be aware of

I feel like security is an often overlooked part of being a Splunk Engineer. This blog post is all about the importance of securing Splunk and the systems that it runs on. In addition to following the Securing Splunk guide in Splunk Docs, here are some other best practices you should be thinking about...

1. Forwarding your local system messages and audit logs to syslog and then of course to Splunk
2. Forwarding all _internal, _introspection, and _internal logs from all non-indexing instance of Splunk to your indexers
3. Configuring host-based firewall rules (e.g. IPTABLES, ufw, or firewalld) for both inbound and outbound connections, specific to port, protocol, and destination host/network(s)
4. Deploying additional open-source security tools to your core Splunk servers, such as OSQuery, OSSEC, or ClamAV
5. Splunking your bash history
6. Disabling the REST port on forwarders when it's not needed
7. Mitigating the POODLE attack in Splunk Web
8. Changing default certificates used for Splunkd, Splunk Web, etc.
9. Using encryption for Splunk to Splunk (S2S) connections
10. Hardening SSH on your Splunk servers
11. Enabling 2FA for SSH 
12. Restricting who has the admin role in Splunk to only a handful of users
13. Implementing patching and vulnerability management policies
14. Hardening the operating system of your Splunk servers using the CIS benchmarks, along with other security controls recommended by NIST, SANS, etc.
15. Enabling data integrity control on all of your indexes
16. Using a common splunk.secret file so that you can securely deploy passwords via configuration files 
17. Using different SSH key pairs for each environment and rotating them periodically
18. Securely storing passwords, secrets, API keys, and other sensitive information using tools like Vault by HashiCorp
19. Disabling Splunk Web on hosts that don't need it (think indexers and heavy forwarders)
20. Changing the default password for the admin account by deploying a user-seed.conf file
21. Using a separate account for distributed search than the admin account
22. Restricting usage of the admin account so that you can properly audit who is making changes through Splunk web or Splunk CLI
23. Enabling SSL on Splunk Web with certificates signed by a trusted CA
24. If you're an app developer, storing passwords using the KV store instead of in plain text contained in configuration files

I recently upgraded a Splunk cluster from v6.5.2 to v7.0.1. There was one thing that wasn't covered in the release notes. After upgrading my first host (master node), I couldn't execute CLI commands. Splunk threw the following error:

Splunk Support admitted that they have some SSL bugs in the new release, and that this was one of them. To workaround this, you can make the following edits in server.conf:

Once this is done, restart Splunk and try the CLI again. You should be back in business.

I had to update server.conf on most of my Splunk server hosts (master node, search heads, deployers, deployment server, license master, etc.) but for some reason not on my indexers. I'm not sure why as both my indexers and search heads run the same OS and had the same OpenSSL package installed. Hopefully this helps anyone out there with a similar issue.

Do you use Heavy Forwarders in your organization? Perhaps you have one installed on your syslog server, or on a dozen syslog servers? Chances are that your host field is already being used to identify which host generated any particular event, which is exactly what it was designed to do. 

But, what if you need to identify where that data is coming from? That's where indexed fields can help out.

I like to call my indexed field, "splunk_forwarder". It's not one of the fields Splunk uses by default (e.g. splunk_server), and it's easy to remember.

First, we'll create a props.conf file to tell Splunk that the new field we are going to create should apply to every host that this forwarder collects data from:

[host::*]
TRANSFORMS-create_splunk_forwarder_field = create_splunk_forwarder_field

Next, we'll create a transforms.conf file to actually create the new field along with its value.

[create_splunk_forwarder_field]
REGEX = .+
FORMAT = splunk_forwarder::"myforwarderhostname"
WRITE_META = true

This configuration will create a new indexed field called, "splunk_forwarder" and will set its value to whatever name you give it in the quotation marks. I typically use the hostname of the heavy forwarder, but you could also use the IP address, FQDN, etc.

Now that you have your configuration, there are two ways to deploy it. The first is to create it locally under $SPLUNK_HOME/etc/system/local. This option is ideal if you're only applying it in a couple places, and you aren't using a configuration management system (e.g. Ansible, Puppet, Chef, Salt). The other method, is to deploy it using a Splunk Deployment Server (DS). If you are using a DS, make sure you create an app to hold your props.conf and transforms.conf files under $SPLUNK_HOME/etc/deployment-apps/<yourapp>/local/.

Next, you need to deploy a fields.conf file to both your search heads and indexers in order to be able to properly search it. The fields.conf file should look like this:

[splunk_forwarder]
INDEXED = true

Finally, restart Splunk on your heavy forwarder. Any new data that gets indexed will automatically have your new splunk_forwarder field!

Bonus: If you don't want to deploy props.conf and transforms.conf, you can also accomplish this via the fields.conf deployment combined with the following inputs.conf configuration on each of your heavy forwarders:

[default]
_meta = splunk_forwarder::myforwarderhostname
​

New to Splunk? This is a list of learning resources that I've curated for new Splunk users over the years. Feel free to share this with your fellow Splunkers!

• Splunk Quick Reference Guide
• Splunk Answers/Community Forum
• Exploring Splunk eBook (free)
• Splunk Intro Course (free)
• Splunk Education Videos
• Splunk Documentation
• Splunk's Global User Group Conference​