Data On-Boarding Best Practices

Just a few notes on settings that everyone should be thinking about when creating custom sourcetypes or technology add-ons in Splunk...

Data Parsing
Do you have these configurations in props.conf?
SHOULD_LINEMERGE =
LINE_BREAKER  =
MAX_TIMESTAMP_LOOKAHEAD = 
TIME_PREFIX =
TIME_FORMAT = 
TRUNCATE =

More Data Parsing...
ANNOTATE_PUNCT = false (if you don't need the punct field)
TZ = (if it's not part of the timestamp in your data)
CHARSET = UTF-8 (usually)
NO_BINARY_CHECK = true
KV_MODE = 

Check out Splunk's documentation on props.conf for help with these settings.

Field Extractions
Are you extracting fields for your users at data on-boarding? You should be! Splunk tends to grow organically and if your data isn't well-groomed when you bring it on, it may never be. Setup your users for success by identifying the fields they need and getting them extracted when you on-board their data. 

Be sure to use either EXTRACT in props.conf or a REPORT in props.conf and corresponding REGEX/FORMAT in transforms.conf.

For CIM compliance, use this as a guide: http://docs.splunk.com/Documentation/CIM/4.12.0/User/Howtousethesereferencetables
 
Or, consider using the Splunk Add-on Builder
​
A word on community-built/3rd party apps and addons....

• For COTS products, be sure to check splunkbase.com and github.com for any community built technology-addons. Sometimes you have to modify them a bit to get them working with your data, but they can potentially save you some serious time.
• Don't be afraid of customizing community-built addons, or ripping out pieces you don't want (like eventgen.conf, indexes.conf, KV stores you don't plan to use, etc.).
• Finally, be sure to test anything you download for Splunk in a development environment before installing it in your production environment.  Setting up a test environment can be as simple as spinning up Splunk on your laptop, setting up a vagrant host to run Splunk, or even using a docker image of Splunk. All you need is a sample of your data and a test environment to see if everything will work right.
• Once you have installed a TA in a test environment, be sure to check for startup errors (or simply run splunk cmd btool check. This will tell you if there are any syntax errors in the config files. You can also search the _internal index on your test instance for log_level=ERROR with something like the following: data_source="*" OR data_host="*" OR data_sourcetype="*" You can replace the "*" with the source, host, or sourcetype that you are trying to ingest. This can uncover problems with data parsing, event breaking, etc. that you might not otherwise be aware of