Securing Splunk

I feel like security is an often overlooked part of being a Splunk Engineer. This blog post is all about the importance of securing Splunk and the systems that it runs on. In addition to following the Securing Splunk guide in Splunk Docs, here are some other best practices you should be thinking about...

1. Running Splunk as a non-privileged user (i.e. not root)
2. Forwarding your local system messages and audit logs to syslog (and then of course to Splunk)
3. Forwarding all _internal, _introspection, and _internal logs from all non-indexing instance of Splunk to your indexers
4. Configuring host-based firewall rules (e.g. IPTABLES, ufw, or firewalld) for both inbound and outbound connections, specific to port, protocol, and destination host/network(s)
5. Deploying additional open-source security tools to your core Splunk servers, such as OSQuery, OSSEC, or ClamAV
6. Splunking your bash history
7. Disabling the REST port on forwarders when it's not needed
8. Mitigating the POODLE attack in Splunk Web
9. Changing default certificates used for Splunkd, Splunk Web, etc.
10. Using encryption for Splunk to Splunk (S2S) connections
11. Hardening SSH on your Splunk servers
12. Enabling 2FA for SSH 
13. Restricting who has the admin role in Splunk to only a handful of users
14. Implementing patching and vulnerability management policies
15. Hardening the operating system of your Splunk servers using the CIS benchmarks, along with other security controls recommended by NIST, SANS, etc.
16. Enabling data integrity control on all of your indexes
17. Using a common splunk.secret file so that you can securely deploy passwords via configuration files 
18. Using different SSH key pairs for each environment and rotating them periodically
19. Securely storing passwords, secrets, API keys, and other sensitive information using a secret manager like Vault by HashiCorp
20. Disabling Splunk Web on hosts that don't need it (think indexers and heavy forwarders)
21. Changing the default password for the admin account by deploying a user-seed.conf file with a pre-hashed password
22. Randomly generating long passwords for service accounts, or using Splunk's new token-based authentication system for REST API access.
23. Restricting usage of the admin account so that you can properly audit who is making changes through Splunk web or Splunk CLI
24. Enabling SSL on Splunk Web with certificates signed by a trusted CA
25. If you're an app developer, storing passwords using the KV store instead of in plain text contained in configuration files