Mass-Updating Knowledge Objects on Splunk Search Head Clusters

Have you ever been in a situation where you needed to mass-edit a large number of knowledge objects on a search head cluster? Any Splunk admin that has ever had to redirect data to a new index knows how painful this can be. Today, I'm going to teach you the easy way to do it, without even having to restart splunk!

Here are the steps:

1. Find the SHC captain via Splunk Web (Settings -> Search Head Clustering) or via CLI splunk show shcluster-status
2. SSH into the captain node and sudo to the splunk user
3. Perform a git clone https://github.com/masonsmorales/splunk_script_update_files or copy the contents of the update_files.sh bash script and update_files.txt file from a browser. (Note: If you copy them manually, you'll need to do a chmod +x on update_files.sh)
4. Move the two files into the topmost directory that you want to change knowledge objects for. This could be the entire $SPLUNK_HOME/etc folder, only $SPLUNK_HOME/etc/apps, or a specific app.
5. Edit the update_files.txt contents to your liking. The file should contain a list of filenames and/or patterns that you want to perform the find/replace operation against.
6. Edit lines 7 of update_files.sh with the original text that you want to find, and line 8 with the new text you want to replace it with.
7. As a best practice, always take a backup of whatever you are going to change. e.g. tar czvf splunk_etc_bak.tar $SPLUNK_HOME/etc
8. After you've taken a backup and completed your edits, run the script to update the configuration files on disk. e.g. ./update_files.sh
9. Once the script has completed, you'll need to force Splunk to reload the on-disk Splunk configurations on the captain. The quickest way to do this is by restarting only the Splunk Web service. Here's the command: splunk restartss (Clarifying Note: SS = Splunk Search. This command is an alias to splunk restart splunkweb)
10. Finally, once the previous command has completed, SSH into each of the slaves and force them to download the latest bundle from the captain by executing: splunk resync shcluster-replicated-config
11. That’s it! All that's left is to validate your changes. From one of the slaves, cat one of the file paths that the script updated and confirm that the file contents reflect your changes.