How to configure Splunk Multi site Indexer Cluster
Deploy a multisite indexer cluster
To deploy a multisite cluster, you configure the set of nodes for each site:
- A single master resides on one of the sites and controls the entire multisite cluster.
- A set of peer nodes resides on each site.
- A search head resides on each site that searches cluster data.
- If you want all searches to be local, you must install a search head on each site.
For example, to set up a two-site cluster with two peers and two search head on each site,
You install and configure these instances:
- One master node on one of the sites, either site 1 or site 2
- Two peer nodes on site 1
- Two peer nodes on site 2
- Two search head on site 1
- Two search head on site 2
On the master, you:
- Enable the cluster for multisite.
- Enumerate the set of sites for the cluster.
- Set a multisite replication factor.
- Set a multisite search factor.
- Master node. Configuration of overall cluster behavior.
- Peer node. Configuration of individual peer node and cluster indexing behavior.
- Search head. Configuration of individual search head and search behavior in an indexer cluster.
Configure the indexer cluster with the dashboards
To configure an indexer cluster node through its dashboard:
- Click Settings on the upper right side of Splunk Web on the node.
- In the Distributed Environment group, click Indexer clustering.
- Select the Edit button on the upper right side of the dashboard.
An index is a collection of databases, which are subdirectories located in $/opt/splunk/var/lib/splunk.
Indexes consist of two types of files: rawdata and index files.
Splunk Enterprise comes with a number of preconfigured indexes, including:
- main: This is the default Splunk Enterprise index. All processed data is stored here unless otherwise specified.
- _internal: Stores Splunk Enterprise internal logs and processing metrics.
- _audit: Contains events related to the file system change monitor, auditing, and all user search history.
Configure the indexer cluster with server.conf
Indexer cluster settings reside in the server.conf file, located in $/opt/splunk/etc/system/local/server.conf.
The main server.conf stanza that controls indexer clustering is [clustering].
Besides the basic attributes that correspond to settings in Splunk Web, server.conf provides a number of advanced settings.
$/opt/splunk/etc/system/local/server.conf
[clustering]
mode = master
replication_factor = 4
search_factor = 3
pass4SymmKey = whatever
cluster_label = cluster1
This example specifies that:
- the instance is a cluster master node.
- the cluster’s replication factor is 4.
- the cluster’s search factor is 3.
- the security key is “whatever”. All nodes in the cluster use the same security key.
- the cluster label is “cluster1.” The optional cluster label is useful for identifying the cluster in the monitoring console.
|
Site 1 New York City
|
||||
| Splunk Instance Role | IP | Listing Port | Replication Port | |
| Deployment Server | 10.20.9.5
|
|||
|
CLUSTER-MASTER Also act License Master |
10.20.9.11 | |||
| Site1-idx1 | 10.20.9.15 | 9997 | 9887 | |
| Site1-idx2 | 10.20.9.16 | 9997 | 9887 | |
| Site1-sh1 | 10.20.9.31 | 9887 | ||
| Site1-sh2 | 10.20.9.32 | 9887 | ||
|
Site 2 Las Vegas
|
||||
| Splunk Instance Role | IP | Listing Port | Replication Port | |
| Site2-idx1 | 10.10.9.15 | 9997 | 9887 | |
| Site2-idx2 | 10.10.9.16 | 9997 | 9887 | |
| Site2-sh1 | 10.10.9.31 | 9887 | ||
| Site2-sh2 | 10.10.9.32 | 9887 | ||
| Installed Splunk Enterprise on all the above host in table using the below commands.
wget -O splunk-7.0.1-2b5b15c4ee89-Linux-x86_64.tgz ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.0.1&product=splunk&filename=splunk-7.0.1-2b5b15c4ee89-Linux-x86_64.tgz&wget=true’
sudo tar zxf splunk-7.0.1-2b5b15c4ee89-Linux-x86_64.tgz -C /opt sudo chown -R splunker:splunk /opt/splunk sudo ls -l /opt/splunk sudo su – splunker sudo /opt/splunk/bin/splunk start –accept-license –no-prompt -answer sudo /opt/splunk/bin/splunk enable boot-start -user splunker sudo /opt/splunk/bin/splunk status /opt/splunk/bin/splunk show web-port -auth admin:changeme /opt/splunk/bin/splunk show splunkd-port -auth admin:changeme /opt/splunk/bin/splunk show appserver-ports -auth admin:changeme /opt/splunk/bin/splunk show kvstore-port -auth admin:changeme /opt/splunk/bin/splunk show servername -auth admin:changeme /opt/splunk/bin/splunk show default-hostname -auth admin:changeme netstat -tuplen | grep splunkd netstat –an | grep 8000
|
| Login into each instance and change their name based on instance role
Example License Master (10.20.9.4) /opt/splunk/bin/splunk set servername License Master -auth admin:changeme /opt/splunk/bin/splunk set default-hostname License Master -auth admin:changeme /opt/splunk/bin/splunk restart /opt/splunk/bin/splunk show servername -auth admin:changeme /opt/splunk/bin/splunk show default-hostname -auth admin:changeme
|
| License Master (10.20.9.11)
/opt/splunk/bin/splunk add license /tmp/file.license /opt/splunk/bin/splunk restart List the license /opt/splunk/bin/splunk list license License of all slaves /opt/splunk/bin/splunk list licenser-salves
|
| Cluster Master (10.20.9.11)
/opt/splunk/bin/splunk edit cluster-config –mode master -multisite true –site site1 –available_site site1,site2 –site_repliaction_factor origin:2,total:4 -search_factor 3 –secret My_First_Splunk_Multi-site_Cluster /opt/splunk/bin/splunk restart /opt/splunk/bin/show cluster status
|
| Indexer SITE 1 New York City Site1-idx1 (10.20.9.15)
/opt/splunk/bin/splunk edit licenser-localslave –master_uri http://10.20.9.11:8089 /opt/splunk/bin/splunk restart This is instance license status /opt/splunk/bin/splunk licenser-localsalve /opt/splunk/bin/splunk edit cluster-confg –master_uri http://10.20.9.5:8089 –mode slave –site site2 –replication_port 9887 –secret My_First_Splunk_Multi-site_Cluster /opt/splunk/bin/splunk restart
|
| Indexer SITE 1 New York City Site1-idx2 (10.20.9.16)
/opt/splunk/bin/splunk edit licenser-localslave –master_uri http://10.20.9.11:8089 /opt/splunk/bin/splunk restart This is instance license status /opt/splunk/bin/splunk licenser-localsalve /opt/splunk/bin/splunk edit cluster-confg –master_uri http://10.20.9.11:8089 –mode slave –site site2 –replication_port 9887 –secret My_First_Splunk_Multi-site_Cluster /opt/splunk/bin/splunk restart
|
| Indexer SITE 2 Las Vegas Site2-idx1 (10.20.9.15)
/opt/splunk/bin/splunk edit licenser-localslave –master_uri http://10.20.9.11:8089 /opt/splunk/bin/splunk restart This is instance license status /opt/splunk/bin/splunk licenser-localsalve /opt/splunk/bin/splunk edit cluster-confg –master_uri http://10.20.9.11:8089 –mode slave –site site2 –replication_port 9887 –secret My_First_Splunk_Multi-site_Cluster /opt/splunk/bin/splunk restart
|
| Indexer Peer SITE 2 Las Vegas Site2-idx2 (10.20.9.16)
/opt/splunk/bin/splunk edit licenser-localslave –master_uri http://10.20.9.11:8089 /opt/splunk/bin/splunk restart This is instance license status /opt/splunk/bin/splunk licenser-localsalve /opt/splunk/bin/splunk edit cluster-confg –master_uri http://10.20.9.11:8089 –mode slave –site site2 –replication_port 9887 –secret My_First_Splunk_Multi-site_Cluster /opt/splunk/bin/splunk restart
|
| Cluster Commands and Troubleshooting
ONLY RUN ON CLUSTER MASTER Cluster-Master /opt/splunk/bin/show cluster status Cluster-Master /opt/splunk/bin/splunk rolling-restart cluster-peer Cluster in maintenance mode Cluster-Master /opt/splunk/bin/splunk [ enable | disable | show ] maintenance-mode Apply apps to all peers Cluster-Master /opt/splunk/bin/spluk apply cluster-bundle Cluster-Master /opt/splunk/bin/spluk apply cluster-bundle-status To take Indexer Peer offline Indexer /opt/spunk/bin/splunk offline $egrep “WARN|ERROR” /opt/splunk/var/log/splunk/splunkd.log $ /opt/splunk/bin/splunk btool check –debug To verify once the Forwarder to sending the logs to indexers. Indexer index=_internal sourcetype=splunkd host={Universal_Forwarder} TcpOutputProc Indexer index=_internal sourcetype=splunkd component=Metrics series={Your_Index} NOTE: /opt/splunk/bin/splunk edit cluster-config –cluster_label < CLUSTER_LABEL > http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/Multisitedeploymentoverview
|
