How to configure Splunk Multi site Indexer Cluster

Deploy a multisite indexer cluster

To deploy a multisite cluster, you configure the set of nodes for each site:

  • A single master resides on one of the sites and controls the entire multisite cluster.
  • A set of peer nodes resides on each site.
  • A search head resides on each site that searches cluster data.
  • If you want all searches to be local, you must install a search head on each site.

For example, to set up a two-site cluster with two peers and two search head on each site,

You install and configure these instances:

  • One master node on one of the sites, either site 1 or site 2
  • Two peer nodes on site 1
  • Two peer nodes on site 2
  • Two search head on site 1
  • Two search head on site 2

On the master, you:

  • Enable the cluster for multisite.
  • Enumerate the set of sites for the cluster.
  • Set a multisite replication factor.
  • Set a multisite search factor.
  • Master node.    Configuration of overall cluster behavior.
  • Peer node.          Configuration of individual peer node and cluster indexing behavior.
  • Search head.     Configuration of individual search head and search behavior in an indexer cluster.

 

Configure the indexer cluster with the dashboards

To configure an indexer cluster node through its dashboard:

  1. Click Settings on the upper right side of Splunk Web on the node.
  2. In the Distributed Environment group, click Indexer clustering.
  3. Select the Edit button on the upper right side of the dashboard.

An index is a collection of databases, which are subdirectories located in $/opt/splunk/var/lib/splunk.

Indexes consist of two types of files: rawdata and index files.

 

Splunk Enterprise comes with a number of preconfigured indexes, including:

  • main: This is the default Splunk Enterprise index. All processed data is stored here unless otherwise specified.
  • _internal: Stores Splunk Enterprise internal logs and processing metrics.
  • _audit: Contains events related to the file system change monitor, auditing, and all user search history.

 

Configure the indexer cluster with server.conf

Indexer cluster settings reside in the server.conf file, located in $/opt/splunk/etc/system/local/server.conf.

The main server.conf stanza that controls indexer clustering is [clustering].

Besides the basic attributes that correspond to settings in Splunk Web, server.conf provides a number of advanced settings.

 

$/opt/splunk/etc/system/local/server.conf

[clustering]

mode = master

replication_factor = 4

search_factor = 3

pass4SymmKey = whatever

cluster_label = cluster1

 

This example specifies that:

  • the instance is a cluster master node.
  • the cluster’s replication factor is 4.
  • the cluster’s search factor is 3.
  • the security key is “whatever”. All nodes in the cluster use the same security key.
  • the cluster label is “cluster1.” The optional cluster label is useful for identifying the cluster in the monitoring console. 
 

Site 1                          New York City

 

Splunk Instance Role IP Listing Port Replication Port
Deployment Server 10.20.9.5

 

 

CLUSTER-MASTER

Also act  License Master

10.20.9.11
Site1-idx1 10.20.9.15 9997 9887
Site1-idx2 10.20.9.16 9997 9887
Site1-sh1 10.20.9.31 9887
Site1-sh2 10.20.9.32 9887
 

Site 2                   Las Vegas

 

Splunk Instance Role IP Listing Port Replication Port
Site2-idx1 10.10.9.15 9997 9887
Site2-idx2 10.10.9.16 9997 9887
Site2-sh1 10.10.9.31 9887
Site2-sh2 10.10.9.32 9887

 

Installed Splunk Enterprise on all the above host in table using the below commands.

 

wget -O splunk-7.0.1-2b5b15c4ee89-Linux-x86_64.tgz ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.0.1&product=splunk&filename=splunk-7.0.1-2b5b15c4ee89-Linux-x86_64.tgz&wget=true’

 

sudo tar zxf splunk-7.0.1-2b5b15c4ee89-Linux-x86_64.tgz -C /opt

sudo chown -R splunker:splunk /opt/splunk

sudo ls -l /opt/splunk

sudo su – splunker

sudo /opt/splunk/bin/splunk start –accept-license –no-prompt -answer

sudo /opt/splunk/bin/splunk enable boot-start -user splunker

sudo /opt/splunk/bin/splunk status

/opt/splunk/bin/splunk show web-port -auth admin:changeme

/opt/splunk/bin/splunk show splunkd-port -auth admin:changeme

/opt/splunk/bin/splunk show appserver-ports -auth admin:changeme

/opt/splunk/bin/splunk show kvstore-port -auth admin:changeme

/opt/splunk/bin/splunk show servername -auth admin:changeme

/opt/splunk/bin/splunk show default-hostname -auth admin:changeme

netstat -tuplen | grep splunkd

netstat –an | grep 8000

 

 

Login into each instance and change their name based on instance role

Example License Master (10.20.9.4)

/opt/splunk/bin/splunk set servername  License Master -auth admin:changeme

/opt/splunk/bin/splunk set default-hostname License Master -auth admin:changeme

/opt/splunk/bin/splunk restart

/opt/splunk/bin/splunk show servername -auth admin:changeme

/opt/splunk/bin/splunk show default-hostname -auth admin:changeme

 

 

License Master  (10.20.9.11)

/opt/splunk/bin/splunk add license /tmp/file.license

/opt/splunk/bin/splunk restart

List the license

/opt/splunk/bin/splunk  list license

License of all slaves

/opt/splunk/bin/splunk list licenser-salves

 

 

Cluster Master (10.20.9.11)

/opt/splunk/bin/splunk edit cluster-config –mode master  -multisite true –site site1 –available_site site1,site2 –site_repliaction_factor origin:2,total:4 -search_factor 3 –secret My_First_Splunk_Multi-site_Cluster

/opt/splunk/bin/splunk restart

/opt/splunk/bin/show cluster status

 

 

Indexer  SITE 1 New York City       Site1-idx1  (10.20.9.15)

/opt/splunk/bin/splunk edit licenser-localslave –master_uri http://10.20.9.11:8089

/opt/splunk/bin/splunk restart

This is instance license status

/opt/splunk/bin/splunk  licenser-localsalve

/opt/splunk/bin/splunk edit cluster-confg –master_uri http://10.20.9.5:8089 –mode slave –site site2 –replication_port 9887 –secret My_First_Splunk_Multi-site_Cluster

/opt/splunk/bin/splunk restart

 

Indexer  SITE 1 New York City       Site1-idx2  (10.20.9.16)

/opt/splunk/bin/splunk edit licenser-localslave –master_uri http://10.20.9.11:8089

/opt/splunk/bin/splunk restart

This is instance license status

/opt/splunk/bin/splunk  licenser-localsalve

/opt/splunk/bin/splunk edit cluster-confg –master_uri http://10.20.9.11:8089 –mode slave –site site2 –replication_port 9887 –secret My_First_Splunk_Multi-site_Cluster

/opt/splunk/bin/splunk restart

 

Indexer  SITE 2 Las Vegas    Site2-idx1  (10.20.9.15)

/opt/splunk/bin/splunk edit licenser-localslave –master_uri http://10.20.9.11:8089

/opt/splunk/bin/splunk restart

This is instance license status

/opt/splunk/bin/splunk  licenser-localsalve

/opt/splunk/bin/splunk edit cluster-confg –master_uri http://10.20.9.11:8089 –mode slave –site site2 –replication_port 9887 –secret My_First_Splunk_Multi-site_Cluster

/opt/splunk/bin/splunk restart

 

Indexer  Peer SITE 2 Las Vegas   Site2-idx2  (10.20.9.16)

/opt/splunk/bin/splunk edit licenser-localslave –master_uri http://10.20.9.11:8089

/opt/splunk/bin/splunk restart

This is instance license status

/opt/splunk/bin/splunk  licenser-localsalve

/opt/splunk/bin/splunk edit cluster-confg –master_uri http://10.20.9.11:8089 –mode slave –site site2 –replication_port 9887 –secret My_First_Splunk_Multi-site_Cluster

/opt/splunk/bin/splunk restart

 

 

Cluster Commands and Troubleshooting

ONLY RUN ON CLUSTER  MASTER

Cluster-Master      /opt/splunk/bin/show cluster status

Cluster-Master      /opt/splunk/bin/splunk rolling-restart cluster-peer

Cluster in maintenance mode

Cluster-Master    /opt/splunk/bin/splunk [ enable | disable | show ] maintenance-mode

Apply apps to all peers

Cluster-Master  /opt/splunk/bin/spluk apply cluster-bundle

Cluster-Master  /opt/splunk/bin/spluk apply cluster-bundle-status

To take Indexer Peer offline

Indexer             /opt/spunk/bin/splunk offline  

$egrep “WARN|ERROR” /opt/splunk/var/log/splunk/splunkd.log

$ /opt/splunk/bin/splunk btool check –debug

To verify once the Forwarder to sending the logs to indexers.

Indexer            index=_internal  sourcetype=splunkd host={Universal_Forwarder} TcpOutputProc

Indexer           index=_internal sourcetype=splunkd component=Metrics series={Your_Index}

NOTE:  /opt/splunk/bin/splunk edit cluster-config –cluster_label < CLUSTER_LABEL >

http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/Multisitedeploymentoverview

 

 

 

Please follow and like us: