Configure Splunk Cluster with Commands

by Posted on

BEST PRACTICE

  1. Please Plan first based on your requirements and verify your plan.
  • A single cluster or segregated clusters ( by sourcetype, department, or use case)
  1. Dedicated hardware to the master node, search head and peer nodes
  2. All members share the same license pool
  3. Each peer must have its own storage
  4. Number of peer nodes is determine by:
  • Expected availability requirements
  • Level of replication required
  • Daily data rate
  • Retention policy
  • Concurrent users
  • CAN NOT USE A DEPLOYMENT SERVER TO DISTRIBUTE CONFIGURATION BUNDLES DIRECTLY TO PEER NODES

Enable clustering on the instances in the order of

  1. Master node
  2. Peer Nodes
  3. Search Heads

[Configuration file located: /opt/splunk/etc/system/local/server.conf ]

$splunk help cluster

$splunk help [ list | edit ] cluster config

Configure Splunk License Master                                                                  

 [ CLUSTER-MASTER ] splunk start –accept-license

 [ CLUSTER-MASTER ] splunk add licenses /opt/license/splunk.license.big.license

[ CLUSTER-MASTER ] splunk restart

 Enable Single-site Indexer Cluster                                                               

 [ CLUSTER-MASTER ] splunk start –accept-license

[ CLUSTER-MASTER ] splunk show servername

[ CLUSTER-MASTER ] splunk show splunkd-port

[ CLUSTER-MASTER ] splunk show web-port

[ CLUSTER-MASTER ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ CLUSTER-MASTER ] splunk edit cluster-config -mode master -replication_factor 2 -search_factor 2 -secret myindexercluster

[ CLUSTER-MASTER ] splunk restart

 [ SITE1-IDX1 ]  splunk start –accept-license

[ SITE1-IDX1 ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ SITE1-IDX1 ] splunk enable listen 9197

[ SITE1-IDX1 ] splunk edit cluster-config -mode slave -master_uri https://CLUSTER-MASTER:8089 -secret myindexercluster -replication_port 9000

[ SITE1-IDX1 ] splunk restart

[ SITE1-IDX2 ] splunk start –accept-license

[ SITE1-IDX2 ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ SITE1-IDX2 ] splunk enable listen 9197

[ SITE1-IDX2 ] splunk edit cluster-config -mode slave -master_uri https://CLUSTER-MASTER:8089 -secret myindexercluster -replication_port 9000

[ SITE1-IDX2 ] splunk restart

 [ SITE2-IDX1 ] splunk start –accept-license

[ SITE2-IDX1 ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ SITE2-IDX1 ] splunk enable listen 9197

[ SITE2-IDX1 ] splunk edit cluster-config -mode slave -master_uri https://CLUSTER-MASTER:8089 -secret myindexercluster -replication_port 9000

[ SITE2-IDX1 ] splunk restart

 [ SITE1-SH1 ] splunk start –accept-license

[ SITE1-SH1 ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ SITE1-SH1 ] splunk edit cluster-config -mode searchhead -master_uri https://CLUSTER-MASTER:8089 -secret myindexercluster

[ SITE1-SH1 ] splunk restart

 Single-site Cluster to Multisite Cluster                                              

 [ CLUSTER-MASTER ] splunk edit cluster-config -mode master -multisite true -site site1 -available_sites site1,site2 -site_replication_factor origin:1,total:2 -site_search_factor origin:1,total:2 -search_factor 1 -secret myindexercluster

[ CLUSTER-MASTER ] splunk restart

[ CLUSTER-MASTER ] splunk enable maintenance-mode

 Configure Peers

[ SITE1-IDX1 ] splunk edit cluster-config -master_uri https://CLUSTER-MASTER:8089 -mode slave -site site1 -replication_port 9000 -secret myindexercluster

[ SITE1-IDX1 ] splunk restart

[ SITE1-IDX2 ] splunk edit cluster-config -master_uri https://CLUSTER-MASTER:8089 -mode slave -site site1 -replication_port 9000 -secret myindexercluster

[ SITE1-IDX2 ] splunk restart

[ SITE2-IDX1 ] splunk edit cluster-config -master_uri https://CLUSTER-MASTER:8089 -mode slave -site site2 -replication_port 9000 -secret myindexercluster

[ SITE2-IDX1 ] splunk restart

[ SITE2-IDX2 ] splunk start –accept-license

[ SITE2-IDX2 ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ SITE2-IDX2 ] splunk enable listen 9197

[ SITE2-IDX2 ] splunk edit cluster-config -master_uri https://CLUSTER-MASTER:8089 -mode slave -site site2 -replication_port 9000 -secret myindexercluster

[ SITE2-IDX2 ] splunk restart

[ CLUSTER-MASTER ] splunk disable maintenance-mode

[ SITE1-SH1 ] splunk edit cluster-master https://CLUSTER-MASTER:8089 -secret myindexercluster -multisite true -site site1

[ SITE1-SH2 ] splunk start –accept-license

[ SITE1-SH2 ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ SITE1-SH2 ] splunk edit cluster-config -mode searchhead -master_uri https://CLUSTER-MASTER:8089 -site site2 -secret myindexercluster

[ SITE1-SH2 ] splunk restart

 Enable Search Head Cluster                                                                        =

[ SITE2-SH1 ] splunk start –accept-license

[ SITE2-SH1 ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ SITE2-SH1 ] splunk edit cluster-config -mode searchhead -master_uri https://CLUSTER-MASTER:8089 -site site2 -secret myindexercluster

[ SITE2-SH1 ] splunk restart

[ SITE2-SH2 ] splunk start –accept-license

[ SITE2-SH2 ] splunk edit licenser-localslave -master_uri https://CLUSTER-MASTER:8089

[ SITE2-SH2 ] splunk edit cluster-config -mode searchhead -master_uri https://CLUSTER-MASTER:8089 -site site2 -secret myindexercluster

[ SITE2-SH2 ] splunk restart

[ SITE1-SH2 ] splunk init shcluster-config -mgmt_uri https://SITE1-SH2:8089-replication_port 9200 -secret shcluster

[ SITE2-SH1 ] splunk init shcluster-config -mgmt_uri https://SITE2-SH1:8089-replication_port 9300 -secret shcluster

[ SITE2-SH2 ] splunk init shcluster-config -mgmt_uri https://SITE2-SH2:8089-replication_port 9400 -secret shcluster

[ SITE1-SH2 ] splunk restart

[ SITE2-SH1 ] splunk restart

[ SITE2-SH2 ] splunk restart

[ SITE1-SH2 ] splunk bootstrap shcluster-captain -servers_list https://SITE1-SH2:8089,https://SITE2-SH1:8089,https://SITE2-SH2:8489

[ SITE1-SH2 ] splunk show shcluster-status

[ SITE1-SH2 ] splunk rolling-restart shcluster-members

[ SITE1-SH2 ] splunk rolling-restart shcluster-members -status 1

[ SITE1-SH2 ] splunk show shcluster-status

[ SITE1-SH2 ] splunk edit shcluster-config -shcluster_label my_search_cluster

 

Published by Upen Patel