SPLUNK useful commands and Search

SPLUNK useful commands and Search

List of commands for the installation of SPLUNK and Searching indexes

sudo groupadd splunk

grep splunk /etc/group

sudo useradd -g splunk splunker

grep splunker /etc/passwd

(Downloading Splunk source file using wget)

wget -O splunk-7.0.3-fa31da744b51-Linux-x86_64.tgz ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.0.3&product=splunk&filename=splunk-7.0.3-fa31da744b51-Linux-x86_64.tgz&wget=true’

sudo tar zxf splunk-7.0.3-fa31da744b51-Linux-x86_64.tgz -C /opt

sudo chown -R splunker:splunk /opt/splunk

sudo ls -l /opt/splunk

sudo /opt/splunk/bin/splunk start –accept-license –no-prompt -answer

sudo /opt/splunk/bin/splunk enable boot-start -user splunker

sudo /opt/splunk/bin/splunk status

/opt/splunk/bin/splunk version

/opt/splunk/bin/splunk show web-port -auth admin:changeme

/opt/splunk/bin/splunk show splunkd-port -auth admin:changeme

/opt/splunk/bin/splunk show appserver-ports -auth admin:changeme

/opt/splunk/bin/splunk show kvstore-port -auth admin:changeme

/opt/splunk/bin/splunk show servername -auth admin:changeme

/opt/splunk/bin/splunk show default-hostname -auth admin:changeme

/opt/splunk/bin/splunk set servername SEARCH1 -auth admin:changeme

/opt/splunk/bin/splunk set default-hostname SEARCH1 -auth admin:changeme

$splunk show config conf_name

$splunk btool check

$splunk show config config_name

$splunk show config inputs

$splunk btool list conf_name –debug

$splunk btool list monitor://var/log – – debug

On Indexer: $splunk enable listen 9997

On Indexer: $splunk display listen 9997

On Deployment_Server: $splunk list deploy-clients

On Deployment_Server: $splunk reload deploy-server

On Forwarder: $splunk add forward-server Indexer:9997

On Forwarder: $spluk list forward-server

On Forwarder: $splunk remove forward-server idx:9997

ON Forwarder: $splunk set deploy-poll deployment_server:8089

On Forwarder: $splunk show deploy-poll

On the Forwarder /opt/splunkforwarder/bin/splunk set deploy-poll deployment_server:8089 -auth admin:changeme

On the Forwarder /opt/splunkforwarder/bin/splunk restart

On the Forwarder /opt/splunkforwarder/bin/splunk show deploy-poll -auth admin:changeme

ON THE DEPLOYMENT SERVER $/opt/splunk/bin/splunk list deploy-clients -auth admin:splunk

To remove all data from an index on indexer :

$ splunk clean eventdata –index index_name

Remove the file pointe for a particular soruce from the fishbucket, :

$splunk cmd btprobe –d /opt/splunk/var/loib/splunk/fishbucker/splunk_private_db –file source – reset

Recreate the idx files for a bucket :

$splunk rebuild path_to_bucket

$splunk add licenses /your-dir/licensefile.xml

$splunk list license

$splunk edit licenser-localslave –master_uri https://License_Master:8089

$splunk list licenser-localslave

$spluk edit cluster-config -mode master –replication_factor 2 –search_factor 2 –secret ‘my_cluster_secret_key’

$splunk edit cluster-config –mode master –multisite true -site site1 –availabel_sites site1, site2 -site_repliaction_factor origin:1, total 2 –secret ‘my_cluster_secret_key’

$splunk edit cluster-config –mode slave –master_uri https://CLUSTER_MASTER:8089 –secret ‘my_cluster_secret_key’ –replication_ports 9887

$splunk edit cluster-config –master_uri https://CLUSTER_MASTER:8089 -mode slave site site1 secret ‘my_cluster_secret_key’ –replication_ports 9887

$splunk add cluste-master – master_uri https://CLUSTER_MASTER:8089 –secret ‘my_cluster_secret_key’

$splunk add cluster-config –mode searchhead -master_uri https://CLUSTER_MASTER:8089 –secret ‘my_cluster_secret_key’

Indexer Cluster Commands

$splunk show maintenance-mode

$splunk enable maintenance-mode

$splunk disable maintenance-mode

Take this peer offline with enforced counts, takes peer offline permanently

$splunk offline { — enforce-counts }

$spluk apply cluster-bundle

$splunk show cluster-bundle-status

$splunk show cluster-status

Cluster_Master $splunk rolloing-restart cluster-peers

Cluster_Master $splun remove cluster-peers –peers idx1

Cluster_Master: $splunk dig –enable –rest

Search head Clustering commands

$splunk edit licenser-localsalve –master_uri https://CLUSTER-MASTER:8089

$splunk edit cluster-config –mode searchhead –master_uri https://CLUSTER-MASTER:8089 –site sit1 –secret ‘my_cluster_secret_key’

$splunk restart

$splunk bootstrap shcluster-captain –server list http://search_head1:8089, http://shearch_head2:8089,http://shearch_head3:8089, http://search_head4:8089

$splunk show shcluster-status

$splunk rolloing-restart shcluster-members –status

$spunk edit shcluster-config –shcluster_label search_head_cluster

$splunk edit shcluster-config –conf_deploy_fetch_url

$splunk show shcluster-status

$splunk list shcluster-member

$splunk rolling-restart shcluster-members

$splunk apply shcluster-bundle

$splunk remove shcluster-member

$splunk disable shcluster-config

$splunk remove shcluster-member –mgmt_uri https://SH:8089

SH CLUSTER captain $splunk diag

Maintenance mode for Indexer cluster

$splunk [ show | enable | disable ] maintenance-mode

$splunk apply cluster bundle automatically invoke maintenance mode

$splunk rolling-restart automatically invoke maintenance mode

Cleaning up excess replicas bucket:

$/opt/splunk/bin/splunk list excess-buckets (index)

$splunk remove excess-buckets (index)

$splunk rebalance cluster-data –action start –index (index)

$splunk rebalance cluster-data –action status

$splunk rebalance cluster-data –action stop

$splunk edit cluster-config –rebalance_threshold 0.90

$splunk edit cluster-config –summary_replication true


On Cluster Master $splunk validate cluster-bundle

On Cluster Master $splunk apply cluster-bundle

On Cluster Master $splunk show cluster-bundle status

Search for events on all “hosts” servers for accesses by the user “root”. It then reports the 20 most recent events.

host=* eventtype=access user=root

Search across all public indexes.


Search across all indexes, public and internal.

index=* OR index=_*

if you often search for failed logins

“failed login” OR “FAILED LOGIN” OR “Authentication failure” OR “Failed to authenticate user”

Web access errors from the beginning of the week to the current time of your search (now).

eventtype=webaccess error earliest=@w0

Web access errors from the current business week (Monday to Friday).

eventtype=webaccess error earliest=@w1 latest=+7d@w6

Subsearches must be enclosed in square brackets in the primary search.

Consider the following search.

sourcetype=access_* status=200 action=purchase [search

sourcetype=access_* status=200 action=purchase | top limit=1 clientip | table clientip] | stats count, dc(productId), values(productId) by clientip

index=_internal earliest=-15m latest=now

The following search example is attempting to return the bytes for the individual indexes.

index=_internal source=*license* type=usage | stats sum(b) BY index

In this search the stats portion of the search is commented out.

index=_internal source=*license* type=usage `comment(“| stats sum(b) BY index”)

Web access errors from the last full business week

eventtype=webaccess error earliest=-7d@w1 latest=@w6

Display customer interactions and retrieve the only by clientip

sourcetype=access_combined* | fields clientip

Display the action, productId, and status of customer

sourcetype=access_combined* action=* productId=* | table action, productId, status

sourcetype=access_combined* action=* productId=* | table action, productId, status | rename productId as “Product ID” , actions as “Customer Purchase”, status as “HTTP Status Code”

Display and ports

sourcetype=linux_secure port “failed password” | rex “\s+(?<ports>port\s\d+)” | top src port

sourcetype=linux_secure port “failed password” | rex “(?i) port (?P<port>[^ }+” | top port

sourcetype=linux_secure port “failed password” | erex Port examples=”4940,4608,4920” | top port

Display the top mail domains from sourcetype=cisco_esa

sourcetype=cisco_esa | rex field=mailfrom “@(?<maildomain>.*)” | top limit=10 maildomain

Most IP or Top product selling in 24 hours.

sourcetype=linux_secure password fail* | top src

sourcetype=access_combined action=purchase status=200 | top product_name

Display the count of retail sales made yesterday

sourcetype=vendor_sales | stats count as “Retail Sales”

Count the number of events

sourcetype=access_combined* | stats count(actions)

How many unique website visited

sourcetype=cisco_wsa_squid | stats dc(s_hostname)

Display the quantity of sales by product name and price

sourcetype=vendor_sales | stats count as quantity by product_name, price

Which website have employees accessed

sourcetype=cisco_wsa_squid | stats list(s_hostname) by cs_username

sourcetype=access_combinde* action=purchase | timechart count(product_name) by categoryId

sourcetype=access_combinde* action=purchase | chart count(product_name) by categoryId

sourcetype=vendor_sales | geostats latfield=VendorLatitude longfiled=VendorLongitude count by product_name

sourcetype=acess_combined* action=purchase | stats sum(price) as count | gauge count 0 10000 20000

Display the errors that our host produce

sourcetype=access_combined* status>299 | chart count over status by host

Display the transactions that failed for each product from the shopping cars online

sourcetype=access_combined* status>299 | chart count over host by itemId

Sourcetype=access_combinded product_name=* | timechart span=30m count by product_name

sourcetype=access_combined status=4* clientip=”″

Indexer verify that universal forwarder make connection?

Indexer: Look for your receiving port to be open on the indexer:

#netstat –an | grep 9997

On indexer go to setting > forwarding and receiving > port 9997 [ if not enable it ]

Tcpdump port 9997data for any errors

# tcpdump –I eth0 port 9997

Next, you should run a search to find the forwarder connection on the indexer:

# index=_internal source=*metrics.log tcpin_connnections

http://indexer:8000 Search: index=_internal host=forwarder_host

http://indexer:8000 Search: index=_internal host=forwarder_host component=”TcpOutputProc”

Splunk server has indexed events to verify

Search index=_internal host=”Your_host” component=”TcpOutputProc”

Search: sourcetype=vendor_sales

Search: sourcetype=access_combined*

Search: sourcetype=access_combined* action=purchase

Search: sourcetype=access_combined* | table clientip, host, action, status

Search: sourcetype=access_combined* action=purchase | table clientip, host, status

Search: sourcetype=cisco_esa mailto=*

Search: sourcetype=cisco_esa mailto=* | erex domain example=”yahoo.com. Hotmail.com”

Search : sourcetype=access_combined* action=purchase | timechart span=2

Search: sourcetype=access_combined* (action=remove OR action=purchase)

Search: sourcetype=access_combined* (action=remove OR action=purchase) | stats sum(price) as totalSales by action, product_name

Search: sourcetype=access_combined* status>399

Search: sourcetype=access_combined* status>399 | chart count by host, status

Search: sourcetype=access_combined* status>399 | chart count by host, status limit=5

Search: sourcetype=access_combined* status>399 | timechart count(action) by host

Search: sourcetype=access_combined* clientip=* status>399

Search: sourcetype=access_combined* clientip=* status>399 | dedup host, clientip

Search: sourcetype=access_combined* clientip=* status>399 | dedup clientip, host

“Brute Force Access Behavior Detected” correlation search without extreme search commands:

| `datamodel(“Authentication”,”Authentication”)` | stats values(Authentication.tag) as tag,count(eval(‘Authentication.action’==”failure”)) as failure,count(eval(‘Authentication.action’==”success”)) as success by Authentication.src | `drop_dm_object_name(“Authentication”)` | search failure>6 success>0 | `settags(“access”)`

| `datamodel(“Authentication”,”Authentication”)` | stats values(Authentication.tag) as tag,count(eval(‘Authentication.action’==”failure”)) as failure,count(eval(‘Authentication.action’==”success”)) as success by Authentication.src | `drop_dm_object_name(“Authentication”)` | search success>0 | xswhere failure from failures_by_src_count_1h in authentication is above medium | `settags(“access”)`

How to Inspecting Buckets

Search: | dbinspect index=name span or timeformat

Display a chart with the span size of 1 day, using the command line interface (CLI)

| dbinspect index=_internal span=1d”

Default dbinspect output for a local _internal index.

| dbinspect index=_internal

Check for corrupt buckets

Use the corruptonly argument to display information about corrupted buckets, instead of information about all buckets.

The output fields that display are the same with or without the corruptonly argument.

| dbinspect index=_internal corruptonly=true

Count the number of buckets for each Splunk server

Use this command to verify that the Splunk servers in your distributed environment are included in the dbinspect command.

Counts the number of buckets for each server.

| dbinspect index=_internal | stats count by splunk_server

Find the index size of buckets in GB

Use dbinspect to find the index size of buckets in GB.

For current numbers, run this search over a recent time range.

| dbinspect index=_internal | eval GB=sizeOnDiskMB/1024| stats sum(GB)

Deleting Events you need to have can_delete role

delete command to make the unwanted data not to show up in searches
Index=web host=myhost source=access_combined_wcookie | delete

splunk clean eventdata indexname wipes out all data from the index

How can you tell your indexer is working


index=_internal LicenseUsage idx=fwlog”

To check the license usage, search

Index=_internal Metrics series=”fwlog” | stats sum(kbps)

index=_internal Metrics group=”per_sourcetype_thruput” series=access* | timechart span=1h sum(kb) by series

index=_internal Metrics group=”per_sourcetype_thruput” series=access* | timechart span=1h sum(kb) by series | sort – sum(MB)

Determine how many active sources are being indexed.

Search | dbinspect index=main OR index=fwlog OR index=oslog OR index=weblog OR index=applog

How to calculate the data compression rate of bucket

Search : | dbinstpect index=main OR index=[your_inedex]

| were eventCount > 10000

| fields index,id,state,eventCount,rawSize,sizeOnDiskMB,sourceTypeCount

| eval TotalRawMB=(rawSize / 1024 / 1024)

| eval compression=tostring(round( sizeOnDiskMB / TotalRawMB * 100, 2 )) + “%”

| table index, id, state, sourceTypeCount, TotalRawMB, sizeOnDiskMB, compression