Splunk Universal forwarder on Linux

Universal forwarder on Linux

Splunk universal forwarder is a best and performance reliable method to forward logs to an indexer which will act as an agent for log collection on Linux machines.


wget -O splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.0.3&product=universalforwarder&filename=splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz&wget=true’

#tar splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz –C /opt

# chown –R splunker:splunk /opt/splunkforwarder

# export PATH =$PATH:/opt/splunkforwarder/bin:

#/opt/splunkforwarder/bin/splunk start –accept-license –answer=yes –no-prompt

#/opt/splunkforwarder/bin/splunk/bin/splunk enable boot-start –user splunker

#/opt/splunkforwarder/bin/splunk splunk status

#/opt/splunkforwarder/bin/splunk show splunkd-port

#/opt/splunkforwarder/bin/splunk add forwarder

NOTE: your forwarder sending the logs to four different indexers

#/opt/splunkforwarder/bin/splunk add forwarder indexer1:9997 –method autobalance -auth admin:changeme

#/opt/splunkforwarder/bin/splunk add forwarder indexer2:9997 –method autobalance -auth admin:changeme

#/opt/splunkforwarder/bin/splunk add forwarder indexer3:9997 –method autobalance -auth admin:changeme

#/opt/splunkforwarder/bin/splunk add forwarder indexer4:9997 –method autobalance -auth admin:changeme

Directory Information logs collects Sending  the data to Splunk Index
/var/log/messages  General message and system related stuff sys_idx
/var/log/auth.log  Authenication logs auth_idx
/var/log/kern.log  Kernel logs Not monitor


 Crond logs (cron job) Not monitor
/var/log/maillog  Mail server logs mail_idx
/var/log/qmail/  Qmail log directory (more files inside this directory) Not monitor
/var/log/httpd/ Apache access and error logs directory web_idx
/var/log/lighttpd/  Lighttpd access and error logs directory web_idx


System boot log Not monitor
/var/log/mysqld.log  MySQL database server log file db_idx
/var/log/secure or /var/log/auth.log  Authentication log auth_idx
/var/log/utmp or /var/log/wtmp  Login records file Not monitor
/var/log/yum.log  Yum command log file. Not monitor


#cat /opt/splunk/forwarder/etc/system/local/inputs.conf


# Most configuration done on the forwarder


# Sets initial fields:  source, sourcetype, host, index


#cat /opt/splunk/forwarder/etc/system/locat/props.conf


# All data modification are based on either source, sourcetype or host


# Break data into events with timestamps




blacklist = \.(txt|gz)$

whitelist = mail.log$

index = mail_idx



sourcetype = mail


[host :: NY* ]





blacklist = \.(txt|gz)$

whitelist = auth\.log$|secure\.log$

index = auth_idx



sourcetype = secure


blacklist = \.(txt|gz)$

whitelist = mysqld.log$

index = db_idx



sourcetype = database


blacklist = \.(txt|gz)$

whitelist = messages$

index = sys_idx



sourcetype = admin




#/opt/splunkforwarder/bin/splunk list monitor /var/log

#/opt/splunkforwarder/bin/splunk show splunkd-port

#/opt/splunkforwarder/bin/splunk status

#/opt/splunkforwarder/bin/splunk list  forward-server

#egrep “ERROR|WARN”  /opt/splunkforwarder/var/log/splunk/splunkd.log

# ping indexer_name


Indexer verify that universal forwarder make connection?

Indexer:        Look for your receiving port to be open on the indexer:

#netstat –an | grep 9997

On indexer go to setting > forwarding and receiving > port 9997  [ if not enable it  ]


Tcpdump port 9997data for any errors

# tcpdump –I eth0 port 9997


Next, you should run a search to find the forwarder connection on the indexer:

# index=_internal source=*metrics.log tcpin_connnections

Search:         index=_internal host=forwarder_host

Search:         index=_internal  host=forwarder_host  component=”TcpOutputProc