Splunk Universal forwarder on Linux

Universal forwarder on Linux

Splunk universal forwarder is a best and performance reliable method to forward logs to an indexer which will act as an agent for log collection on Linux machines.

https://www.splunk.com/en_us/download/universal-forwarder

wget -O splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.0.3&product=universalforwarder&filename=splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz&wget=true’

#tar splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz –C /opt

# chown –R splunker:splunk /opt/splunkforwarder

# export PATH =$PATH:/opt/splunkforwarder/bin:

#/opt/splunkforwarder/bin/splunk start –accept-license –answer=yes –no-prompt

#/opt/splunkforwarder/bin/splunk/bin/splunk enable boot-start –user splunker

#/opt/splunkforwarder/bin/splunk splunk status

#/opt/splunkforwarder/bin/splunk show splunkd-port

#/opt/splunkforwarder/bin/splunk add forwarder

NOTE: your forwarder sending the logs to four different indexers

#/opt/splunkforwarder/bin/splunk add forwarder indexer1:9997 –method autobalance -auth admin:changeme

#/opt/splunkforwarder/bin/splunk add forwarder indexer2:9997 –method autobalance -auth admin:changeme

#/opt/splunkforwarder/bin/splunk add forwarder indexer3:9997 –method autobalance -auth admin:changeme

#/opt/splunkforwarder/bin/splunk add forwarder indexer4:9997 –method autobalance -auth admin:changeme

Directory Information logs collects Sending  the data to Splunk Index
/var/log/messages  General message and system related stuff sys_idx
/var/log/auth.log  Authenication logs auth_idx
/var/log/kern.log  Kernel logs Not monitor
/var/log/cron.log

 

 Crond logs (cron job) Not monitor
/var/log/maillog  Mail server logs mail_idx
/var/log/qmail/  Qmail log directory (more files inside this directory) Not monitor
/var/log/httpd/ Apache access and error logs directory web_idx
/var/log/lighttpd/  Lighttpd access and error logs directory web_idx
/var/log/boot.log

 

System boot log Not monitor
/var/log/mysqld.log  MySQL database server log file db_idx
/var/log/secure or /var/log/auth.log  Authentication log auth_idx
/var/log/utmp or /var/log/wtmp  Login records file Not monitor
/var/log/yum.log  Yum command log file. Not monitor

 

#cat /opt/splunk/forwarder/etc/system/local/inputs.conf

 

# Most configuration done on the forwarder

 

# Sets initial fields:  source, sourcetype, host, index

 

#cat /opt/splunk/forwarder/etc/system/locat/props.conf

 

# All data modification are based on either source, sourcetype or host

 

# Break data into events with timestamps

 

 

[monitor://var/log]

blacklist = \.(txt|gz)$

whitelist = mail.log$

index = mail_idx

 

[source:://var/log/mail.log]

sourcetype = mail

 

[host :: NY* ]

TZ = US/EASTERN

 

 

[monitor://var/log]

blacklist = \.(txt|gz)$

whitelist = auth\.log$|secure\.log$

index = auth_idx

 

[source:://var/log/secure.log]

sourcetype = secure

[monitor://var/log]

blacklist = \.(txt|gz)$

whitelist = mysqld.log$

index = db_idx

 

[source:://var/log/mysqld.log]

sourcetype = database

[monitor://var/log]

blacklist = \.(txt|gz)$

whitelist = messages$

index = sys_idx

 

[source:://var/log/messages]

sourcetype = admin

 

 

Forwarder

#/opt/splunkforwarder/bin/splunk list monitor /var/log

#/opt/splunkforwarder/bin/splunk show splunkd-port

#/opt/splunkforwarder/bin/splunk status

#/opt/splunkforwarder/bin/splunk list  forward-server

#egrep “ERROR|WARN”  /opt/splunkforwarder/var/log/splunk/splunkd.log

# ping indexer_name

 

Indexer verify that universal forwarder make connection?

Indexer:        Look for your receiving port to be open on the indexer:

#netstat –an | grep 9997

On indexer go to setting > forwarding and receiving > port 9997  [ if not enable it  ]

 

Tcpdump port 9997data for any errors

# tcpdump –I eth0 port 9997

 

Next, you should run a search to find the forwarder connection on the indexer:

# index=_internal source=*metrics.log tcpin_connnections

Search:         index=_internal host=forwarder_host

Search:         index=_internal  host=forwarder_host  component=”TcpOutputProc

Please follow and like us: