Splunk Universal forwarder on Linux
Universal forwarder on Linux
Splunk universal forwarder is a best and performance reliable method to forward logs to an indexer which will act as an agent for log collection on Linux machines.
https://www.splunk.com/en_us/download/universal-forwarder
wget -O splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.0.3&product=universalforwarder&filename=splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz&wget=true’
#tar splunkforwarder-7.0.3-fa31da744b51-Linux-x86_64.tgz –C /opt
# chown –R splunker:splunk /opt/splunkforwarder
# export PATH =$PATH:/opt/splunkforwarder/bin:
#/opt/splunkforwarder/bin/splunk start –accept-license –answer=yes –no-prompt
#/opt/splunkforwarder/bin/splunk/bin/splunk enable boot-start –user splunker
#/opt/splunkforwarder/bin/splunk splunk status
#/opt/splunkforwarder/bin/splunk show splunkd-port
#/opt/splunkforwarder/bin/splunk add forwarder
NOTE: your forwarder sending the logs to four different indexers
#/opt/splunkforwarder/bin/splunk add forwarder indexer1:9997 –method autobalance -auth admin:changeme
#/opt/splunkforwarder/bin/splunk add forwarder indexer2:9997 –method autobalance -auth admin:changeme
#/opt/splunkforwarder/bin/splunk add forwarder indexer3:9997 –method autobalance -auth admin:changeme
#/opt/splunkforwarder/bin/splunk add forwarder indexer4:9997 –method autobalance -auth admin:changeme
| Directory | Information logs collects | Sending the data to Splunk Index |
| /var/log/messages | General message and system related stuff | sys_idx |
| /var/log/auth.log | Authenication logs | auth_idx |
| /var/log/kern.log | Kernel logs | Not monitor |
| /var/log/cron.log
|
Crond logs (cron job) | Not monitor |
| /var/log/maillog | Mail server logs | mail_idx |
| /var/log/qmail/ | Qmail log directory (more files inside this directory) | Not monitor |
| /var/log/httpd/ | Apache access and error logs directory | web_idx |
| /var/log/lighttpd/ | Lighttpd access and error logs directory | web_idx |
| /var/log/boot.log
|
System boot log | Not monitor |
| /var/log/mysqld.log | MySQL database server log file | db_idx |
| /var/log/secure or /var/log/auth.log | Authentication log | auth_idx |
| /var/log/utmp or /var/log/wtmp | Login records file | Not monitor |
| /var/log/yum.log | Yum command log file. | Not monitor |
| #cat /opt/splunk/forwarder/etc/system/local/inputs.conf
# Most configuration done on the forwarder
# Sets initial fields: source, sourcetype, host, index
|
#cat /opt/splunk/forwarder/etc/system/locat/props.conf
# All data modification are based on either source, sourcetype or host
# Break data into events with timestamps
|
| [monitor://var/log]
blacklist = \.(txt|gz)$ whitelist = mail.log$ index = mail_idx
|
[source:://var/log/mail.log]
sourcetype = mail
[host :: NY* ] TZ = US/EASTERN
|
| [monitor://var/log]
blacklist = \.(txt|gz)$ whitelist = auth\.log$|secure\.log$ index = auth_idx
|
[source:://var/log/secure.log]
sourcetype = secure |
| [monitor://var/log]
blacklist = \.(txt|gz)$ whitelist = mysqld.log$ index = db_idx
|
[source:://var/log/mysqld.log]
sourcetype = database |
| [monitor://var/log]
blacklist = \.(txt|gz)$ whitelist = messages$ index = sys_idx
|
[source:://var/log/messages]
sourcetype = admin |
Forwarder
#/opt/splunkforwarder/bin/splunk list monitor /var/log
#/opt/splunkforwarder/bin/splunk show splunkd-port
#/opt/splunkforwarder/bin/splunk status
#/opt/splunkforwarder/bin/splunk list forward-server
#egrep “ERROR|WARN” /opt/splunkforwarder/var/log/splunk/splunkd.log
# ping indexer_name
Indexer verify that universal forwarder make connection?
Indexer: Look for your receiving port to be open on the indexer:
#netstat –an | grep 9997
On indexer go to setting > forwarding and receiving > port 9997 [ if not enable it ]
Tcpdump port 9997data for any errors
# tcpdump –I eth0 port 9997
Next, you should run a search to find the forwarder connection on the indexer:
# index=_internal source=*metrics.log tcpin_connnections
Search: index=_internal host=forwarder_host
Search: index=_internal host=forwarder_host component=”TcpOutputProc”
