How to create filters in data parsing?

UTMStack allows you to customize filters in data parsing:

1. Click on the hamburger positioned in the upper right part of the dashboard. Select the option Data Parsing.

2. In this panel, you will see a list of the filters implemented by UTMStack according to your user, each one of them can be modified or deleted using the options at the right, optionally you can search for a filter by name. Click on the button Add filter to create your own.

UTMStack uses filter plugins from Logstash to process a log line, map specific parts of the line into dedicated fields, and perform actions based on this mapping. Once you are in the editor by selecting the Add filter or Edit filter options, you must ensure that your code has the following structure:

filter {

#Some filter operations like grok, geoip, json, kv and so on.

}

UTMStack needs that the filters include the following output fields so that the correlation engine can analyze data.

• dataType

This field indicates what kind of log you are processing, like: AWS log, Mac OS log, o365 log, Windows Event log, iis log, NIDS, VMware ESXi, filebeat module log, etc. For naming conventions, the value of this field has to be lower case; and can only have letters, numbers, hyphen and underscore; because UTMStack uses this value to create the index on elastic to each filter created.

• dataSource

This field represents the source where the log was generated. Normally, the value of this field is an IP address, hostname or user defined string.

• [logx][utm][action]

This field has to be in the output only to indicate when a connection was stablished between a source and a destination according to the log, and it value must be "Success"

Let's see a real filter example in UTMStack: Input vmware-esxi log:

This filter takes the vmware-esxi logs and validates that data input is by Syslog and does not contain datatype field. If the condition is true applies a Grok plugin filter, matching the message with the log structure.

Then it's applied a condition to validate if the log is the vmware-esxi type, which, if true, removes the message field and adds the field dataType with the value "vmware-esxi".