How to create filters in data parsing?

Created by Juan Manuel Libera Frómeta, Modified on Fri, 17 Feb 2023 at 07:05 AM by Juan Manuel Libera Frómeta

UTMStack allows you to customize filters in data parsing:

  1. Click on the hamburger positioned in the upper right part of the dashboard. Select the option Data Parsing.
  2. In this panel, you will see a list of the filters implemented by UTMStack according to your user, each one of them can be modified or deleted using the options page1image26067568at the right, optionally you can search for a filter by name. Click on the button Add filter to create your own.

UTMStack uses filter plugins from Logstash to process a log line, map specific parts of the line into dedicated fields, and perform actions based on this mapping. Once you are in the editor by selecting the Add filter or Edit filter options, you must ensure that your code has the following structure:


filter {

#Some filter operations like grok, geoip, json, kv and so on.



As you can see in the picture above, you must provide a filter name, then in the filter definition area goes your code. At the bottom of the editor you can see the options to cancel your changes or save them. After apply a filter, you will see it in the Data Parsing panel.

UTMStack needs that the filters include the following output fields so that the correlation engine can analyze data.

  • dataType

This field indicates what kind of log you are processing, like: AWS log, Mac OS log, o365 log, Windows Event log, iis log, NIDS, VMware ESXi, filebeat module log, etc. For naming conventions, the value of this field has to be lower case; and can only have letters, numbers, hyphen and underscore; because UTMStack uses this value to create the index on elastic to each filter created.

  • dataSource
This field represents the source where the log was generated. Normally, the value of this field is an IP address, hostname or user defined string.
  • [logx][utm][action]
This field has to be in the output only to indicate when a connection was stablished between a source and a destination according to the log, and it value must be "Success"

Let's see a real filter example in UTMStack: Input vmware-esxi log:

This filter takes the vmware-esxi logs and validates that data input is by Syslog and does not contain datatype field. If the condition is true applies a Grok plugin filter, matching the message with the log structure.

Then it's applied a condition to validate if the log is the vmware-esxi type, which, if true, removes the message field and adds the field dataType with the value "vmware-esxi".

After applying the filter, these are the output fields:

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons

Feedback sent

We appreciate your effort and will try to fix the article