If you are looking for an advanced filter with Lua script, jump to Fluent Bit Modify Nested JSON log with Lua script

Fluent Bit allows users to modify log data through a Modify filter with conditions. This is can be used for multiple reasons like filtering data to reduce noise, parsing logs, removing unwanted fields, or mutating some fields. See the examples below.

We are using below configuration file for testing

[INPUT]
  Name tcp
  Port 8888
[OUTPUT]
  Name stdout
  Match **

This configuration will accept input in TCP protocol so it can be tested with a curl request and output is configured to stdout

Fluent Bit Modify filter

Fluent bit modify filter can do multiple things. We shall look into some important ones.

Rename Key

Rename filter can rename a key in parsed log, for example, a key "status" can be renamed to "status_code"

Sample input

{"status" : 404,"message" : "url note found"}

Filter for renaming keys

[INPUT]    Name tcp    Port 8888[FILTER]    Name modify    Match *    Rename status status_code    Rename message status_message[OUTPUT]    Name stdout    Match **

Sample Output

{"status_code" : 404,"status_message" : "url note found"}

Remove Key and Value

Remove a key and its value. This can be done with a condition as well.

Example 1

Remove the status code field without condition

[FILTER]    Name modify    Match *    Remove_Wildcard status_code

Example 2

Remove the status message field if the status code is 200

[FILTER]    Name modify    Match *    Condition Key_Value_Equals status_code 200    Remove_Wildcard status_message

Sample Input

{"status_code" : 200,"status_message" : "Success"}

Sample Output

{"status_code" : 200}

Change Value

Filter can change value of a key with or without condition. See an example of replacing value "0" with "False"

Example

[FILTER]    Name modify    Match *    Condition Key_Value_Equals status 0    Set status False

There is a problem when the key is in a nested structure as following

{ "status":{   "code":0,   "message":"error",  }}

Advanced Filtering with Lua script

If you want to replace 0 in the above sample with False, you need to write a custom Filter using LUA script, see the details in Fluent Bit Modify Nested JSON log with Lua script

For more Basic Filter examples and configuration parameters, see the official documentation.

TLDR;

  • How to use filter in Fluent Bit to modify fields
  • Examples for Rename field, Modify with condition, etc.