Welcome to Part 2 of our "Getting Started with LogQL" blog series. If you haven't read Part 1 yet, we highly recommend checking it out here to get a solid foundation in the basics of LogQL and understand the concepts covered in the previous article. In Part 2, we will dive deeper into LogQL and explore the powerful capabilities of filtering and formatting expressions.

Line Filter expressions​​

Filter expression generally checks whether the log line contains (or not) a particular text​

  • |= : contains or equals​​
  • != : not contains or equals​​
  • |~ : regex matches​​
  • !~ : regex does not match​​​

​Let's try some sample queries​

{ namespace="loki"} |= "cluster.local"​

this will filter all logs in log stream which contains "cluster.local" where the label is matching​

loki-3
{ namespace="loki"} |~ "cluster.local|loki-gateway"​

this will filter all logs in the log stream which either contains "cluster.local" or "loki-gateway" or both where the label is matching​

loki-4
{namespace=~"loki"} |~ "cluster.local" |~ "loki-gateway"​

this will filter all logs in log stream which contains "cluster.local" and "loki-gateway" where label is matching​

Line filter expressions can be inserted at any point in a log pipeline, but it is nearly always preferable to have them at the start. The query runs more quickly when they are at the start since additional processing only takes place when a line matches.

Label Filter expression

Label filter expression allows filtering log line using their extracted labels. It can contain multiple conditions.

A label filter expression has following structure

<label_identifier> <operator> <value>

The operators use in label filter expression are Comparators and these are the available comparators

  • "=" or "==" equals
  • "!=" not equals
  • and & or for clubbing conditions
  • > & >=
  • < & <=

The type of values Loki supports are

  • String can be quoted or back-ticked
  • Duration can be time in ns, us, ms, s, m, h
  • Number
  • Bytes, can be "b”, “kib”, “kb”, “mib”, “mb”, “gib”, “gb”, “tib”, “tb”, “pib”, “pb”, “eib”, “eb”.

Now let's try some sample queries

{app="loki", component="gateway"}|pattern`<ip> - - [<datetime>]  <status> "<method> <uri> <_>" <size> <_> "<agent>" <_>`|status = 400

This will first parse the gateway logs according to the pattern and filter basis on status code which is 400

loki-17
{app="loki", component="gateway"}|pattern`<ip> - - [<datetime>]  <status> "<method> <uri> <_>" <size> <_> "<agent>" <_>`|status = 200 and uri = "/"

This will first parse the gateway logs according to the pattern and filter basis on status code which is 200 and with uri "/".

loki-18

Line format expression​​

Line formate expression is used to rewrite the log line to a particular format and it is done by using the text/template format. It takes a single string parameter | line_format "{{.label_name}}", which is the template format. We can pass single or multiple labels into the formate and it's resolved into the log line. We can try this using promtail logs which are in logfmt format

{ app = "promtail" } | logfmt ​
loki-12

​Now we can try line formatting

{app = "promtail" } | logfmt | line_format "namespace is {{.namespace}}. message is {{.msg}}"

We can see that log is changed into the format we specified​

loki-13

Label Format expressions​​

The "label_format" expression can rename, modify or add labels. let's try some basic operations with label

{ app = "promtail" } | logfmt | label_format new_app=app​

this query will remove label app and add a new label new_app with the value of the app​

loki-14
{ app = "promtail" } | logfmt | label_format new_app="{{.app}}"​

this query will add a new label new_app after resolving(text/template) the values in expression​

loki-15

Label drop expression​

label drop expression can drop labels from the log. Label matching operator can be used with drop-to-drop particular labels which match the expression​

{ app = "promtail" } | logfmt | drop app

this query will drop the label app

loki-16

For matching expressions take this example, ​this will drop label app that matches the value "promtail"​

{ app = "promtail" } | logfmt | drop app="promtail"

​The data so far is returned as streams of log lines. And it cannot be used in the creation of graphs as it's not aggregated. We can aggregate the lines into numeric values, such as counts, which then become known as a scalar vector. For converting this into scalar vectors we need aggregation operators which will be explained in the next part​