In the world of observability, the ability to monitor, troubleshoot, and optimise systems and applications is crucial. Observability engineers are those who are responsible for these systems in every organizations. As an observability engineer, your job is to ensure that your organisation's systems and applications are running smoothly and that any issues are identified and resolved quickly. And they have to identify the gaps and resolve those gaps in systems to make it bullet/fault proof.
In this blog, we'll explore some of the use cases for ChatGPT in observability which will make the life of a observability engineer more easy.
Generating Log patterns / Grok / Regex from logs
Logs are a critical component of observability, providing engineers with valuable information about the behaviour of systems and applications. Each systems logs in different format and because of that logs can be difficult to analyse, particularly when dealing with large volumes of data. One way to make log analysis easier is to generate log patterns, also known as grok or regular expressions (regex), from logs.
A common problem with log parsing is regex. Writing regex takes lot of time. Let’s see if chatGPT can make it easier or not!
ChatGPT as a large scale language model have enough capabilities to understand a text and read through it. For now we can copy the sample prompt into chatGPT and see how it reacts.
User :I want you to act as a Log parser. I will give you 3 or more sample logs each line by line as input. You should generate following1. Grok pattern to parse the log2. Log pattern which you have identified3. Regex for parsing the logsDo not write explanations on replies. The output should be in following way in which it should display grok : {grok of logs} , pattern: {log pattern}, regex: {regex} line by line.
chatGPT :
Sure, I can help you with that. Please provide me with the sample logs and I'll generate the grok, log pattern, and regex for parsing the logs.
User :66.249.65.159 - - [06/Nov/2014:19:10:38 +0600] "GET /news/53f8d72920ba2744fe873ebc.html HTTP/1.1" 404 177 "-" "Mozilla/5.0 (iPhone; CPUiPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)"66.249.65.159 - - [06/Nov/2014:19:10:38 +0600] "GET /news/update HTTP/1.1" 200 177"_" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26(KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible;Googlebot/2.1;+http://www.google.com/bot.htmlj")66.249.65.159 - - [06/Nov/2014:19:10:38 +0600] "POST /news/index.html HTTP/1.1" 301177 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26(HTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible;Googlebot/2.1;+http://www.google.com/bot.htmlj"66.249.65.159 - - [06/Nov/2014:19:10:38 +0600] "GET /news/redirect.html HTTP/1.1" 500177 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26(HTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible;Googlebot/2.1;+http://www.google.com/bot.htmlj"
chatGPT :grok: %{COMBINEDAPACHELOG} pattern: %{} %{IPORHOST} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) (?:-|%{QS:referrer}) %{QS:agent}regex: ^(?<host>%{IPORHOST}) (?<identd>[^ ]) (?<user>[^ ]) (?<timestamp>[^])] "(?<verb>[^ ]) (?<request>[^ ]) HTTP/(?<httpversion>[^"])" (?<response>[0-9]+) (?<bytes>[0-9]+|-) "(?<referrer>[^"])" "(?<agent>[^"])"$
The output results were interesting. Writing regex is one of the boring tasks in the life of an observability engineer, with chatGPT its a piece of cake.
Choosing observability stack
An observability engineer interact with multiple observability tools in his daily life which helps him in analysing this data. In the market now there are many tools which can do this. As there are many tools, it's often very difficult to select a stack which can solve the problems. ChatGPT is trained with the data in the internet which includes the blogs/documentations about these tools as well.
Let’s see if chatGPT can suggest us a stack !
User :
I want you to act as a Observability engineer. I will give you 3 or more requirements you need to full fill as input. The inputs will be in a format i want {requirement} or i don't want {requirement}. Using the inputs i want you to suggest a Observability stack which can satisfy the requirements. The output should be in bullet points and one bullet points should describe about one solution. Do not write Larger explanations on replies.
chatGPT :
Sure, please provide me with the requirements in the format you prefer.
User :
I want to monitor a Microservice application running in kubernetes
I dont want any paid tools and looking for self hosted solutions
I want to Collect logs, metrics from the application
I want a retention of 1 month and daily ingestion will be 500MB
chatGPT :
Based on the requirements you have provided, here are some suggestions for an
Observability stack that can satisfy your needs:
• Prometheus + Grafana + Fluentd: Prometheus is an open-source monitoring system
that can be used to collect metrics from Kubernetes clusters. Grafana is an open-
source visualization tool that can be used to create dashboards and alerts based on
the metrics collected by Prometheus. Fluentd is an open-source log aggregator that
can be used to collect logs from Kubernetes clusters. All of these tools can be self-
hosted and are free to use. The retention of data can be set as per requirements and
data ingestion can be set up using log exporters.
• ELK Stack: The ELK stack is a popular open-source log management platform that
consists of Elasticsearch, Logstash, and Kibana. Elasticsearch is used to store and
index logs, while Logstash is used to collect and filter logs. Kibana is used to visualize
and analyze the logs. The ELK stack can be deployed on Kubernetes and is self-
hosted. The retention of data can be set as per requirements and data ingestion can be
set up using log collectors.
• Jaeger: Jaeger is an open-source distributed tracing system that can be used to
monitor microservices running in Kubernetes clusters. It can be used to trace requests
across multiple microservices and can help identify performance bottlenecks. Jaeger
can be self-hosted and is free to use. It can be used in conjunction with Prometheus
and Grafana to provide a comprehensive observability stack.
When we tried the prompt with ChatGPT, instantly it switched context to an observability engineer and gave what required.
We have only seen few use cases where chatGPT can help observability engineers work more easier. At this point we can say that this could change the way we monitor and understand complex computer systems. In the next blog we will discuss few more use cases like capacity planning, querying, etc. Please stay tuned for more upcoming content.
