Part 9. Log Normalization

Applications and Vendors and Custom Logs, oh my!

Walkthrough Video

Intro

Throughout this series we have started ingesting Wazuh logs from our Windows and Linux endpoints (PART 4) and from a pfSense Firewall (PART 7) into our SIEM stack.

Log Ingestion

In PART 6, we created our first dashboard within Grafana to detail Network Connections made by our endpoints.

Network Connections Dashboard

However, this only details network connections made by our Windows endpoints.

So What’s The Problem?

Dashboards are built by selecting our key fields from our raw logs to display via Grafana Panels. For example, the key field detailing the destination IP address that our Windows endpoint connected to is stored within data_win_eventdata_destinationip .

data_win_eventdata_destinationip

Where if we look invoke a network connection via our Linux endpoint, we notice the destination IP is stored within the key field: data_eventdata_DestinationIp .

data_eventdata_DestinationIp

While the difference is subtle, this makes our lives as SOC engineers difficult because we cannot create universal dashboards or alerts that pertain to network connections made by either a Windows or Linux endpoint.

We would have to double our efforts to create 2 Dashboards (Linux and Windows Network Connections) and 2 Alerts (Linux and Windows Network Connections). We need a way to standardize our log fields so that common fields, such as fields containing destination IP addresses, are always written to the same field name regardless of the log source (Linux / Windows / MacOS / etc.)

Log Normalization To The Rescue

To combat this issue, we take advantage of Graylog’s Pipeline rules to rewrite our unique field names, such as data_win_eventdata_destinationip and data_eventdata_DestinationIp , to a universal field name dst_ip .

Create a Pipeline Rule

Using a pipeline rule, we instruct Graylog to create a new field, dst_ip , that contains the same value of that we received via our data_win_eventdata_destinationip and data_eventdata_DestinationIp . This then allows us to create just 1 dashboard that refelcts just contents of the dst_ip field rather than having to create 2 dashboards for our data_win_eventdata_destinationip and data_eventdata_DestinationIp fields.

  1. Select System -> Pipelines -> Manage rules
Manage Rules

2. Create a LINUX SYSMON EVENT 3 LOG NORMALIZATION rule:

rule "LINUX SYSMON EVENT 3 LOG NORMALIZATION"
when
$message.rule_group1 == "linux" AND $message.rule_group3 == "sysmon_event3"
then
set_field("process_image", $message.data_eventdata_image);
set_field("process_id", $message.data_eventdata_processId);
set_field("protocol", $message.data_eventdata_protocol);
set_field("user_name", $message.data_eventdata_user);
set_field("dst_ip", $message.data_eventdata_DestinationIp);
set_field("dst_port", $message.data_eventdata_destinationPort);
set_field("src_ip", $message.data_eventdata_sourceIp);
set_field("src_port", $message.data_eventdata_sourcePort);
end

3. Apply then Save and Close

4. Create a WINDOWS SYSMON EVENT 3 LOG NORMALIZATION rule:

rule "WINDOWS SYSMON EVENT 3 LOG NORMALIZATION"
when
$message.rule_group1 == "windows" AND $message.rule_group3 == "sysmon_event3"
then
set_field("process_image", $message.data_win_eventdata_image);
set_field("process_id", $message.data_win_eventdata_processId);
set_field("protocol", $message.data_win_eventdata_protocol);
set_field("user_name", $message.data_win_eventdata_user);
set_field("dst_ip", $message.data_win_eventdata_destinationIp);
set_field("dst_port", $message.data_win_eventdata_destinationPort);
set_field("src_ip", $message.data_win_eventdata_sourceIp);
set_field("src_port", $message.data_win_eventdata_sourcePort);
end

5. Apply then Save and Close

6. Select the GeoIP Pipeline that we configured in PART 6.

7. Add a new stage and select our newly created rules

Viewing Results

Now let’s generate a network connection via our Linux endpoint to 8.8.4.4 and view our results within Grafana.

data_eventdata_DestinationIp
dst_ip

The above 2 images display that Graylog has now added our new normalized field dst_ip with the same value that was contained within the data_eventdata_DestinationIp field.

Let’s do the same on our Windows endpoint.

data_win_eventdata_destinationIp
dst_ip

The above 2 images display that Graylog has now added our new normalized field dst_ip with the same value that was contained within the data_win_eventdata_destinationIp field.

We can now build a search query for dst_ip:8.8.4.4 to see any Linux or Windows endpoint that invoked a network connection to 8.8.4.4 .

search query

Before our field normalization, we had to write 2 different search queries to achieve the same results, data_win_eventdata_destinationIp:8.8.4.4 and data_eventdata_DestinationIp:8.8.4.4 .

Conclusion

Unfortunately, there is no common standard when it comes to how developers and vendors output common field names within their logs. This results in increased time and effort for our SOC analysts to pinpoint activity that want to quickly spot, such as the question What endpoints connected to IP address 8.8.4.4?

With Graylog’s Pipeline rules we enforce log normalization to follow our own internal naming convention and set standards to how SOC analyst can run searches and view results within the SIEM platform. So what are you waiting for? Take control over your SIEM logs no matter of the log source with Log Normalization today! Happy Defending 😄.

Need Help?

The functionality discussed in this post, and so much more, are available via SOCFortress’s Professional Services. Let SOCFortress help you and your team keep your infrastructure secure.

Website: https://www.socfortress.co/

Professional Services: https://www.socfortress.co/ps.html

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
SOCFortress

SOCFortress is a SaaS company that unifies Observability, Security Monitoring, Threat Intelligence and Security Orchestration, Automation, and Response (SOAR).