HackBlue

Tools · SIEM and detection engineering · 21 / 24

Onboarding a new log source into a SIEM: parsing, normalization, and health checks

Plan what a new source must provide, map its fields to a common schema like ECS, ASIM, or CIM, check timestamps and time zones, and monitor the feed so silence becomes an alert.

10 min read · practice only with authorization

Plan before you ingest

Decide what questions the source answers and which fields are needed: identity, source and destination, action, outcome, timestamp. Estimate volume and retention costs. Confirm whether the source can send structured data such as JSON or syslog with structured data, which avoids fragile regex parsing.

Normalize to a schema

Normalization maps vendor field names to common names so one detection works across products. Elastic uses ECS (source.ip, user.name, event.outcome). Microsoft Sentinel uses the Advanced Security Information Model (ASIM). Splunk uses the Common Information Model (CIM) with data models such as Authentication and Network_Traffic. Pick the model your SIEM uses and map at ingest or search time.

Lab: map a firewall log

Take five sample lines from a lab firewall or a vendor documentation example. Write a field mapping table from vendor names to your schema, implement it in your SIEM’s parser or an ingest pipeline, and test with the samples. Check that the timestamp is parsed with the correct time zone by comparing event time with ingestion time.

Snippet
# example mapping (vendor -> ECS)
src       -> source.ip
sport     -> source.port
dst       -> destination.ip
dport     -> destination.port
act       -> event.action        (allow|deny)
result    -> event.outcome       (success|failure)
user      -> user.name
ts        -> @timestamp          (convert from local time to UTC)

# Splunk: compare event time and index time
index=firewall | eval lag=_indextime-_time | stats avg(lag) max(lag) by host

Feed health

Create an alert when a source sends no data for longer than expected, and watch for sudden volume drops or parsing failures. Many investigations stall because a critical log source silently stopped months earlier.

Diagram

Onboarding a log source

  1. Requirements

    Questions it answers, fields, volume, retention.

  2. Collect

    Agent, syslog, API, or cloud connector, preferably structured.

  3. Normalize

    Map fields to ECS, ASIM, or CIM; fix time zones.

  4. Monitor

    Alert on silence, volume drops, and parse errors.

Check yourself

1. Why normalize log fields?

2. Which schema does Microsoft Sentinel use for normalization?

3. Events appear five hours in the future in the SIEM. What is the likely cause?