Attributed metadata history

Before It Reaches Sentinel: How Your Own Pipeline Can Silently Drop Security Evidence

How to detect Azure Monitor transformation drops before they create blind spots in your security monitoring.Every hunt, detection rule and investigation starts with one assumption: the data in your workspace shows what actually happened.But it doesn’t always. It only shows the data that made it through the pipeline.That pipeline has a filter that many security teams have never reviewed. They may not know how much data it drops or monitor it for changes. If someone changes the filter, important events may stop reaching your tables. No alert fires because the data needed to trigger it is missing. Those events were dropped before they were ever stored.The good news is that dropped data leaves clues, just not where you would normally look.This article explains where the filter lives, why it sits outside the SOC’s usual view and which signals you can monitor to catch both malicious changes and simple configuration mistakes.Understanding Data TransformationsIn Azure Monitor and Microsoft Sentinel, data does not go straight from the source into your workspace. It first passes through a Data Collection Rule (DCR).The DCR decides what data to collect, how to change it and where to send it.One part of the DCR is the transformation. This is a KQL query that runs on every record as it arrives, before the record is written to a table. Microsoft defines the order clearly the transformation runs after the source sends the data and before it reaches the destination.Transformations are useful. You can drop noisy data you do not want to pay to store, remove columns you do not need or reshape logs into a common format. Microsoft recommends them for cost control, and in normal use they make sense.Here is the part of the DCR that matters:{ "dataFlows": [ { "streams": [ "Microsoft-Syslog" ], "destinations": [ "MyWorkspace" ], "transformKql": "source | where SeverityLevel != 'info'", "outputStream": "Microsoft-Syslog" } ]}The key line is transformKql. It is a KQL query stored in the configuration and every record must pass through it.In this example, it drops informational syslog messages. That is cheap, sensible and something nobody would question.But watch what happens when one extra line is added.The one-line blind spotStart with a transformation nobody would look at twice. It drops debug logs.source | where LogLevel != "DEBUG"Add one more condition.source| where LogLevel != "DEBUG"| where SourceIP != "10.20.20.20"Every event from that IP is now gone. It is not hidden in a table you can still query and it is not moved to a cheaper storage tier. It is removed before anything is written.No detection rule can alert on it, because rules query tables and the record never reached the table.An attacker using that IP address becomes invisible. And the thing hiding them is just one line in a configuration file.A transformation can quietly remove one column that your detection depends on.In that case, the events still arrive. The row count looks normal and nothing appears missing at first glance. But the field your rule relies on is blank in every record. The detection still runs, finds nothing and reports that everything is healthy.The Blind Spot Before the SOCA normal attacker with admin rights has to take action, generate logs and then clean up afterwards. That cleanup is still an action and it leaves traces. Purging a table, changing retention or deleting records are all things a decent SOC can detect.A transformation works differently. It acts before the log is created. There is no “after” to clean up.The evidence is not deleted. It is never stored in the first place. And you cannot write a detection for the deletion of something that never reached your tables.That is why whoever controls the DCR sits above the SOC not inside it.The SOC studies the data it receives. The DCR decides which data reaches the SOC. One controls

This record contains metadata and attribution only. It does not host, download, unpack, render, or execute malware samples.