Improving Performance of Detection Rules That Are Using Broad tstats Filters

[a5hk]

Summary

While investigating performance issues on our Splunk environment, I identified several detection rules from the ES Content Updates project that consistently consume excessive resources. The performance impact appears to stem from search design patterns that can be optimized. This issue summarizes the problematic patterns and lists specific rules affected.

Observed Performance Problems

1. Inefficient tstats Usage (Wide Filters + Large BY Clauses)

Many rules use a tstats search with:

• A very broad filter (e.g., Processes.file_name!="unknown")
• A BY clause containing too many fields
• A follow-up lookup, join, or secondary search to narrow the results after the fact

This leads to:

• Large result sets being produced unnecessarily
• High memory and CPU usage
• Delays in downstream searches (lookups, joins, subsearches)

Additional Impact

Including too many fields in the BY clause also causes result loss:

• Several of these fields are not required or not recommended by the CIM data model.
• These fields may be null or missing depending on the source (e.g., Windows Event Log 4688).
• Null values in several BY fields cause distinct combinations to collapse, which removes rows unintentionally.

Example:
In the Suspicious Reg exe Process rule:

• The subsearch groups results by Processes.process_name but discards it at the end using table, making the grouping unnecessary.
• The outer search includes fields like Processes.parent_process_guid in the BY clause, even though it is not CIM-recommended and not present in key data sources.

This pattern appears in several rules and results in unnecessary processing and inconsistent output.

2. Duplicate or Near-Duplicate Rules

Some rules could potentially be merged under a single analytic, as they perform very similar detections.

Examples:

• System User Discovery With whoami
• Windows System User Discovery Via quser

These could potentially be consolidated to reduce maintenance overhead and search executions.

Rules Identified as High Resource Consumers

Below is the list of rules that, in our environment, consistently show high resource usage (based on job inspector metrics and internal monitoring):

• Windows LOLBAS Executed Outside Expected Path
• Single Letter Process On Endpoint
• System Processes Run From Unexpected Locations
• Windows Defacement Modify Transcodedwallpaper File
• Windows DotNet Binary in Non Standard Path
• Windows Command and Scripting Interpreter Hunting Path Traversal
• Windows NirSoft Utilities
• Common Ransomware Extensions
• Common Ransomware Notes
• Detect Remote Access Software Usage File
• Detect RTLO In File Name
• Windows InstallUtil Remote Network Connection
• DLLHost with no Command Line Arguments with Network
• Rundll32 with no Command Line Arguments with Network
• SearchProtocolHost with no Command Line with Network
• Unknown Process Using The Kerberos Protocol

Request / Proposed Next Steps

• Review the use of large BY clauses across the rules.
• Consider removing fields that are non-CIM, unused, or sparsely populated.
• Evaluate whether post-tstats filtering can be pushed inside the tstats constraints instead of performed later.
• Examine opportunities to consolidate near-duplicate rules (e.g., user discovery analytics).

[nasbench]

Hey @a5hk thanks for reporting this. I will investigate this and report back.

Just FYI. There are already efforts going to remove overlapping / duplicate detections.

For the too many fields in the by clause. The idea here is to create a common output fields format to have a normalized output where the data that is shown is consistent across detections and in the potential of correlations between other DS / DMs the fields are readily available. I agree with the portion that some could be removed but keep in mind that when writing detection for many data sources you have to account for the variety of data sources themselves. For example ParentCommandLine is not available in 4688 but is in Sysmon EID1 or Crowd strike and others. But as I said we will explore if there i a way to reduce it to the actual useful things.

For the rest of the comments, I agree filtering should not be broad nor after tstats where possible. So I will keep you posted.

Thanks once again for raising this. Stay tuned.