How SOCaaS engineers optimise log costs without losing visibility
Table of Contents
At its core, a Security Operations Centre (SOC) is a function that collects logging data, continuously analyses it to look for malicious activity and then exports key insights from that data in the form of incidents to be investigated and responded to. The amount of data SOCs rely on is vast, but as environments scale, so do log ingestion and storage costs.
Without careful optimisation, these costs can spiral out of control, often becoming one of the largest ongoing expenses. The challenge is to reduce log costs without sacrificing the visibility that enables threat detection. To learn more, visit our blog on Managed SOC Costs, or view our quick breakdown of SOC costs here.
You may also find our SOC Cost Calculator useful when considering a SOC for your organisation.
When you onboard a log source into the SOC’s SIEM platform, the ingested data must be stored somewhere. In most cases, it lands in a table within your SIEM provider’s cloud storage, which charges per GB ingested and retained.
Systems that generate high log volumes can drive significant annual storage costs. The SOC must apply a robust validation process during onboarding to ensure it only ingests logs with genuine security value. Where possible, the team should filter logs to remove unnecessary data before storage.
In this blog, we explain the methods Evalian uses to maintain a careful balance between visibility and log ingestion cost.
If you want to understand more about a SOC, read our comprehensive Guide to SOC.
Understanding log security value through threat modelling
When onboarding log sources onto a SOC, the most important aspect of the whole procedure is mapping out which data sources within the environment need to be fed into the SIEM. This activity is done as part of the threat modelling process. SOCs will examine each data source and put together a priority list of areas in the environment which provide the most security value. In this context, value is measured using the following criteria:
- How well can it be used to generate SIEM rules or used in threat hunting?
- How well they can be used within forensic investigations.
Log security value can be categorised into the following categories:
High-security value logs
Logs with higher security value usually contain data which can strongly point towards malicious activity within an environment. We identify these logs based on the criticality of the attack surface they expose and how much usable data they provide. A high-security-value example is endpoint network logs, typically collected from an EDR solution.
Endpoints are a primary attack surface and a common target for attackers. Their network logs show how an attacker interacts with the device. That visibility makes this log source high value, even if it generates medium or high volumes.
Prioritise high-security-value logs for onboarding. Retain them for longer periods. They form the backbone of SOC detection and investigation.
Some logs generated by an environment may have medium security value, meaning they may contain useful information that could indicate malicious activity; however, they may not be a strong indicator by themselves, making it more likely that detections generated from them will produce false positive alerts.
Medium-security value logs
Examples of medium security value logs would be DNS resolver logs picked up from requests made by devices on an estate.
Monitoring DNS resolver logs gives a SOC visibility into your attack surface by showing which domains devices across your estate are connecting to. Though this is useful information, DNS resolver logs often lack details about traffic sent to and from a given domain. Rather, only that a given endpoint has attempted to establish a connection, even if a firewall may later block that connection. Because of the lack of a payload visible in the logs, the security value is diminished. This means it may be less valuable for use in detection rules as it can’t as strongly point towards malicious activity.
That doesn’t mean that DNS resolver logs shouldn’t be ingested at all. They can still provide valuable context for correlation during an incident. However, because their security value is typically lower than that of other sources, they should be ingested at a lower priority and filtered to control ingestion cost. See more about log filtering in the Log Filtering section later in this blog.
Low-security value logs
Finally, low-value logs/data have little security value. Whilst this means there is a possibility they can be used to produce some detections or support investigations, it is unlikely.
An example of a low-value log could be where technical limitations prevent certain useful fields from being populated, e.g. networks logs which don’t include any external IP addresses. In cases such as these, it may only be valuable to ingest logs where heavy filtering has been done to exclude irrelevant logs or send these logs to a cheaper form of storage, such as a data lake or the basic log analytics tier for users of Azure, or Cold Storage for Splunk Enterprise.
It may be more expensive to query, but the low value of the logs means that querying would not likely be done very often. More information about different tiers of log storage can be found later in this article.
Log onboarding (Sampling)
During threat modelling, the SOC temporarily ingests a log source to assess its value. For high-volume sources, the team may collect logs for an hour. For lower-volume sources, they may monitor activity for a full day.
They review the types of activity captured during that period and assess what security value the logs provide before deciding how to handle ongoing ingestion.
Summary of log value
Prioritise the ingestion of high-value logs. These logs form the backbone of how the SOC detects and investigates malicious activity across the organisation’s infrastructure.
Deprioritise medium and lower-value logs. Exclude low-value logs from the SIEM entirely if the cost of ingesting them outweighs the security benefit.
Log filtering
A common method which SOC Engineers will utilise to reduce log volume is through filtering. Each system generates raw logs in its own defined schema. That schema often includes data with little security value to the SOC. If you ingest everything, you increase storage costs without improving visibility.
Filter logs at the source wherever possible. Stop low-value events before they enter the pipeline. For example, forward only specific log types to the storage solution your SIEM pulls from.
If you can’t filter at the source, filter further down the pipeline. When logs enter the SIEM, apply a transformation to control what data ultimately lands in log storage.
In Azure, use a Data Collection Rule (DCR) to do this. A DCR lets you transform incoming data before it reaches the Log Analytics workspace. See Figure 1 for an example.
Figure 1: Data Collection Rule with transform pipeline
Something to watch out for is that there is often a trade-off between security and log ingestion. Some sources, though high or medium value, may still produce too many logs to economically store them all. In this case, the SOC should risk-assess the logs it ingests. The team should identify where it can apply meaningful filters to reduce volume without driving up costs or weakening detection coverage.
Example of how Evalian conducts log ingestion filtering
Whilst onboarding a customer onto a SIEM, our engineers ran into a common issue while onboarding their VPC flow logs (VPC logs are a data source which logs all traffic moving around the customer’s virtual private cloud). The log source was producing vast amounts of traffic, over 13 times the volume of other sources. The issue – if a decision were made to exclude this log source due to cost, it would produce a significant gap in security coverage. This was unacceptable. However, the cost of storing all the logs would be economically unviable.
To combat this issue and allow protective monitoring to still occur, engineers at Evalian conducted a threat model of the inflow of data by sampling a day’s worth of traffic from VPC Flow Logs and from that sample, documented a priority list of log events based on their security value.
In the case of this data source, the events which represented the highest security value were VPC flow traffic flowing to and from the production environment outside of the network. We considered traffic flowing between internal addresses, or to non-production IP ranges, lower value and filtered it out.
This resulted in a significant decrease in log volume ingested into Sentinel, reducing our customers’ monthly cost by £200 whilst maintaining the effectiveness of their security monitoring.
Managing log ingestion during service operation
After log ingestion has been setup there are still ways SOCs can manage the flow of data, ensuring that it remains within a tolerable amount and large increases in log ingestion don’t go unnoticed.
When onboarding a new log source, Evalian’s SOC often sets a daily ingestion cap. If log volume exceeds that threshold, the system automatically stops ingesting data to prevent excessive charges. In Microsoft Sentinel, if you exceed the daily cap on a Log Analytics Workspace, Sentinel stops all ingestion into that workspace, except for data in auxiliary tables.
This could leave you at risk of losing security coverage. To mitigate this risk, we configure alerts that warn us before log ingestion reaches the cap. Giving our engineers time to find the log tables which have an increased log volume and investigate why this is happening and how to resolve it before log ingestion stops altogether.
Ingestion caps
After a data source has been threat modelled and onboarded into a SIEM, the SOC should have already established what the expected log volume of that source will be and how much that source will cost per month. Sometimes, however, systems producing log data may have errors or periods of substantial, unexpected log volumes, which, if the SOC isn’t prepared for, could cause gigantic spikes in log costs. To combat this, SOCs will set ingestion caps on log sources to minimise the cost of large log ingestion spikes.
Log retention
The length of time an organisation chooses to retain logs for will affect how much it will cost to store them. Store logs for at least 90 days. This gives your team enough lookback to generate incidents and support investigations properly.
Beyond investigation needs, organisations often must retain logs for defined periods to meet regulatory requirements. Examples include:
- PCI-DSS – If you operate within the PCI DSS framework as a merchant (Level 1–4), service provider, or third party, you must retain logs generated by systems in the cardholder data environment (CDE) for at least one year.
- NIS 2 Directive – In certain cases, you must retain cybersecurity logs for a minimum of 18 months.
- NCSC – The NCSC recommends that you retain important security logs for at least six months.
- Telecommunications Security Act – Requires telecoms companies to retain security logs for at least 13 months.
There are options for log storage for longer periods of time for regulatory or security purposes, or for logs which provide less immediate security value; SOCs can utilise hot, warm and cold storage to achieve this.
Different types of Log storage
Another tool our SOC engineers can utilise is choosing which type of log storage to place ingested logs. You can group log storage into different categories:
- Hot Storage – Higher storage costs, cheap to query data, e.g. Sentinel Log Analytics.
- Warm/cold storage – Lower storage costs, more costly to query data, e.g. A data lake solution.
Send most logs to hot storage first. Your SIEM relies on this data to run detection rules.
However, use cold storage to extend log retention at a lower cost. You can also route logs there that won’t add value to detections but may support specific investigations or forensic requirements.
An example of a new cold storage solution for users of Microsoft Sentinel is Microsoft Sentinel Data Lake.
Microsoft Sentinel Data Lake
Microsoft have recently released into public availability a new cold storage solution for Microsoft Sentinel customers called Microsoft Sentinel Data Lake.
What is a data lake?
A data lake is a cloud storage solution which stores data in a raw, unstructured format as opposed to a structured format, as you might find in a table (e.g. log analytics table). Data stored this way will often incur less storage costs, but the costs to access that data will be greater.
In a SOC environment, increased querying costs matter less if analysts rarely need to access the data, for example, with low-security-value logs. Data lakes make long-term storage far more economical when you need to retain data for compliance or forensic reasons but expect to access it infrequently.
What is Microsoft Sentinel Data Lake?
Microsoft Sentinel Data Lake is an implementation of a data lake, but built to suit a SIEM environment, working out of the box alongside Microsoft Sentinel implementations with very little setup.
When you activate it, Microsoft automatically mirrors data ingested into Sentinel Log Analytics (hot storage) into Microsoft Sentinel Data Lake (cold storage).
As logs in hot storage reach the end of their retention period in Log Analytics, they expire there but persist in the data lake tier, where you typically set much longer retention periods.
This provides Microsoft Sentinel users a cost-efficient method of retaining logs in the SIEM for long periods of time, allowing incident investigators to utilise historical data within investigations, allowing SOCs to easily run analytics on data from a much longer period and helps organisations to meet their data retention requirements with low complexity.
Figure 2 Sentinel Data Lake data flow diagram
On top of mirroring logs from the analytics tier over to the data lake tier, it is possible with Microsoft Sentinel Data Lake to ingest logs into it from the get-go, ignoring the analytics tier altogether. Use this approach for logs that carry low security value or that you mainly retain for forensic purposes over a longer period.
For SOC teams, Sentinel Data Lake offers a clear advantage over traditional data lakes: easier access to stored data. Analysts can query data using KQL (Kusto Query Language), the same language used in the Log Analytics tier, directly from the Microsoft Defender XDR portal. As a result, SOC analysts and engineers can access and investigate data quickly without leaving the Defender or Sentinel environment.
Conclusion: How SOC teams control log ingestion costs without weakening security
Optimising log ingestion and storage is a balancing act between cost efficiency and maintaining the visibility necessary for effective threat detection.
At Evalian, we achieve this balance by carefully assessing the security value of each log source. We filter out low-value data and strategically manage ingestion and retention across different storage tiers.
By leveraging tools such as ingestion caps, sampling, and different storage tiers, we ensure that the SOC remains both cost-effective and operationally robust.
Ultimately, within a SOC environment, more data isn’t always better.
Through careful planning, ongoing optimisation, and the intelligent use of modern SIEM and storage technologies, SOCaaS engineers can deliver comprehensive visibility, regulatory compliance, and strong security outcomes without unnecessary expense.
SOCaaS supports your organisation’s security posture
If you’d like to understand how our CREST accredited SOC as a Service can help you strengthen your organisation’s security framework, get in touch with our friendly team.
