“We are spending more money storing CloudTrail than we are on the actual infrastructure it monitors.”
If you run a security team at a growing company, you have likely had a conversation that sounds exactly like that quote.
The reality of modern security operations is that the volume of audit events is growing exponentially, while security budgets are remaining flat. You need comprehensive visibility to detect threats and pass compliance audits. But achieving that visibility often means writing massive checks to your SIEM vendor.
The Data Deluge
Consider the logging output of a standard mid-market organization. To get a baseline level of visibility, you need to collect:
- AWS CloudTrail: Every API call, management event, and role assumption.
- Identity Providers (Okta/Entra ID): Every login, MFA prompt, and token refresh.
- Code Repositories (GitHub/GitLab): Branch protections, PR approvals, and repository access.
- Endpoint Detection (EDR): Process executions, network connections, and registry modifications.
These are not small text strings. These are deeply nested, highly verbose JSON payloads. A single AWS CloudTrail AssumeRole event can be over 2KB of text. When you have hundreds of developers and thousands of endpoints, you are quickly generating terabytes of data per month.
The Traditional SIEM Trap
The traditional SIEM business model is built on ingestion volume. Vendors charge you per gigabyte of data you send them, regardless of the value of that data.
This creates a massive conflict of interest. Your vendor wants you to send everything raw. You want to keep your job, which means keeping your budget under control.
The result is usually a compromise that hurts security. Teams start dropping entire log sources (like VPC Flow Logs or EDR telemetry) because they are “too expensive.” You are forced to choose between cost and visibility, creating blind spots that attackers will inevitably find.
A Better Way: Transformation and Optimization
You cannot stop the logs from being generated, but you can control what you store. The solution to the cost crisis lies in efficient data transformation at the edge and optimized storage architectures.
At Xpernix, we approach the data problem differently to keep storage and analytics cost-effective.
1. Transform and Drop at the Edge
Before a log ever hits your storage backend, it needs to be processed. Raw audit logs are full of useless data: null values, deeply nested metadata you will never query, and duplicate events.
By placing an ingest pipeline (like Vector) in front of your storage, you can:
- Drop useless fields: Remove the 50 lines of HTTP headers in a CloudTrail event that you will never search for.
- Filter noise: Drop known-safe, high-volume events (e.g., automated health checks) entirely.
- Enrich immediately: Add GeoIP or threat intel data to the event before storage, so you don’t have to do expensive joins at query time.
2. Purpose-Built Analytical Storage
Once the data is clean, you need a storage engine designed for logs, not a generalized database.
Traditional indexing engines (like Elasticsearch) have massive overhead. The index size can often eclipse the raw data size, requiring huge clusters of RAM and CPU just to keep the lights on.
Modern analytical storage engines use column-oriented architectures with aggressive schema-less compression. This means identical JSON keys and repeated values compress down to fractions of a byte. The result is lightning-fast queries over terabytes of data, running on a fraction of the compute infrastructure.
Final Thought
Your security budget should be spent on security engineers and stopping attackers, not on storing JSON files. By implementing aggressive data transformation and modern storage architectures, you can retain full visibility across your environment without bankrupting your department.
If your current logging bill is forcing you to make bad security decisions, it is time to rethink your pipeline.
Need Help?
Xpernix can help you build cost-effective security pipelines. Reach out in your dedicated channel or book a discovery call if you want help reviewing your log storage and detection strategy.