Looking back, you shouldn’t have celebrated quite so vocally when you got your requested small fortune at the year start to build the company’s Security Operation Centre (SOC). March came, and that pot of gold looked less reliable as the CIO’s pet project – a cloud migration doubtless promised to sales teams as the answer to all their upsell dreams – hit the rocks. Finance went looking for where big projects had unspent budget and your pot of gold got raided.
And you now have a quandary. You had your eyes on all those Gartner leader products you diligently did proof of concepts on last year, but you just can’t afford those along with all the now rising staff costs for the new recruits.
Then someone in the IT department starts talking about how they are moving a system to a better and faster free/open source alternative…and the seed is sewn. Couldn’t you maybe save that missing million in your budget by doing the same?
A quick Google search will bring up a near infinite array of cyber-related zero cost options. So, I recently set myself the challenge in my lab environment of building a SOC platform that had no license or support costs associated. Here’s what I learned.
Challenge accepted
I started by setting out a mixture of real endpoints (mostly Centos/Rocky) that I could install agents on, and a log simulator I wrote that can create large volumes of real-looking logs from typical enterprise endpoints and Cisco network security tools that show sequences of events that would need to be correlated in a SIEM (Security Information and Event Management). This fed through to an aggregator (nxlog CE) that streamed encrypted logs through a firewall (pfSense) to a data routing layer (NiFi), which then fed data to my SIEM (OSSIM) and my archive (OpenSearch) as well as an MI toolset (TIG).
My Incident Response tools and endpoint monitoring came from SecurityOnion, which also emitted logs into NiFi. Threat Intel was managed by MISP and Yeti, and for good measure I added a case management tool (Helpy.io). Phew.
My Incident Response tools and endpoint monitoring came from SecurityOnion, which also emitted logs into NiFi. Threat Intel was managed by MISP and Yeti, and for good measure I added a case management tool (Helpy.io). Phew.
Just selecting those took a long time, as dissecting whether free/community versions include the required features (or only the licensed versions) is challenging. Predictable caveat here – this is not a recommendation by myself or BAE Systems Digital Intelligence, and products come and go so may be different now. You might also choose one of the many other options available. For example, I ruled out several tools that looked great on paper but demanded endless configuration of microservices to make them work – largely because I believe that kind of support overhead is unsustainable for most SOCs who are experimenting with open source.
Was it successful?
In short, yes. I got everything working, with logs flowing in all the right directions and with a basic setup that would be plenty good enough for many SOCs.
However, it had its issues. It was fiddly and a lot of the documentation, when it existed, was messy and contained gaps. I also had to carry out plenty of forum searching to understand how to make integrations work.
One factor to the complexity was that certain components were community editions of otherwise licensed software. The difference between the two is the removal of out-of-the-box content like integrations and pre-canned detection rules and, often, the removal of functionality that simplifies the tool’s usage (which is clearly the incentive to utilise the licensed versions). Moreover, for these you often get the source or basic rpm/deb packages from GitHub and no nice wrappers to install or maintain them. That all contributes to the technical debt you have to put back in yourself at some point.
Similarly, I faced the common issue of community product versions lacking key features compared to their licensed equivalents. Unsurprisingly, the missing bits tend to bring the most value – that’s the nature of product licensing. Backfilling using other products resulted in the whole solution being very fragmented. It’s all workable, but it took time to find a solution and integrate.
Updates and patching also provided some challenges, as free software versions usually only provide the git path to the source. This requires you to figure out how to install updates without data loss or major outage.
Should you do it?
The biggest concern I came out of this challenge with is the breadth and depth of skills needed from your engineering team to keep everything running and performant, let alone evolving the system to grow coverage and detection maturity. This is going to be an issue for most.
I was trying to get docker instances to play nicely with other local services, I had ‘make’ files that just failed, or had dependency issues all over the place. This just doesn’t fit well with me for a business-critical capability. It also will inflate salary costs and net out any savings if not very careful.
That aside – and some organisations will feel comfortable with the build and support complexity – there were also some areas where it felt completely ineffective to be trading time for license fees. I found tools that were in the single digit thousands of dollars to license, yet I was spending days making alternatives work. It would have been quicker and easier to just license – and I’d have got all the best features along with a support contact.
Similarly, some examples include NiFi, which consumed a huge effort to codify my bespoke SOC data routing whilst remaining performant, yet by the end I virtually replicated the commercial Cribl platform.
Other components just did not do what I wanted. I simply cannot find a SIEM solution that does not have a cost associated and actually delivers in a mature SOC with any kind of proportional effort. It seems that the ability to create alerts is common, but to correlate those alerts together is far less so – something that is a must for all SOCs. However, I have found several paid-for SIEM candidates that seem to do more of what I want, such as OSSIM’s commercial cousin AlienVault, or Greylog. Case Management also continues to be a headache for me, commercial or free, to support both ITIL-esque ticketing and CERT (Computer Emergency Response Team) incident investigations as one. That’s an ongoing challenge and a story for another day.
The good news is that some of the components I used, such as SecurityOnion, performed admirably. There are also plenty of other tools out there – I had to draw a line somewhere – such as TheHive, Cortex and OpenCTI that would do the job, while new tools such as n8n show how this field is evolving.
In conclusion, this is a field where many contribute but where a minority are outstanding. Rather than zero-cost, the better aspiration would be a SOC for under £50k or £100k to cover off those few areas where a small spend pays huge dividends. For that spend, you could do an awful lot and be considerably better off on licensing over three years.
But you still need to factor the unavoidably inflated staff costs driven by the added complexity for engineering and analysts, so don’t go banking that extra budget straight away.
We believe that strong digital defences come from security of both the Enterprise and the Nation
Understand the evolving threat landscape is a key part of maintaining robust defences. BAE Systems' Threat Intelligence team generate original insights through research and collaboration with customers and partners