Organisations across the public and private sectors are increasingly prioritising the role of data engineers – and rightly so, says Alex Richards
I got talking about careers with my kids the other night. Normally, it’s a struggle to get them to look up from their respective screens, but in fact it was their reliance on their devices that gave me my chance.
As their latest social media addiction blared out, I asked them if they realised that such sites are silently collecting data on them; where they are in the world, their type of phone, which videos they watch and so on.
My revelation, while not startling enough for them to, you know, actually put their phones down, did give them pause. And it enabled me to point out that, as they grow up, gaining skills and knowledge about how to use and channel data will leave them in good stead when it comes to the job market.
Take data engineers, for example. While data scientists deploy scientific methods, creativity and domain understanding to extract knowledge and insights from various forms of data, the areas where firms are struggling the most – data preparation and insights implementation – are outside of most data scientists’ expertise. And that’s where data engineers come in.
Their role is rooted in constructing the architecture, systems and pipelines which can turn such insights into better customer engagement or strengthened large scale processes for millions of customers or decisions every day.
While data science has been a “hot job” for a good few years now, data engineering is increasingly taking root in both the public and private sectors too. That’s because any organisation – from utility company to government department – that has millions of customer interactions requires engineers, and plenty of them.
This is particularly acute for policymakers because data is critical to delivering effective government and public services that respond to users’ needs. For example, it can be used in real-time by frontline staff to ensure the person they are serving gets the best possible support. And predictive analytics helps anticipate demand for services or policy changes.
It is data engineers who possess the nous and knowledge to scale up and turn this vision into reality – but that doesn’t mean it’s straightforward.
From insight to impact
Data insights can transform operations by enabling powerful data-driven decisions across not only industry, but also critical-to-life services such as policing or defence. However, many organisations face significant challenges when trying to productionise this initial insight.
For example, the sheer breadth of skills required creates its own difficulties. Translating data solutions into productionised, engineered components which are secure, scalable and resilient enough to underpin critical parts of organisations’ operations is no small feat. They need to cover a whole lifecycle of activities encompassing everything from strategy and architecture to data management. This means that experience in modern data technologies, together with data science, analytical techniques and software engineering disciplines are all pre-requisites.
Then there are the levels of security, risk and operational assurance required – when analytical models are deployed in large scale business critical systems, it demands a high level of assurance together with deep customer knowledge as analytical solutions need to be pertinent enough to satisfy multiple levels of users. It is critical to make sure that insight is delivered to users in a way that suits their skills, culture and ways of working.
All this is an encouraging start but in reality that’s all it is. After all, data is the lifeblood of pretty much everything we do these days. From multinational transactions to smartphone apps, streamed entertainment to COVID-19 research, data ricochets in all directions all day, every day.
Data’s ubiquity has also heralded huge consequences for organisations large and small. Only those which have the ability to capture it, understand it and use it to power strategic and operational decisions will be able to reap the competitive and operational dividends on offer.
No wonder data engineers are now so in demand. Now, I just need to convince my kids…
About the author
Alex Richards is Head of Data, Digital & Cloud at BAE Systems Applied Intelligence
- Bringing data to the party. Caroline Bellamy is on a mission to transform how the UK Ministry of Defence uses data. She tells Mivy James about her 30-year career in industry and why data holds the key to smarter and faster decision-making across Defence
- Delivering data dividends. Mivy James examines what needs to be done to help the military be more data centric
- Creating the Navy of the future. The Royal Navy is in the midst of a concerted effort to exploit and deploy advanced data analytics and Artificial Intelligence. Hannah Green explains why it’s full speed ahead
- An information transformation. The British Army’s Chief Information Officer, Major General JJ Cole, is on a mission to digitally transform military operations from barracks to battlespace. He sits down with Mivy James to talk data, delivery and digitisation
- From silos to solutions. How can organisations tackle the age old problem of silos? Holly Armitage maps the sometimes torturous route towards joined-up collaboration
- Cloud burst: securing data cross domains. Cloud computing is now a fact of life for organisations in both the public and private sectors – but how can their data be shared securely? The solution, says Sam Neath, lies in secure cross domain…