Data, data everywhere, too much for us to link?
It’s been said that good ideas are the backbone of good government. I don’t disagree, but nowadays we can surely add data to that list. Its sheer ubiquity, combined with its potential to help deliver better citizen outcomes, means that it has taken firm root across all levels of the public sector.  
Of course, data is not brand new – policymakers have long relied on it to account for things like spending levels and welfare recipients – but today it is now increasingly used to design citizen-centric public services, generate sweeping operational efficiencies and much else besides. But as government adapts to this new data-driven environment, a number of consistent themes and challenges are emerging.

The data age

Data is itself now seen as an asset. As technology advances, and government’s reservoirs of data grow ever deeper, data has become a value generator. Digital transformation is now being used as the catalyst for putting data at the heart of the organisation, as opposed to being a transactional by product of operation.
Perhaps not surprisingly given data’s growing use, issues surrounding governance and ethics are increasingly coming to the fore. As requirements on data protection and data privacy become more stringent, ensuring data is both fit for purpose and used appropriately is now a key priority – and rightly so.
At the same time, though, the pressure to deploy machine learning, artificial intelligence and automation is intensifying. These advances remain something that our clients in government think they ought to be doing – the result, I think, of some effective marketing and the perception and peer pressure of see others are doing it.
Don't get me wrong, these exciting technologies certainly have their place. But key challenges arise when the right data management and governance isn’t yet in place.  And in their rush to deploy these advances, there is a risk that organisations haven’t worked out what such capability is or isn’t good for, and whether there is an ethical case for using it.
We’re also seeing government organisations increasingly explore the use of federated search of data. If only a subset from a larger pool of data is required, federated search enables linking and joined up analysis, without moving the data itself. Whilst easing the governance burden, as data owners retain control of their data at the most granular level, the opportunity to explore ever greater volumes of data increases
And of course, organisations continue to hunger for fresh data insights – that ability to unlock value from multiple datasets when analysed as a whole. Although such insights are predicated on the quality of the data being analysed, using machine learning and automation where appropriate allows much more significant volumes of data to be parsed, generating more insights, more quickly, and hopefully leading to better citizen outcomes.
But of course, it’s not always smooth sailing – several challenges continue to adorn the horizon.


We’re seeing an increasing number of conversations around the technology and how to enable teams to get to government data at a national, regional and local level. Making the right choice and then extracting the most value from it remains a challenge.
The perennial issue of legacy IT, the systems and processes which predate this explosion in data, has often led to a spaghetti of linked data sets in spreadsheets, databases and myriad repositories. As organisations try to further exploit cloud and digital transformation, this data and information spaghetti is under a fiercer spotlight and significant investment is required to generate some clarity amidst this complexity. 
As efforts to exploit the latent value in their new data assets intensify, the question also arises as to whether or not it is the right thing to do, and if it is, can it be done in a way that protects privacy and upholds ethics. COVID 19, certainly in healthcare, saw the advent of the COPI notice, where government enabled data sharing on a massive scale. While this was a significant improvement in many ways, it also came at a potential cost of eroding public trust.
The challenge is how to manage this new opportunity in a secure and ethical way that doesn’t destroy confidence. In our experience, whilst the will to share may be there, it is often the longest pole when moving at pace; it takes considerable time and effort to establish the correct governance to ensure that trust and confidence isn’t lost.
And finally, skills and tradecraft remain in short supply. Many of our clients have a desire to be more self-sufficient; but recognising where there are capability gaps is crucial and making the best use of the capability available is key.
We have increasingly found that ambition and ability are imbalanced and without the right skills, compromises and mistakes are often made. These skills are in ever greater demand and, as my colleague Alex Richards has pointed out, it is a shared challenge across the public and private sectors to have the right data skills in the right place at the right time.

Buckle up

Such challenges, however, are no reason to press pause. By their very nature, data-led transformations rarely run in a straight line. On the contrary, they are often more akin to a zig zagging rollercoaster fuelled by an ever-shifting array of technological advances.
And so as we start to slowly emerge, blinking, into the light of a post-pandemic world, now is the ideal time to redouble our efforts to use data for the public good. The prize of better outcomes, more effective government and tailored public services awaits.

About the author
Andy Lethbridge is Head of Consulting, central government, at BAE Systems Applied Intelligence:

Explore more content from our Global Executive Client Forum

Explore more content from our Global Executive Client Forum

At our recent Global Executive Client Forum, we brought together leaders from around the world to discuss the trends, threats and technologies which are reshaping our world around us. Check out our range of new content sourced from the event.
Find out more

Recommended reading

  • Using data to secure the border. Learn how the efficient use and sharing of data can help create the most effective border in the world
  • Bringing data to the party. Caroline Bellamy is on a mission to transform how the UK Ministry of Defence uses data. She tells Mivy James about her 30-year career in industry and why data holds the key to smarter and faster decision-making across Defence
  • Delivering data dividends. Mivy James examines what needs to be done to help the military be more data centric
  • We need to talk about data engineering. Organisations across the public and private sectors are increasingly prioritising the role of data engineers – and rightly so, says Alex Richards
  • Tuning up data trust. How can governments generate greater trust when it comes to data? Nicola Eschenburg says it can be done, and the sooner the better
  • Homing in on data-driven government. Statistical models, data, and analytics have always loomed large in Andy Gregory’s in-tray, but now he’s putting his expertise to good use at the UK’s Home Office. He tells Dylan Langley about his eclectic career and adjusting to life as a senior civil servant

Subscribe to Government Insights

Please enter your email address to opt-in and receive our Government Insights.

Thank you for your subscription to Government Insights.

Andy Lethbridge

Global Consulting Director and Head of Consulting, Central Government, BAE Systems Applied Intelligence