Title

DIGITAL

Is your 'dark data' helping frontline services?

Michael Burton reports on the key findings of a PA Consulting/The MJ survey of senior managers aimed at finding out if councils are getting the most from their data.

Most of us have heard of ‘big data'. But there is also the vast wealth of data which is collected but never properly exploited. Techies call it ‘dark data' because, like mushrooms, it never properly sees the light of day even though it is collected and stored – and local authorities have volumes of it.

To explore this topic, PA Consulting and The MJ recently conducted a survey of council senior managers to find out if they are making the best out of their data.

Dark data is the data held for a single purpose, which is acquired through various computer network operations, but not fully used beyond its original capture. As a result, potential value is left unexploited.

Councils already use data to make key decisions, and the introduction of the General Data Protection Regulations (GDPR) mean there is a better understanding of what data is held by organisations. While in some cases GDPR has been seen as yet another regulation, it has also made councils more data-aware; the next step is how to use this data strategically to improve services.

Just under half of the survey respondents said data was consistently used to provide evidence for strategic decisions and that their key decision-makers were data-literate. Fifty-four per cent of the respondents stated that data was only used occasionally.

There was a more mixed picture on data analysis. Most commonly (38%) analysis was based on a wide variety of internal and external data including data from other public sector partners such as police and the NHS, with just under a quarter supplementing their data with other nationally available local authority data. A quarter used datasets from within specific service areas.

Respondents were less confident about data quality. Just 38% said it was ‘very reliable', 54% said it was only ‘somewhat reliable' and 8% even admitted it was ‘not at all reliable'. Reasons ranged from lack of capacity to maintain data sets (more on that to come) and variations caused by base data coming from different sources.

Interestingly, 77% of respondents agreed that investment in data tools and technology would improve decision-making. A similar majority said it would also provide more accurate evidence for strategic decisions while an overwhelming 93% said it would improve transparency. Asked whether such investment would equally improve public perception of their council, respondents were more circumspect, with almost 38% saying it would not and the same proportion having no view.

Respondents were much more positive about their record of sharing data with other areas of local government and the public sector (such as the police or the NHS), with 84% saying they do. However, lack of capacity to combine datasets and exploit their potential is an issue. Just under half (46%) said they disagreed with the statement ‘my service area has sufficient access to internal resources' while 38% agreed. Nor does there appear to be much likelihood of this changing, since only 15% are planning to expand their resources, either internally or externally, while over half have no plans and the rest are unsure.

As one respondent commented: ‘There is a great deal of misunderstanding about data use'. Another added: ‘Almost all of our current challenges can be informed by data insight, especially if combined with customer insight.'

One principle is important to respond to these findings – start with the decision you would like to evidence, rather than the data itself. Forget whether it is ‘big data' or small data'. It's not the size or format of the dataset that matters but how it analysed.

The key is to ask yourself about the hunches you have always wanted to prove or disprove, or the gut-based decisions that need to become evidence-based. In that way appropriate dark data analytics can be applied and the full potential of data to support better outcomes for residents can be uncovered.

DIGITAL

The unshakeable imperative: planting community and wellbeing into the DNA of new communities

By Johanne Thomas | 08 October 2025

Research by Coventry University is prompting a more conscious, evidence-based approach, allowing Urban&Civic to create spaces that truly transform how commun...

DIGITAL

DCN is 'punching above its weight'

By Martin Ford | 08 October 2025

As Sam Chapman-Allen’s tenure as chair of the District Councils’ Network (DCN) comes to an end, he talks to Martin Ford about his time in the hotseat

DIGITAL

Time to step up the pace

By Heather Jameson | 08 October 2025

Heather Jameson canvasses sector experts’ views on the New Towns Taskforce report, and finds a high level of support for a document that ‘doesn’t pull its pu...

DIGITAL

Growing London benefits everyone

By Antonia Jennings | 07 October 2025

London must not be ignored as the Government focuses on spreading devolution across England – for the good of the whole country, writes Antonia Jennings.

Michael Burton

Popular articles by Michael Burton