Title

DIGITAL

Is your 'dark data' helping frontline services?

Michael Burton reports on the key findings of a PA Consulting/The MJ survey of senior managers aimed at finding out if councils are getting the most from their data.

Most of us have heard of ‘big data'. But there is also the vast wealth of data which is collected but never properly exploited. Techies call it ‘dark data' because, like mushrooms, it never properly sees the light of day even though it is collected and stored – and local authorities have volumes of it.

To explore this topic, PA Consulting and The MJ recently conducted a survey of council senior managers to find out if they are making the best out of their data.

Dark data is the data held for a single purpose, which is acquired through various computer network operations, but not fully used beyond its original capture. As a result, potential value is left unexploited.

Councils already use data to make key decisions, and the introduction of the General Data Protection Regulations (GDPR) mean there is a better understanding of what data is held by organisations. While in some cases GDPR has been seen as yet another regulation, it has also made councils more data-aware; the next step is how to use this data strategically to improve services.

Just under half of the survey respondents said data was consistently used to provide evidence for strategic decisions and that their key decision-makers were data-literate. Fifty-four per cent of the respondents stated that data was only used occasionally.

There was a more mixed picture on data analysis. Most commonly (38%) analysis was based on a wide variety of internal and external data including data from other public sector partners such as police and the NHS, with just under a quarter supplementing their data with other nationally available local authority data. A quarter used datasets from within specific service areas.

Respondents were less confident about data quality. Just 38% said it was ‘very reliable', 54% said it was only ‘somewhat reliable' and 8% even admitted it was ‘not at all reliable'. Reasons ranged from lack of capacity to maintain data sets (more on that to come) and variations caused by base data coming from different sources.

Interestingly, 77% of respondents agreed that investment in data tools and technology would improve decision-making. A similar majority said it would also provide more accurate evidence for strategic decisions while an overwhelming 93% said it would improve transparency. Asked whether such investment would equally improve public perception of their council, respondents were more circumspect, with almost 38% saying it would not and the same proportion having no view.

Respondents were much more positive about their record of sharing data with other areas of local government and the public sector (such as the police or the NHS), with 84% saying they do. However, lack of capacity to combine datasets and exploit their potential is an issue. Just under half (46%) said they disagreed with the statement ‘my service area has sufficient access to internal resources' while 38% agreed. Nor does there appear to be much likelihood of this changing, since only 15% are planning to expand their resources, either internally or externally, while over half have no plans and the rest are unsure.

As one respondent commented: ‘There is a great deal of misunderstanding about data use'. Another added: ‘Almost all of our current challenges can be informed by data insight, especially if combined with customer insight.'

One principle is important to respond to these findings – start with the decision you would like to evidence, rather than the data itself. Forget whether it is ‘big data' or small data'. It's not the size or format of the dataset that matters but how it analysed.

The key is to ask yourself about the hunches you have always wanted to prove or disprove, or the gut-based decisions that need to become evidence-based. In that way appropriate dark data analytics can be applied and the full potential of data to support better outcomes for residents can be uncovered.

DIGITAL

Adult social care: Rolling up our sleeves on reform

By Phil Holmes | 11 November 2026

Writing in the week of this week's National Children and Adult Services Conference, Phil Holmes says it is time to meet the Prime Minister’s commitment to ad...

DIGITAL

Children's services: Homes, not placements

By Sukriti Sen | 11 November 2026

Sukriti Sen explains how Coventry has kept children out of unregistered care.

DIGITAL

EXCLUSIVE: Too few adult social care directors for new unitaries, ADASS warns

By Neil Merrick | 11 November 2026

There may not be enough directors of adult social services to lead all the unitary councils being created through reorganisation, the president of the Associ...

DIGITAL

The devolution scrutiny challenge

By Joe Lepper | 11 November 2026

As mayoral strategic authorities take on new powers and their number is set to grow, is audit and scrutiny robust enough to keep pace and should MPs be invol...

Michael Burton

Popular articles by Michael Burton

-->