What we don’t know: Rethinking evaluation in creative practice

Words by our Future Creative Leader, Elsa Monteith

What happens when evaluation stops being a box-ticking exercise and becomes a deeper kind of listening? This reflection draws on learnings from Lighthouse’s youth arts programmes and a serendipitous clipboard discovery in an Edinburgh gallery, to explore how we might rethink impact through presence, absence, and the stories that sit in between.

Following a spell in Edinburgh in spring 2023, I spent some time in the Fruitmarket, a playful arts and exhibition space accompanied by a quiet cafe and a small gift shop at the foot of the staircase. Having spent time observing a large, tent-like bear canopy strung up to the rafters, I came across a clipboard with a brief description of what was crudely termed “Data Diversity”. Whilst I have my reservations about the use of the word “diversity” in place of perhaps the more explicit descriptors of race, gender, and other protected characteristics that they are actually trying to articulate, I was struck by how comprehensive, thorough, and thoughtful this approach to data collection was, and wanted to learn more.

‘what if data collection didn’t feel like bureaucracy, but instead felt like an invitation?’

Later that same year, I had a call with Caspar (of Studio Caspar) who was the mind behind this initiative. He shared some background to his work within and beyond public art installations, community engagement, and interactive design, and provided much inspiration for my own understanding of ‘inclusive’ (or perhaps, more ‘welcoming’) data collection, and how we structure the metrics of success (or perhaps, calculate “impact”).

Caspar’s work as a Resident Entrepreneur with Fruitmarket, and his ongoing practice through Studio Caspar, centres on a quietly radical question: what if data collection didn’t feel like bureaucracy, but instead felt like an invitation? During his work with Fruitmarket, Caspar developed the digital tool called “Data Diversity” that I spied on the clipboard that spring, a deceptively simple survey app that emerged from a rigorous process of public consultation and expert input. It stood out not just for what it asked, but how it asked it: with openness, humour, and a refreshing departure from the checkbox-heavy language of traditional audience monitoring.

The app was part of a broader investigation into the friction between engagement and measurement, specifically, how arts organisations can better understand who’s walking through their doors (and who isn’t), without reducing people to ticked boxes. This led to the development of the Personality Machine, a structured, yet flexible service design product created for organisations seeking more human, and more textured ways of evaluating audience experience. It sits on three pillars: Empowered Language, Human Centred Design, and Exchange of Value. Together, these principles push against the idea of audience data as a one-sided extraction process, and instead frame it as an ongoing, mutual dialogue.

Caspar’s thinking felt especially resonant in the context of my work with Lighthouse, where we’ve been asking similar questions: how can we make evaluation feel less like a formality, and more like an extension of the creative process itself? The Personality Machine model helped clarify how careful design, through language, visuals, and tone, can become a tool for inclusion. Its emphasis on celebrating representative (moving away from the prescriptive language of “diverse”) identities on their own terms, rather than forcing them into pre-set categories, offered both a provocation and a practical framework.

This approach is informed by the ethos of Creative Informatics, a Scottish funding and research programme that supports the intersection of data, creativity, and experimentation.

Creative Informatics explores data not as a cold, corporate asset but as something deeply pliable and personal, with their supported projects showing that data, when treated with care, can become a site of connection, not just collection. Something that I took on with these evaluations.

These conversations, between artists, organisations, and communities, are reshaping what evaluation can be. They remind me that measuring impact isn’t about fixing outcomes in place, but rather about staying open to the evolving, subjective, and often surprising ways people experience art, mentorship, and connection.

One of the most interesting ideas that emerged from evaluating both Lighthouse Young Creatives and Future Creative Leaders was what we learn from what we don’t know. Or more specifically, what it means when there’s an absence of information.

For many, the programmes were undoubtedly successful – there are stories of growing confidence, new networks, creative breakthroughs, and professional achievements that speak volumes. But for a smaller group of participants, the story is less clear.

Some dropped off halfway through. Some didn’t return after a few sessions. Others didn’t maintain contact with Lighthouse once the programme ended, despite thoughtful follow-ups and continued invitations.

‘But what about the people we didn’t hear from?’

It’s easy, in evaluations, to focus on the loudest, most complete stories. The ones where growth is visible, where feedback is offered, where the journey is traceable. But what about the people we didn’t hear from? In traditional evaluation models, silence often gets interpreted as neutrality, or worse, as success by default. But absence doesn’t equal impact. Sometimes it just means we didn’t quite reach someone in the way we hoped.

This isn’t a unique problem. It’s something that sits at the heart of most mentorship and even education-based models, when engagement fades, outcomes slip into the unknown. But I think acknowledging this is important. There’s value in naming what we don’t know, instead of pretending that neatly packaged data tells the whole story.

In one of our interviews with the Lighthouse team, someone offered a helpful analogy that’s stayed with me since. They described a kind of triangle, with three corners representing: total accessibility, deep inclusivity, and a well-run programme that operates within team capacity. “You can maybe have two of those,” they said, “but not all three; not all the time.” That framework felt honest, and also quietly generous. It made space for the messiness of real-life programming, where trade-offs are inevitable, and perfection is a myth.

So much of this work is about holding contradictions: celebrating the big wins, while also holding space for what didn’t quite land. We have to count the missing voices, not to diminish the success of others, but to better understand the full scope of what success and failure actually look like. Or maybe to stop thinking of them as binaries at all.

Read each of our Future Creative Leaders' personal reflections on an aspect of the programme in the digital booklet below.