The Cart Shed impact management project
Presenting the new theory of change at the Cart Shed summer celebration.
Case study: Data maturity
Client: The Cart Shed
Date: 2018-19
Work: Developing data maturity
INTRODUCTION
Data Orchard worked with The Cart Shed over one transformational year to focus, align, and mobilise the organisation towards greater impact.
The Cart Shed is a charity that enables people to manage their mental/physical health and well-being in a sustainable way. It operates outdoors in the natural environment and provides therapeutic support, creative and practical woodland/horticulture training, and group activities. It works with adults, young people and children including veterans and their families.
THE PROBLEM
The Cart Shed approached Data Orchard because they were:
Unclear about the impact they were having and lacked the knowledge and skills to measure it
Unable to extract any reliable or meaningful analysis for decision making despite investing huge amounts of staff and participant time capturing data
Frustrated with increasing numbers of unwieldy spreadsheets
Struggling with old and unreliable digital tools and systems
Unconfident about their data protection and security policies and practices.
At the outset of the project we asked all the staff to describe how they felt about data. This word cloud captures the essence of people’s attitudes. They knew it was important and useful but it was really DIFFICULT!
WHAT WE DID
We worked with The Cart Shed team (staff, trustees and volunteers) and through a series of workshops, meetings, and training sessions, we:
Reviewed the key areas of data capability (Uses, Data, Analysis, Leadership, Culture, Tools and Skills), identifying strengths, weaknesses and priorities.
Looked under the bonnet to assess digital tools and systems and produced a strategic plan for improvement.
Assessed staff roles, knowledge and skills.
Conducted a data audit.
Redefined the target population, inclusion/exclusion criteria, and referral processes for their beneficiaries.
Created a ‘Theory of Change’ and robust set of outcomes measures (short and long term) for evaluating impact.
Collated an evidence base of external research from government, academia, and local data relating to the needs and problems the charity addresses.
Implemented new digital tools and systems: standardised software, security, virus protection with charity licencing; migrated, cleaned and organised 30,000 files to a secure cloud-based shared storage solution; and advised on options and requirements for a new CRM (Customer Relationship Management) system.
Established new policies, practices and processes around the protection, security and management of beneficiary data.
Supported funding applications to secure financial resources.
Sourced suppliers and appropriate not-for-profit sector professional networks to connect them into good sources of knowledge and expertise.
THE IMPACT
Data Orchard’s support has enabled The Cart Shed to establish clarity and consistency on how to 'tell their story' - about who they serve, what they do, and how they make a difference. Moreover, the project established the critical foundations (digital tools, processes, skills and knowledge) for them to continue building their data capabilities to both prove and improve their impact in the future.
Chart showing all staff responses (10 staff) using our data maturity assessment ‘before’ and ‘after’ support from Data Orchard.
Improved overall data maturity
At the outset of the project our data maturity assessment identified three priority areas: data, skills and tools.
Twelve months later the re-assessment showed there had been improvement across all seven themes of which skills was the most notable.
The right data
At the heart of the project, most importantly, was getting the right data.
The key steps we went through were to:
Co-produce a new theory of change and set of measures for capturing outcomes
Establish key questions for analysis to test the theory
Developed a set of key performance metrics with staff and trustees.
Increased capacity, knowledge, and skills
Our initial review of staff roles and skills showed data was part of everyone’s job yet no one had any formal responsibility for it. Staff skills and knowledge in impact measurement was mixed and generally low. We ran a series of workshops and training sessions with the team and supported Katie, the CEO, to negotiate new responsibilities for existing staff e.g. managing digital tools, systems and licencing.
Clinical impact in the future will be overseen by the Clinical Director and measured and reported against the Theory of Change. Another manager took on responsibility for the implementation of a new CRM (Customer Relationship Management) system with support from a technically skilled volunteer. A new compliance officer was recruited to support consistency and accuracy in data collection, and to ensure all documents are filed and protected appropriately. In addition, new connections and relationships were established with external communities of practice and suppliers to ensure ongoing support and continued learning.
Boxplot chart showing how staff rated their knowledge and skills in impact measurement before and after Data Orchard’s support. The coloured box represents the middle range of responses, the ‘x’ is the mean average, and the horizontal line is the median average response.
The chart above is interesting, as it shows an increase in the average (both mean and median), and that the range of responses has reduced. i.e. it’s not just one or two individuals who have increased their knowledge and skills, but there’s been an overall shift in the organisation as a whole.
Better digital tools
Our data maturity assessment showed tools were the charity’s weakest area.
A follow-up comprehensive review of digital tools and systems revealed:
much of the hardware was old and poor quality
software and licencing was patchy and inconsistent
data was inefficiently stored and difficult/impossible to share
there were many disparate systems for collecting and managing data.
Our recommendations set out a prioritised 3-5 year plan for building a robust, secure, reliable digital infrastructure.
Work to improve things began immediately:
New software, security and protection systems
Migration to a secure cloud file sharing platform
New digital devices for capturing data out in the woods
A new CRM system, plus plans for a wide range of other tools and systems to improve efficiency in information and communications.
Funding secured
Data Orchard’s independent assessment of needs and support with funding applications led to The Cart Shed securing £70,000 towards the cost of their impact management project, and improving their data, digital tools and systems.
Cultural change
At the end of the project we asked staff again how they felt about data. The word cloud here, compiled from their comments, suggests ‘Useful’ and ‘Important’ are most prominent and whilst things are less ‘difficult’, data can still be seen as a necessary evil and a bit frustrating!
Perhaps most promising is a culture of ‘team approach’ to data. All the staff, trustees, and volunteers understand the organisation’s desired impact and why data is important. Almost all will be engaged in some way in the process of collecting, managing, or using data as part of their daily roles.
LOOKING FORWARD
The key foundations are now in place for The Cart Shed to continue their journey towards data maturity. It’s going to take time for new tools, systems, and impact measurement plans to become fully embedded before there’s enough reliable data for analysis.
There remain frustrating challenges with unreliable rural broadband and limitations of the CRM system (look out for our forthcoming provocation paper on the world of not-for-profit CRMS!)
The Cart Shed will spend the next 12 months:
Embedding new tools, in particular the new CRM system
Training and supporting staff to capture the right data in routine ways
Upgrading computers and other devices for collecting data
Ensuring these are properly supported.
Investing in long-term improvement will continue to take stamina, resources, and time. But they are able to clearly see the progress that’s been made already and have a road-map for the future.
“I believe we will go from strength to strength. The knowledge and understanding about what impact is now flows through the entire team.
The experience has enabled us to have different kinds of conversations with commissioners (using the Theory of Change) with greater clarity about our purpose and a marked improvement in our overall professionalism.
We will continue to challenge and review, adapt, adopt and change as necessary. I hope we continue to grow and enable those we work with to live more wholeheartedly.”
FIND OUT MORE
You can follow the journey we travelled together in more detail by reading the four blog posts we published over the 12 months: