How impactful is your distance training program?
Since Covid, more and more organisations are rolling out mobile training programs, sometimes replacing face-to-face training sessions. While these programs, when based on lightweight and inclusive formats, help firms improve training continuity at scale — they are left with a big challenge: how to measure their effective impact.
Fortunately, mobile technologies improve impact measurement just as well as they improve content distribution ; but they require the introduction of technologies and practices that can be new to an organisation, and this is a challenge. Fortunately, we have solutions!
What is Impact?
“Impact” can be many things, but it is always what we pay training for. This is why impact definition and measurement is so important for distance training programs: it could an improvement in beneficiaries’ financial performance, a boost in staff’s productivity, an advancement in service delivery, a better product, or simply a higher satisfaction rate from beneficiaries, partners, or funders. Bottom line here is that impact is (1) a positive outcome of our action (2) that needs to be measured and proven.
Why set up Impact Measurement when rolling out distance training programs?
Mobile training programs convey educational materials to beneficiaries, with the purpose of equipping them with activable skills and knowledge. This means that implementation of a training program is generally not enough of proof of its effectiveness, even with traction. It must also show specific factors of success that can be retention, behavioural change, income generation, or many more — and success should be a combination of several factors, broken down into measurable KPIs, and defined according to your objectives. But what do measuring impact this way achieve?
1. Measurement content quality & assessing delivery
Training materials and delivery are the two pillars of a successful training program. They can both make or break the outcomes of your training program, and for this reason need constant attention, and often serious iterations to make materials format and delivery mode frictionless.
Measuring the success of each, separately, (materials on one side, delivery on the other) is critical for rolling out distance training programs. Imagine your only metric is graduation rate, and that rate drops suddenly: would you be able to tell whether this is because materials turned obsolete, or because something broke in your delivery process?
An adequate impact measurement system, with KPIs set at each stage of your program and for each element, will allow you to understand how you are fulfilling your goals, know what is working best, and refine and tailor your operations over time for maximal impact. You will be able to know for a fact which materials work best with each of your target audiences ; and which delivery channel is most effective. Is it face-to-face? Is it a web-app? Is it a series of SMS? Chances are, it will depend, and it will evolve over time!
2. Cost benefit analysis & ROI
Return on Investment — often a literal dollar figure attached to each program— is an increasingly popular and strategic metric amongst nonprofits, often under scrutiny from donors, governments or partners. In this context, collecting and delivering meaningful training impact data is essential, as it is the primary material from which reporting and advocacy campaigns can be generated.
Firms who engage in mobile training need to determine whether this investment has been beneficial. For NGOs and social organizations, the cost will be compared to impact, and impact means KPIs. Impact reports will include KPIs such as overall reach, time spent, scoring, retention rate or satisfaction rate over time, broken down per age group, gender, region or any other significant cohorts, and completed with qualitative data from polls, exit questionnaires, or long term impact surveys.
This data can help organizations switching from measuring efficiency to measuring effectiveness: what is the complete extent, the full impact of their actions? We do not just save one life: we allow one person to grow, contribute to their environment and their community, generate an income, fulfil their objectives, their dreams… This is impact: this person’s experience and achievements, when measured, will produce a dollar figure — the ROI. And this dollar figure will help advocating for concentrating investments on the most impactful programs, hence maximising the impact of a limited resource: funding.
3. Reporting to partners, sponsors & stakeholders
Advocacy is a continuous effort for an organisation to retain its partners support and engagement into its actions. It is essential when ROI and long term impact can take years or decades to measure while budgets are decided every year.
Strong reporting will allow organisations to retain their stakeholders’ confidence throughout rollout — a key element notably for mobile-based distance training programs, often innovative and unprecedented, which will need multiple years of operations before research can analyse and measure their impact.
Continuous impact measurement, with automatic KPI tracking and regular milestones, will help firms to develop well-rounded reports matching their sponsors and partners’ needs.
The evaluation of training programs allows firms to develop well-rounded reports to sponsors and partners. These stakeholders who are normally prime investors require updates on how the company is seeking to improve performance, financial bottom-line and satisfying end-users.
What tools can be used to assess the impact of training programs?
Mobile technologies offer unprecedented — and mostly untapped — tracking and reporting abilities, generally highly cost-efficient, and easily scalable. Here are a few generic examples of tools you can leverage to track and measure relevant KPIs:
Mobile-based Surveys
Surveys are very effective tools for evaluating mobile training programs. They provide a means to collate both quantitative and qualitative data, at scale, with A/B testing and randomisation options. Beneficiaries can be asked questions in a variety of ways such as multiple-choice or numerical evaluation questions that can be compiled and processed automatically, or open-ended and ranking questions that give an opportunity to collect first-hand insights, at scale and seamlessly: firms can roll out these surveys at a much higher frequency than if they have to send field surveyors.
Trainees’ Usage Data
Usage data is an extraordinary and still widely untapped statistical tool to ascertain course interaction and user performance. With mobiles, firms can measure with unprecedented precision which percentage of trainees actually utilised the course content, what topics they struggled with, how much time they spent in average or individually on each page or exercice, at which time they are the most activable and efficient; and breakdown this data per age group, gender, region, language, device used, and so many more. This tracking can even be pursued for trainees that are offline, thanks to asynchronous analytics technologies.
Evaluation (Quizzes)
Assessments, quizzes and exercices are excellent tools to fulfil two goals: they provide an opportunity to both the trainee and the trainer to assess the trainee’s performance and progression ; but they also generate interaction and calls to action highly beneficial to trainees attention and motivation. Mobile training programs should have assessments and quizzes regularly spread throughout their materials, and because they are automated, adding more of them does not surcharge the trainers.
Eye-Test
After all, this is the ultimate test: are trainees applying their learnings in real life? Are they mobilising acquired information at the right time and changing their behaviour accordingly? Quizzes and assessments can be one way to measure knowledge retention, as well as post-surveys taken months or years after training ; but the best indicator of success remains unprompted action taking from the trainee. With mobile technologies, trainees can be empowered to share their stories and relate their successes — also a measurable KPI for impact measurement. Still want a literal eye-test? They can photograph their achievements.
About Chalkboard Education
Chalkboard Education provides mobile-based distance training solutions tailored for NGOs. Lightweight, offline-first and complete with full analytics capabilities, Chalkboard Education is designed for rural and hard-to-reach community training. Currently used in 10+ countries in Africa and South America, Chalkboard Education is available worldwide.
