BLOG

The Evolution of Measuring Training Performance

Measuring the effectiveness of training programs is evolving. Historically, the learning and development industry has looked at four basic pillars of measurement based on the Kirkpatrick Model of Training Effectiveness. These include the learner’s reaction to training, the transfer of knowledge, the application of new behaviour and the overall organizational impact. Capturing this information was laborious and time-consuming and typically only organizations that cared deeply about their learning cultures would make the effort to understand the impact of their investment into training.

This method has evolved into a modern system where crucial training data and variables are stored in a Learning Record Store (LRS), a data warehouse that stores and illuminates a rich variety of data that extends beyond simple course completions. By capturing granular details of the learning journey and its real-world application, organizations can now gain a comprehensive understanding of a training program’s impact.

A fundamental component of the data stored in an LRS is the protocol xAPI, which stands for Experience Application Programming Interface. It follows a basic structure of Actor + Verb + Object and is a natural evolution from the traditional SCORM protocol. This format allows for the tracking of a wide array of learning experiences, both formal and informal. The benefit of xAPI is the ability to capture data from a wide variety of learning modalities, from the classroom to eLearning, into immersive experiences like augmented and virtual reality. This allows organizations to aggregate data into an overall holistic look into their learning culture.

With the acceleration of artificial intelligence, we are now able to capture and harness this information in real time, giving learners more relevant and authentic learning experiences and organizations a closer look at their workforce, but before we get into that, let’s look at the data needed to properly measure training performance.

Learner Engagement and Experience Data

Understanding how learners interact with training materials is the first step in assessing efficacy. If learners are not engaged, it’s unlikely the desired learning outcomes will be achieved. Key variables to capture include:

Content Interaction: Track how learners engage with various learning materials.
- Verbs: experienced, viewed, played, interacted with
- Objects: video, simulation, eLearning module, article, specific slide or page
- Example: “Jane did not listen to the video, and she buried the video behind another window on her screen.”

Time and Duration: Monitor the time spent on specific learning activities to gauge effort and identify potential areas of difficulty.
- Variables: duration of a video watched, time spent on a quiz, total time in a course.
- Example: “John spent 45 minutes on the 5 minute simulation.”

Navigation and Pathing: Analyze the paths learners take through content to understand their learning preferences and identify areas where they may be struggling or seeking additional information.
- Example: Tracking if a learner revisits a particular module multiple times.

Social Learning and Collaboration: Capture interactions in collaborative learning environments.
- Verbs: posted, commented on, shared, asked
- Objects: forum post, chat message, shared document
- Example: “Carlos commented on Jane’s forum post in the course group page.”

Knowledge Acquisition and Skill Development Data

The next layer of data should focus on whether the learner has grasped the intended knowledge and skills. This moves beyond engagement to measure comprehension and competence.

Assessment and Quiz Data: This is a direct measure of knowledge gain and should be as detailed as possible.
- Verbs: attempted, passed, failed, scored
- Variables: raw score, scaled score, individual question responses, number of attempts.
- Example: “Maria scored 85% on the quiz.”

Pre- and Post-Training Assessments: Comparing scores from before and after a learning intervention provides a clear measure of knowledge lift.
- Example: Storing both “Maria scored 45% on the pre-assessment” and “Maria scored 85% on the post-assessment.”

Confidence and Self-Efficacy Ratings: Ask learners to rate their confidence in applying new skills before and after training.
- Verbs: rated
  Objects: a specific skill or competency
- Example: “David rated his confidence in as 3 out of 5 before the training and 5 out of 5 after.”

Simulations and Performance-Based Tasks: Capture detailed interactions within simulations to assess practical skill application in a controlled environment.
- Verbs: completed, failed, successfully performed
- Objects: specific tasks within a simulation
- Example: “Susan successfully performed the ‘de-escalation’ step in the ‘Difficult Customer Conversation’ simulation.”

Behavior Change and Application Data

The ultimate goal of most training is to change on-the-job behavior. An LRS can be a powerful tool for tracking this transfer of learning.

On-the-Job Observations: Data from managers or peers observing the application of new skills can be sent to the LRS.
- Verbs: observed, demonstrated, applied
- Example: “Supervisor observed Mark applying the proper preventative maintenance procedure on the rail car.”

Project and Task Completion: Track the successful completion of real-world tasks that require the application of learned skills.
- Example: “The new product launch, which required new skills from the operator training course, was completed on time.”

Informal Learning: Capture learning that happens outside of formal training, which can indicate a commitment to continuous improvement.
- Verbs: read, watched, attended
- Objects: an industry article, a relevant webinar, a community of practice meeting
- Example: “Monica read an article on ‘The Future of AI in Marketing’.”

Performance and Business Impact Data

Connecting learning data with business outcomes is the holy grail of learning analytics. By integrating the LRS with other business systems, you can directly measure the impact of training on organizational key performance indicators (KPIs).

Operational Efficiency: Correlate maintenance training with asset management.
- Example: Tracking parts repair vs replace costs. Or asset utilization vs human capital efficiency.

Customer Satisfaction Scores: Link customer service training to improvements in safety or Net Promoter Scores (NPS).
- Example: A decrease in negative customer reviews for agents who completed a communication skills workshop.

Productivity and Efficiency Metrics: Measure the impact of process or software training on output and efficiency.
- Example: A reduction in call handling time in a call center after training on a new system.

Safety and Compliance Data: In relevant industries, connect safety training with a decrease in workplace incidents or compliance breaches.
- Example: A decrease in conflicts for operators who completed a de-escalation skills simulation. By thoughtfully capturing and analyzing these diverse data points and variables within an LRS, organizations can move beyond simply tracking learning activities to truly understanding and proving the efficacy of their learning and development initiatives.

Data Analytics and AI

Artificial Intelligence is quickly disrupting the learning and development industry. The opportunity in this disruption comes from using this data for a better human learning experience. We are now at a point where we can learn from this data in real time and offer learners valuable insight into their own personalized learning journey. This adaptive learning journey will allow a person to stay safer, learn faster, and be empowered to have a more successful career path based on their own goals, all leading to higher organizational performance. It truly is the dawn of a new era in learning and development.

Where to Start

At Xpan, we often get asked where to start on the journey of building and measuring a sound learning culture. Every organization is on their own path and typically has a unique need. Some do not have the infrastructure and interoperability to capture data, some do not look at organizational outcomes, while others are heavily regulated and have specific legal requirements or binding labor agreements that limit data gathering.

The good news is that we can help, no matter where a client may be on this journey. There is no one-size-fits-all all for LRS reporting systems, and to truly provide organizational value, measuring training performance requires a custom and empathic view of the workforce, one human at a time. This takes us beyond the LRS system towards a more holistic view of data analytics.

Since 2001, our research has shown that a truly successful knowledge experience is built upon a solid organizational culture, relevant learning principles that maximize the transfer of knowledge, and a conduit between knowledge and learner that maximizes the user experience. All these variables need to be considered for one person at a time, a truly human-centered approach. If these elements are cared for, the lagging indicators of training performance take care of themselves. Every time.

At Xpan, we believe in a simple, practical approach to sustainably building a stellar learning culture, one that works for each client. We continue to remain agnostic in the implementation of educational technology, understanding that our clients know their business far better than us. We are grateful that we are able to join their journey of building great learning experiences and impacting learners to support advancements in both the learner and the organization.

Reach out if you want to learn more about how Xpan can help you on your journey.

TRUSTED BY GREAT BRANDS