Evidence Engine Overview

The Qualified platform takes an evidence-based approach to targeting specific competencies through the use of its Evidence Engine.

What is the Evidence Engine?

The Evidence Engine is based on the idea that the evaluation process produces data, and that data can be used to signal things about candidates. Those things are often times skill based, such as if a candidate is able to productively work with a technology such as React, Java, or SQL. Those things can also be non-skill based, such as signaling if a candidate is a good culture fit, or if they are motivated to do the type of work involved in a given role.

The purpose of the engine is to allow your team to design challenges that fit your process, by opening up a number of "signals" that can be translated into meaningful understandings about the candidates you wish to evaluate. A signal could be how many unit tests a candidate passed on a coding challenge, how many answers they got correct on a Q&A challenge, how they are scored by your own team, or even things like how quickly they were able to solve the coding task.

The engine is a tunable AI that assists your challenge design by allowing you to configure an advanced set of tunings that determine how a candidate is scored on each challenge. For each challenge the engine enables you to select which signals are important, target specific subject areas through subscores, set minimum thresholds (cut scores), and weight the important of each signal and how it impacts various scores.

We call it an engine because it opens up a whole world of flexibility in how you conduct your evaluation process. It allows you to create structure in your process and it drives that process to help you get maximum efficiency and output out of it.

What is considered evidence?

Evidence is considered anything which can produce a signal that can be scored by the engine. There are currently two key types of evidence in the system. More may be added later:

• Candidate Solutions
• Reviewer Scorecards

Candidate solutions are scored when their assessment result is submitted. Reviewer scorecards are scored when a review is submitted by one or more members of your team.

What is it designed to measure?

Criterion Referenced

The ultimate purpose of the Evidence Engine is to measure competency and job fitness. The measurements taken are criterion referenced, and have a "perfect score". This places the focus on determining if a candidate meets minimum expectations for what is necessary to succeed on the job. It also means that there is a chance that a candidate can get a perfect score, which always represents their performance related to the criteria covered on the assessment. The measurements are not norm-referenced. Perfect scores show that the candidate met the full objectives assessed for that item, but later we will cover how cut scores can be used to set minimum thresholds that should be met. A perfect score is not always required.

Here is a good article that introduces the differences between norm and criterion referenced tests.

Competency Measurements

When we mean competency, we refer to the skills and abilities that a candidate is expected to have in order to perform their duties effectively within a given role. This is the primary function of Qualified. Coding challenges focus on coding competency and can include problem solving, decision making, and many other abilities. Q&A challenges can be knowledge focused (multiple choice) or response focused (essays) and can therefore test many additional competencies (including non-technical skills such as communication).

When designing a challenge, it is important to have a plan for which competencies it should target. You can configure the Evidence Engine to report on those competencies through the use of subscores.

Job Fitness Measurements

Qualified can also be used to measure job fitness, such as scoring a candidates pre-requisite experience, skills, or work preferences. You can utilize Q&A challenges to setup weighted questions, with the most ideal responses being a perfect score. Typically when using job-fit measurements, candidates will not be expected to achieve a perfect final score.