Signals

A signal is data that can be used as a measurement to signal competency or job fitness for a candidate.

Signals are measured and then mapped to various output scores. The two main types of output scores the Qualified Score, and Subscores. The Qualified Score gives you a way of assessing candidate's overall process, while subscores give you a way to understand specific competencies and job fit areas that a candidate might be weak in.

Signal Types

There are multiple types of signals that can be measured. The following signals can currently be measured:

• Unit Test Accuracy
• Answer Accuracy
• Reviewer Ratings (Scorecards)

The following signals are planned for future release:

• Process Efficiency (Level of time & effort)
• Cyclomatic Complexity
• Test Case Coverage
• Runtime Performance
• Grammar
• Readability

The signals chosen for a given challenge depend on what the challenge is designed to test. Signals can be weighted, so that one signal is more relevant to a score than another.

System Scored Signals

The two system scored signals currently supported are Unit Test Accuracy and Answer Accuracy. Unit Test Accuracy are relevant for coding challenges, Answer Accuracy is relevant for Q&A challenges. These are also the data that is used to determine the Accuracy score for a solution.

System scored signals are scored once an assessment has been submitted.

Reviewer Scored Signals

Once an assessment has been submitted, it can be reviewed by your team. This is where reviewer scored signals come in. They allow your team to objectively score certain aspects of a solution that require a human touch. We will refer to these types of signals as scorecards, since the way they are scored is by giving a scorecard (set of ratings) to reviewers to fill out and submit.

One or more scorecard signals are assigned to each challenge. For example, "Code Readability" is typically assigned to coding challenges. These scorecards are rated by your team, and those ratings later get translated into various scores. Scorecard signals are always mapped to the Qualified Score, and also are usually mapped to at least one Subscore. For example, "Code Readability" by default is mapped to a "Code Maintainability" subscore.

Once a scorecard is submitted by a reviewer, the Qualified Score will be updated to incorporate the new information.

When to use Scorecards

Scorecards are an important part of an evaluation process, because it allows your team to evaluate aspects that a machine can't quantify or qualify on its own. However rating solutions takes time and manpower. In order to save your team time, we recommend that you only do a deep dive review on candidates who have performed well.

Standard Scorecards

When configuring a challenge through the evidence engine configuration view, you will start on the signals tab, where you will have a number of predefined signals to choose to enable. A set of those are for scorecards.

Customized Scorecards

The system does not currently provide a UI for creating custom scorecards from scratch, you must select a standard scorecard. This is a feature that is planned. However you can customize the standard scorecards.

Let's go over the customization options:

Considerations

When a reviewer rates a Scorecard signal, they are provided a list of considerations that they can review. These considerations act as a rubric. Each standard scorecard signal comes with its own pre-defined considerations list.

You are free to update these considerations. Markdown formatting is supported, so you are not forced to stick with the structure initially provided by Qualified.

Candidate Review Checklist

By default Qualified shows the names of the Scorecard signals that will be rated. This is done so that candidates are always made clear about what is expected from them. This ensures that submissions are not made that are not valid due to candidate's not clearly understanding expectations. The option is provided to disable this feature on a per signal basis, just in case a challenge design may be negatively affected by showing this information.