Cheating Mitigation

Addressing cheating starts with planning, both when creating new content and when using existing content.

We suggest starting by defining "cheating" as it applies to your unique case. What may be cheating in certain contexts may be permissible in others.

Some forms of cheating such as impersonation or live human assistance are universally apparent, but other potential cheating risks may not be so clear-cut. For example, powerful tools such as generative pre-trained transformers (GPTs) and large language models (LLMs) like ChatGPT and GitHub Copilot are part of normal developer workflows and may be allowed in some assessments.

Types of Cheating

Cheating can involve:

• Using hardcoded tables to look up solutions per test case instead of solving the problem as intended
• Copying solution code from an external source
• Accessing information or utilizing techniques or tools prohibited by the instructions, including external resources or AI assistance (ChatGPT, GitHub Copilot)
• Gaining privileged knowledge of the questions to be asked before taking an assessment
• Using unauthorized human help while taking a challenge
• Outsourcing the assessment to a third party to impersonate a candidate for the technical interview
• Reverse-engineering the challenge in a manner that isn't in the spirit of the Task.

Some of these forms may not apply to your case.

Mitigation Strategies

Anticipate LLM/GPT Usage

We suggest two general approaches for preparing for candidates using large language models (LLMs) and other forms of AI coding assistance to write solutions:

1. Explicitly allow (or even encourage) LLMs and offer more difficult challenges and quizzes that candidates likely can't solve completely with AI assistance. After all, AI is part of a normal development workflow now.

   Candidates may presume AI assistance is allowed if not informed otherwise. Other candidates may abstain from using it if it's not explicitly allowed, creating an uneven playing field.

2. Explicitly disallow AI assistance with an agreement quiz explaining that you'd like to test baseline development skills without AI. Use our cheating detection flags and optionally hand-check passing submissions to reject candidates who don't abide by the prohibition.

Regardless of which approach you follow, quizzes with short per-question time limits are a useful tool to force candidates to use their own knowledge. Additionally, coding challenges with limited submissions require candidates to check and trust their code to a greater extent, making AI assistance risky to rely on indiscriminately given its tendency to hallucinate and produce incorrect answers.

Keeping an eye on incoming submissions helps identify patterns of AI usage. If a challenge is consistently defeated without effort and isn't helping you measure your candidates' skills, rotate it out. LLMs are evolving quickly, so prepare to adapt your assessments accordingly.

Prevent Hardcoded Solutions

A cheating technique involves using a lookup table to provide hardcoded responses to each test input, thereby circumventing writing an algorithm as intended.

Randomized Tests

You can use randomized test cases to prevent this strategy from achieving a passing score on a submission.

Many of our challenges in our library already contain randomized tests. Note that within high-stakes situations such as job selection, candidates very rarely (< 1%) cheat in a way that randomized testing is necessary. However, in cases of education with large classrooms or other situations where you anticipate cheating to be a significant issue, randomized testing may be something you want to consider.

Hidden Tests

In addition, you can select challenges with hidden test suites that push the code in deterministic but unexpected ways that are beyond the tests visible to the candidate. If the candidate has used a lookup table based on the visible tests, it's extremely difficult for the candidate to anticipate entries for unknown submission tests and their score will suffer.

Select Creative Challenges

Certain styles of challenges naturally discourage cheating by design. Unusual, novel, open-ended challenges that require candidates to make design decisions tend to promote distinct, harder-to-copy solutions. Plagiarized solutions to such challenges are typically obvious.

In contrast, it tends to be easier (and therefore more tempting) for candidates to look up solutions to well-known algorithm challenges. Submissions for such challenges often vary less between candidates. Novel approaches or designs tend to appear less frequently for these classical algorithm or well-known single-function utility-style challenges. These factors can frustrate cheating prevention and detection efforts.

Add Novelty to Existing Challenges

If you wish to use a well-known challenge, try spicing it up with a novel twist or follow-up requirement that helps thwart would-be cheaters.

As a simple example, adding a third string to a "Fizz Buzz" problem and/or using different, less easily-searchable words than "Fizz Buzz" can reduce the likelihood of cheating while improving the challenge's effectiveness.

Encourage External Resources

If you're confident that your challenge is designed to be robust to cheating using the other suggestions in this guide, you can invite your candidates to consult documentation and perform web searches (with the caveat that you still prohibit fully plagiarized solutions).

Allowing access to resources increases candidate satisfaction, boosts the realism of challenges (programmers always have access to docs and sites like Stack Overflow in the real world), and can provide incentive for candidates to work with the challenge, not around it.

With proper design of an assessment that allows external resources, candidates who attempt to learn everything on the spot will inevitably score lower than those with prior knowledge. Relying on artificial and difficult-to-enforce prohibitions may have the opposite effect, making it harder to distinguish truly knowledgeable candidates who respected an external resource prohibition and cheaters who dishonorably accessed prohibited external resources. You can design question difficulty and set time limits bearing in mind how easy it might be to find a solution through external resources.

Use Time Limits

Time constraints can be used to help prevent the candidate from typing and running code snippets they're expected to mentally evaluate or performing extensive research of a concept.

Be wary of unnecessarily strict time limits, as this can have a detrimental effect on results and unfairly penalize slower readers and those with language barriers.

Use Written or Verbal Reflections

Many of Qualified's pre-built challenges request candidates to submit a brief written response to accompany a code solution. Written responses offer important insight into candidate communication skills while also acting as a deterrent to cheating. A candidate unable to explain their code probably didn't author it!

Nowadays, written reflections are less useful since LLMs can author them. On the other hand, LLM writing styles tend to be fairly easy to identify.

Follow-up questions about the candidate's code during a live interview are a great way to get a sense of attribution as well. For example, you can ask the candidate to explain why they chose the approach to a coding problem that they did.

If you don't have the resources to follow up with everyone, you can clearly indicate to your candidates that they may be selected to explain their solution. This may be enough to discourage some would-be cheaters.

Use Live Interact Sessions

For the most important assessments, consider a live one-on-one session conducted on a video call with the candidate. Live pair coding sessions simultaneously decrease the likelihood of attempted plagiarism and cheating while increasing the likelihood of catching cheating should it occur.

Rotate Content Frequently

Rotating various aspects of assessments frequently can help combat cheating. You might consider rotating:

• entire challenges
• hardcoded magic values (cheat detection "canaries")
• test case data
• easily-searchable challenge keywords and plot

Consider replacing challenges that have worn out their usefulness or that appear to correspond to high occurrences of cheating.

You can apply the Pareto principle to these rotations: a good deal of cheating can be prevented with a bit of planning and effort; there are diminishing returns to spending excessive time on rotation. Higher-stakes, difficult challenges might be deserving of more attention than easy pre-screen challenges.

Playtest Your Content

Just like a regular application, manually testing your challenges on a regular basis (in other words, "eating your own dog food") and examining results critically will help you locate exploitable flaws in the challenge.

Use Honor Agreement Quizzes

Studies show that offering honor agreements before tests reduces cheating rates significantly. You can use our quiz feature to create your own honor agreement where candidates testify to abide by the rules you set forth. Offer this brief quiz at the start of your assessments and explicitly state which resources are allowed and which aren't.

It's natural for candidates to share interview questions and solutions online asking for tips, postmortems, reviews and help. If your goal is to keep content secure, an agreement can prohibit sharing questions and solutions as well as accessing them.