"

5-3: Usefulness of Selection Devices

A road sign with "Usefull" in black text on a yellow background at the top, and "Useless" crossed out with a red diagonal line. The sign is set against a tree-lined road.Not all selection systems are worth the effort and expense. Some provide tremendous value by dramatically improving hiring outcomes, while others are costly exercises that don’t actually help you make better decisions. Understanding what makes selection systems useful helps organizations invest their resources where they’ll get the biggest bang for their buck. The answer to “Is it worth it?” often comes directly from an assessment of utility, and questions of utility are always framed in terms of “how much better is this new system than the one we used to use?”

Decision Accuracy

Every selection decision creates one of four possible outcomes, and understanding these outcomes helps you evaluate how well your system is working:

Hits are your home runs — people you hired who turned out to be successful employees. False alarms are your expensive mistakes — people you hired who didn’t work out and had to be replaced. Correct rejections are the bullets you dodged — people you didn’t hire who would have been unsuccessful if you had. Misses are the ones that got away — people you didn’t hire who would have been great employees.

Obviously, you want to maximize hits and correct rejections while minimizing false alarms and misses. But here’s where it gets interesting: different organizations might prioritize these outcomes differently based on the costs and consequences of each type of error.

Scatter plot with a red dotted regression line. The x-axis is labeled "Predictor Battery" ranging from "Reject" to "Hire," and the y-axis is labeled "Criterion" ranging from "Failure" to "Success." The plot is divided into four quadrants: Misses, Hits, Correct Rejections, and False Alarms. The R² value of the regression line is 0.821.
In this diagram, R² is very high (.821), therefor misses and false alarms are low, but still inevitably occur.

If you’re hiring air traffic controllers or nuclear power plant operators, false alarms (hiring someone who fails) could literally be catastrophic. These organizations typically use extremely strict selection criteria, accepting that they might miss some potentially good candidates to minimize the risk of hiring someone who could cause disasters.

For other positions, missing good candidates might be more problematic than occasionally hiring someone who doesn’t quite work out, leading to more relaxed selection standards that cast a wider net.

You can actually calculate specific decision accuracy metrics. Decision accuracy for hires looks at the percentage of hiring decisions that are correct: Hits/(Hits + False Alarms). This is particularly important for critical positions like surgeons, police officers, or pilots where you absolutely need to get it right. Overall decision accuracy focuses on maximizing both hits and correct rejections while minimizing misses and false alarms: (Hits + Correct Rejections)/(All Applicants). The challenge is that while hits and false alarms are easy to measure (you know how your hires perform), you know the combined size of misses and correct rejections but not their individual values.

Validity Considerations

The validity of your selection system directly impacts how useful it is. Higher validity means more accurate predictions, which translates to more hits and correct rejections and fewer costly mistakes. When your selection methods lack validity, you’re essentially making random hiring decisions — you might as well flip a coin and save yourself the time and expense. If there’s no correlation between your predictors and actual performance, your selection battery does nothing useful, and you might want to start looking for a new job!

Base Rate and Selection Ratio

Base rate — the percentage of your current employees who are performing successfully — provides important context for evaluating whether new selection systems are worth implementing. If 90% of your current employees are already successful, it’s much harder to demonstrate improvement from a fancy new selection system than if only 60% are meeting expectations.

Equation for base rate
This equation will give the base rate when measuring the value of a new selection battery.

Selection ratio — the number of job openings divided by the number of applicants — dramatically affects how much value you can get from sophisticated selection systems. When you have tons of applicants for just a few positions (low selection ratio), even small improvements in prediction accuracy can provide huge benefits because you can afford to be extremely picky. When you have few applicants for many openings (high selection ratio), elaborate selection systems provide less value because you don’t have much choice anyway. If your selection ratio is 1.00 (same number of openings as applicants), don’t bother using a selection battery — it has no value!

Think about the difference between Harvard’s admissions process (thousands of applicants for limited spots, making sophisticated selection extremely valuable) versus a fast-food restaurant during a labor shortage (limited applicants for many openings, making elaborate selection procedures less worthwhile). The selection ratio is often used as a measure of exclusivity for universities, and research shows that clinical PhD programs are among the most difficult to get into, though this doesn’t account for how many different schools each applicant applied to or the number of offers rejected.

Cost-Benefit Analysis

Selection systems range from simple and cheap (basic interviews and application reviews) to comprehensive and expensive (extensive testing, assessment centers, multiple interview rounds). The key is making sure the benefits of better hiring decisions outweigh the costs of more sophisticated selection procedures. As selection batteries get larger, they will, by definition, get better at prediction — but they also cost more money.

Organizations like NASA and the FBI invest heavily in selection because the costs of hiring the wrong person far exceed even the most expensive selection procedures. Why does NASA spend billions on selection? Because the consequences of hiring someone who makes critical errors in space missions or national security are catastrophic. For other organizations, simpler approaches might provide the best return on investment.

You can use tools like Taylor-Russell tables to estimate the improvement in your workforce from a new selection battery (Taylor & Russell, 1939), but you still need to answer the crucial questions: What is that improvement worth in dollars? How much will increased productivity offset the costs? How much have you already spent developing and testing the new battery? Does this new system protect you better from lawsuits than your previous approach?

A scatter plot with axes labeled "Criterion" (vertical, ranging from Failure to Success) and "Predictor Battery" (horizontal, ranging from Reject to Hire). The plot is divided into four quadrants: Misses, Hits, Correct Rejections, and False Alarms. Light blue dots are scattered throughout, and a red dotted line runs horizontally across the middle. A label in the top right corner reads "R² = 0.0028."
If your predictor battery does not have validity (R² is nearly 0 in this case), then you might as well save your money on the selection process and start relying on coin flips for selection decisions.

Contemporary research has shown that compensatory selection models generally outperform multiple-hurdle selection models in terms of overall utility, despite the higher costs associated with administering complete predictor batteries to all candidates (Ock & Oswald, 2018). However, organizations must balance this finding against practical considerations such as time constraints and resource limitations.

The validity and utility of selection methods continues to be refined through ongoing research. A comprehensive analysis of 100 years of research findings shows that certain selection methods consistently demonstrate superior predictive validity and utility across different organizational contexts (Schmidt, Oh, & Shaffer, 2016). Organizations should consider these research findings when designing their selection systems to maximize both effectiveness and cost-efficiency.

A decision matrix comparing hiring outcomes with the actual job performance of candidates. The table includes four cells: correct decision (not hiring a low-performing candidate), Type I error (hiring a low-performing candidate), Type II error (not hiring a high-performing candidate), and correct rejection (hiring a high-performing candidate). Each error type is associated with specific costs like wasted salary or lost productivity.
Just like you learned in your statistics class, when we make decisions, we will make mistakes.

Media Attributions

  • Useful © Unknown is licensed under a Public Domain license
  • Decision Accuracy © Jay Brown
  • Base Rate © Jay Brown
  • Decision (in)Accuracy © Jay Brown
  • Decision-Making © Jay Brown

License

Industrial/Organizational Psychology TxWes Copyright © by Dr. Jay Brown. All Rights Reserved.