The 20 Most Common Things That Come Up During Reference Checks

At the same time, companies are increasingly abandoning the practice, most often due to one or both of these factors: the length of time it takes to conduct a reference check by phone or letter, and the lack of meaningful information that is provided by the references. Too often, managers are left relying on a change in tone – or even a pause — during a phone check, which hardly seems like the best way to gather information about a potential employee.

In fact, we don’t know much about what works and what doesn’t when it comes to reference checks. That’s because there’s very little research that’s been conducted on the practice, in large part due to the fact that they typically take place via phone or letter. The few studies that do exist, which are primarily on checks via letter in higher education, find that they may contain content that shows bias against protected classes of job applicants, and that the length of the letter is sometimes disproportionally used in decision-making. A review of the literature does not turn up any evidence that the length of a recommendation is correlated with subsequent job success or any other desirable work outcomes.

So how can we gain insight into what reference checks actually do? The company I work for, SkillSurvey, is in the business of facilitating checks online. To date, we have reference feedback on approximately 3.2 million job candidates, across most job titles and industries. The feedback involves ratings on work competencies, as well as open-ended comments. Over a year ago, we launched a study to describe the content and nature of the comments provided by the references, yielding data from a sample of 12,800 references — the largest analysis of feedback that we’re aware of. Specifically, we focused on open-ended text comments that ask about an applicant’s work-related strengths and areas for improvement — and even though it is optional for references to provide these comments, we found that the majority of references did indeed provide this information on the applicant’s strengths and areas for improvement (89% and 83%, respectively).

All told, our research yielded 44,941 words or short phrases across all the feedback we reviewed. Our team then used what is called a grounded theory approach to our research, as there is no library or lexicon that exists on the content and nature of reference feedback (the open-ended feedback also went beyond existing frameworks for personality traits and competency models). We used text analytics software to identify and extract actual short words or phrases provided directly by the references themselves, and in doing so, began building the first library of reference feedback, from the ground up. The actual words or short phrases provided by the references, such as ‘accepting criticism’ or ‘communication’ or ‘problem solving,’ were used to name the themes found in the text data.

Here’s what we found:

Considering that delivering negative feedback is not a desirable task, and many times is actively avoided, it was particularly surprising to find 106 different areas for improvement. These findings are in contrast to earlier research that shows very little such information in letters of recommendation. We suspect that the confidential nature of the online process may be somewhat responsible for reference providers being open and candid in their feedback.

Three themes in the areas for improvement — confidence, knowledge, and communication — were in the top 10 for most of the jobs we studied.

Yet the top themes for work improvement appeared to be more job specific, compared to those themes provided for the strengths. Some examples of top, job-specific themes in the areas for improvement were: Delegating for project managers; Time Management/Prioritizing for a phone customer service rep; Attention to Detail/Accuracy for a software developer; Work-Life Balance for a director of human resources; and Handling Stress for a nursing manager.

We then looked at the most frequently occurring themes in the work-related strengths. Across all jobs studied, Commitment/Dedication was among the top 10 themes. The following themes were in the top 10 strengths for the majority of jobs: Dependable/Meets Deadlines; Team Orientation; and Attention to Detail/Accuracy.

Other themes were more specific to a particular job. For example, Building Relationships for field sales reps, Honesty/Trustworthiness for skilled trades workers, Compassionate/Caring for registered nurses, and Punctuality/Attendance for retail cashiers were in the top 10 for these categories. These qualitative findings are consistent with quantitative findings that soft skills (e.g., personality traits) tend to vary in their importance across jobs.

At present, our research highlights a number of things. One is basic and reassuring: referees will still take the time to provide additional feedback in the form of open-ended text comments, satisfying a very basic human need to provide and receive narrative information.

It is also noteworthy that for the most part referees are electing to mention soft skills, supporting recent studies and articles in the popular press on the importance of soft skills for workplace success. These are areas that can most often get in the way of an employee’s success — and they’re also very difficult to screen and train for. This is why getting this type of specific feedback from references can be valuable to employers.

The findings also give us details on what managers and coworkers value in their employees and colleagues, and where most people stand to improve. We think this means that employers would be well-served to focus their on-boarding, training and development efforts on key areas where we know that many candidates can get better — like helping them to gain more confidence and sharpen their communication skills.

At the same time, there’s still so much we don’t know, and we plan to add even more depth to our feedback library by increasing our sample size in order to try to answer outstanding questions. Based on our previous research, we already know that the percentage of referees who respond, in addition to the behavior ratings that look at an employee’s competency, are predictive of work outcomes such as turnover and manager ratings of new hires. We are now curious about which themes (both strengths and areas for improvement) in the text feedback can be added to our current models of predicting work outcomes from references. And of course, we are all interested to see what the similarities and differences are between feedback provided by managers vs. coworkers, and how these findings might relate to work outcomes.

The bottom line is that, especially when gathered in a confidential manner, references can expose real strengths and weaknesses. By taking a hard, analytical look at the candid verbatim feedback that referees share – and not relying on the sound of a voice or gut feel — we expect it will be possible for employers to make even better hiring decisions.