Daniil Zhukov, head of the Executive Assessment practice at Ward Howell, talks about the use of psychometric tests and surveys in assessing top managers.
If you have had an opportunity to participate in discussions about executive assessment, you should have noticed that there are certain topics for which opinions vary drastically between people and companies. One such area is the use of psychometric tests and surveys. Some indiscriminately reject them outright, and others are fanatically committed to them. We should understand what truly goes on in this area and how it applies to the assessment of top managers.
Personality assessments have already been widely used in business for over 40 years. Hundreds of players on the market use them nowadays. But how accurately do these assessments predict the leadership potential and effective management skills of senior managers? In the last 40 years, have there been any significant breakthroughs in research, developments in fundamentally new approaches, or, even more interesting, irrefutable evidence supporting the use of personality assessments in evaluating top managers?
On the one hand, yes. We have reached a certain consensus on the “big five” personality traits – extroversion, agreeableness (flexibility, friendliness, amicability), conscientiousness (integrity), neuroticism (at the opposite end of the spectrum – emotional stability), openness to experience (intellect) – as the main elements of personality structure. This discovery was the powerful impetus behind research into the correlation between personality traits and leadership potential. The first set of data was sufficiently encouraging: the “big five” demonstrated a combined validity of 0.39: (1) openness to experience and conscientiousness (integrity) were identified as indicators of task-oriented efficiency, (2) emotional stability, extroversion and agreeableness showed interpersonal effectiveness to be a component of leadership.
On the other hand, criticism that cites the declining validity of personality tests for assessing executives is not unfounded. In comparison, for example, with the results of conventional tests of ability, it’s doubtful that a personality test adds much value (3). The main argument of critics is that there is a sizeable difference between stating that personality impacts the behavior, style, and effectiveness of leadership (a statement with which it is difficult to argue) and being confident that personality profiles built from self-evaluation are in fact able to predict leadership effectiveness. It’s impossible to deny the possibility that the profile of your best top manager could be exactly the same as that of your average second grader.
It’s not so simple
These facts point to several limitations inherent to personality assessment tools such as questionnaires.
The number one problem is that intelligent candidates use their intelligence and intuition to select the specific characteristics that they know are relevant to their concrete situation (4). We are not talking about recent graduates, but about experienced managers who thoroughly understand what works in one business or another. Vulnerability to false answers has always been the Achilles’ heel of personality questionnaires. There is plenty of evidence to show that a significant portion of respondents give the “right” answers when filling out questionnaires (5). But even if socially desirable answers don’t critically impact the validity of such questionnaires, they do call into question the effectiveness of the selection process. It seems that the candidates who answer questions honestly are at a disadvantage compared with those who are willing to manipulate the truth.
As for the scales of coherence and social desirability, they are intended to show the coherence (consistency) of respondents’ answers and, in theory, protect against manipulation by respondents. But do these measures work? According to the developers themselves, “high marks on the ‘social desirability’ scale should arouse suspicion” (6), but there are several factors, alongside attempts to manipulate the results, that can affect the results of this scale: self-criticism/analyticity, self-esteem, level of conformity, motivation for filling out the questionnaire, clear wording, etc. So this doesn’t provide any precise guarantees.
There is another factor worth considering: time. Most of the methodologies used today (15FQ, 16PF, Dimensions, DISC, Hogan, NEO, OPQ, PAPI and so on) that are, to varying degrees, derived from the “Big Five,” rely on theories and knowledge about leadership from the 70s and 80s. Do you think our understanding of leadership has changed significantly since then? How significantly have the requirements for leaders and businesses changed? The answers to these questions are obvious.
What to do
The goal of this article is not to encourage the abandonment of psychometric assessment. For a certain employee level, a psychometric assessment is the optimal method in terms of “price – quality – speed.” Furthermore, a massive staff assessment (particularly of those in entry-level positions) that can help with early identification of those that are predisposed to one or another type of behavior can be extremely useful. The natural limitations of validity may not be that critical in a large assessment. The fascination with psychometric tools for executive leaders could come with a significant margin of error, which makes it necessary to confirm results with other, more valid and costly methods of research.
The future of psychometric assessment of top managers is not in simplifying and standardizing, but in measuring dynamic traits and their influence on the creation, rise, and decline of leadership. To do that, there are two simple things to note:
1) Any kind of personality can become a leader. This can be confirmed by analyzing examples of successful leaders over the last three decades. They include shy introverts, hyper-anxious workaholics, and even sociopaths. It’s hard to imagine how a personality questionnaire would help find the next Bill Gates (Microsoft), Elon Musk (Tesla) or Al Dunlap (Sunbeam).
2) Different personalities will be effective leaders in different contexts. It’s not possible to put together a single (global) index or portrait of a leader, the same way it is impossible to standardize the leadership-related tasks, business strategies or organizational contexts that leaders will face. Sufficient abilities, agreeableness and conscientiousness will only speak to a person’s integration into a corporate environment, but it cannot predict a person’s effectiveness as a leader in any particular scenario.
Where to start
These are some simple recommendations for those who are looking to use personality questionnaires to assess executives:
- · Start with the results, and then slowly move toward the assessment. Of course you can fire an arrow and then figure out where it lands. But that’s not how you learn to shoot. When you know precisely what you’re aiming for, your accuracy increases. That’s why it’s important to very clearly define the goals of the assessment and how they relate to the questions to which you seek answers (hiring, replacement, promotion, training, etc.). It’s good to analyze which indicators will be more important for particular leadership-related scenarios in your organization.
- · Demand as many specifics as possible. Don’t be distracted by universal models and global surveys. Ask specific questions, learn about concrete traits that are important for your evaluation, and choose the tools that are most appropriate for the tasks at hand. Be critical about your choice.
- · Integrate the results properly. Gathering information isn’t difficult. But integrating data from different sources is not so simple. The accuracy of assessments significantly increases why you fully understand the limitations of each assessment method. In order to see this, it’s important to look not just at the final tally, but at each point and the essence of the assessment.
- · In future articles, we’ll talk about other tools for evaluating leaders: business simulations, assessment centers and interviews.
If you have any questions, please feel free to contact me via email: firstname.lastname@example.org.
1. Judge, T. et al. “Personality and Leadership: a Quantitative and Qualitative Review.” Journal of Applied Psychology, 2002.
3. Schmidt, F. et al. “Implications for the role of personality and general mental ability in job and training performance.” Personnel Psychology, 2008.
5. Rosse, J. “The impact of response distortion on preemployment personality testing and hiring decisions.” Journal of Applied Psychology, 1998.
6. Description of OPQ32 scale, SHL Group LTD.