Determining the Quality of a Psychometric Test

This section is better explained via use of an example, and Myers-Briggs Type Indicator (MBTI) fits the situation. It’s easy to use this reference owing to its extreme popularity. Reportedly, 80% of the Fortune 500 use it with their employees. [1] This includes 89 of the Fortune 100 companies also. In 2017 though, CNBC reported cited scientific concerns about the validity of the assessment.[2]

The test even has its set of consultants, which require certification. Nonetheless, and while it is difficult to believe, the test itself is built on zero scientific foundation. It traces back to 1921 when famed psychologist - Carl Jung - unveiled a theory. He surmised that human beings could be broken down to 8 different personality types. But the theory came at a time before psychology used scientific methods such as data or controlled experiments, making the 8 types nothing more than guesses based on Jung’s personal experience.[3]

These findings were later used in 1943 by mystery novelist Isabel Briggs-Myers and her mother Katharine Cook Briggs to lay the foundation of an influential test that would be used for decades to come[4]

They doubled Jung’s hypothesis of 8 personality types into 16, and published the test in 1944. Ironically, the very idea of personality types was dismissed by the original inventor himself - Carl Jung.[5]

“Every individual is an exception to the rule. This kind of classification is nothing but a childish parlor game.”Carl Jung, Swiss Psychiatrist and Founder of Analytical Psychology

Many other tests have also spawned in the wake of MBTI, resulting in questionable qualities, rendering organizational processes dependent on the same as unscientific at best. Today, it’s important to try and understand qualitative metrics that actually make a test effective. This is all the more important with psychometric tests.

Psychometric tests have found use in different stages of the employee life cycle, and also businesses close to the human resource job description - appraisals, hiring, learning & development and more. It’s been known to increase chances of employee success given the correct use of both cognitive and personality tests, two of the most important components to a psychometric test.

But as explained in the section - Determining the Quality of a Psychometric Test, too many organizations use the wrong psychometric tests in the wrong way. But there are measures known to minimize risk and maximize predictive accuracy for said tests.

  1. Understanding the Law:

    HR generalists, specialists or organizational influencers are often advised to maintain legal compliance with the addition of psychometric tests to organizational processes. Anti-discrimination laws require - especially cognitive ability tests as referenced in [1.1.1] - to remain job-relevant and strongly validated.

    A recent example could be traced to the National Football League, an organization that changed its assessment battery due to concerns around racial discrimination and poor job-performance prediction.[6] Tests are generally required to respect privacy and not endeavor to diagnose candidates.

  2. Understanding the Business Needs:

    Organizations are known to focus a lot more on the “independent variables” or predictors over what’s being predicted - the “dependent variables”. If quantitative measures for employee job-performance doesn’t exist, there is little to no basis for statistical correlations for how accurately psychometric tests predict performance. Consider the following:

    1. Purpose: A qualitative test is measured based on validity, and it is essential to ensure that the test being used measures what it is intended to measure. At the same time, an organization must understand the purpose for which they require an assessment before making any selections.
    2. Job Roles: Psychometric tests are often a combination of different assessments, these combinations are best determined based on job roles. For example, content writers would require an assessment that measures for verbal comprehension, while hard labor would mandate a physical fitness test - both cognitive tests.
    3. Industry: Understanding industries form an important part of your assessment battery. If you look at sales, even within the same job role, skills and functionality vary depending on product and buyer sophistication.[7]A salesperson selling pens undeniably require a different set of skills from one that sells IT services.
    4. Geography: A test developed in India using the Indian population as a standard is remarkably more accurate than one that uses an American norm group. For example, it’s more effective - in context - to use cricket analogies in India against baseball analogies, a sport most Indians are unfamiliar with. Likewise, an American audience scarcely tests well off an Indian standard.
  3. Built to Withstand Malpractices:

    Some candidates may be tempted into “gaming” results. It’s commonly referred to as “impression management”, a method used to come across as the more ideal candidate.[8]It’s recommended to compare references and ratings to test results to identify both consistency and correlation.

    Some psychometric tests work with in-built measures to decipher if a candidate’s responses reflect impression management, or if they are incongruent with one another. But security measures aside, even a well-designed, legally defensible, and predictive test battery is likely to fail in adding value should a candidate find the test intrusive or time-consuming.
  4. Assessing the Assessments:

    igh-Performance organizations are in constant requirement of change and improvement, improving candidate evaluation systems - for example - via utilizing predictor, outcome variables, and the correlation between them.[9]Psychometric tests should also be subject to similar validation and intensive testing as the candidates they are being utilized to assess. Parameters for validity, reliability and norming weigh into this scenario.

It’s assumed that when organizationally relevant professionals utilize appropriate methodologies to either retain, develop or select the right psychometric tests, they stand a chance to significantly improve the probability of selecting, developing and retaining the right talent also.[10]This holds true all the more when considering outside consultation or third-party assessment technology firms.