By Dr Alex B Siegling
Originially published in Assessment and Development Matters (Spring 2016)
Reprinted with permission by the British Psychological Society
This article aims to offer succinct and current practical advice on efficiently 'screening' the market and selecting appropriate tests, based on relevance, practicality, and quality. Subsequently, advice on finding the information necessary to evaluate available tests is offered.
When examining the business/HR test market, practitioners are exposed to an abundance of products. Competition within a free market should, in theory, impact positively on the quantity and quality of measures. However, the pervasive commercial dimension of psychometrics, along with a certain degree of evangelicalism attached to several tests, does have its drawbacks. It is not uncommon for tests to be selected based on marketing/sales strategies and popularity, rather than their psychometric quality. The commercialisation of psychometrics, coupled with cheap and relatively sophisticated software, lowers the barriers to entry, but also opens the door for questionable products.
The question is: 'What do I need?' In business psychology (or organisational, occupational and industrial psychology), psychological assessment often concerns a person's current or prospective level of functioning in a particular role. Its purpose may be to select or develop.
The first steps normally involve an analysis and description of the role, including its
responsibilities and requirements. This process greatly facilitates the specification of competencies a qualified candidate would bring to the role. This person specification narrows the focus of the assessment and facilitates the identification of relevant tests (of appropriate breadth and depth). At this stage, one is likely to discover several relevant tests.
A next logical step in shortening the list of candidate tests is to scrutinise the available norms in terms of their similarity to the role. How general (or specific) are the norms in terms of country/culture, role, level of ability, and other demographic variables? Does the test feature any specific norm groups particularly suited for one's purposes? If so, how specific are these? norm-referenced tests, which constitute the majority of commercial psychometrics, are only as appropriate as the relevance of the available norms for one's assessment needs.
Prior to scrutinising the quality of relevant tests, practical considerations based on one's requirements and resources can further refine the list of options. Some tests are only available for manual administration, requiring the presence of a qualified tester. However, an increasing proportion of tests can be administered digitally and remotely, and therefore easily to a large number of people. measures not (yet) featuring digital assessment may nonetheless support group-based assessment. The length of administration time, specific demands of a test (e.g. need for special equipment), accessibility to suitable informants, and other factors can determine a test's level of pragmatism for one's situation.
In an ideal world, cost would not be a factor. Theoretically, costs ought to be considered at the very end of the test selection process, such as where the same test is available via competing sources or where discounts can be obtained for bulk purchase.
Having screened for relevance and practicality, the task is to scrutinise the psychometric properties and norm group quality of the identified options. This process can start at a broad level, by asking questions such as: Have test findings been published in the scientific literature? Is a technical or user manual available? What are the publisher's procedures and criteria in publishing tests? Generally, one should be wary of tests (and their publishers) if no supporting evidence is available.
While availability of a manual and research evidence is a good sign, it is not sufficient in itself. The fundamental question in evaluating tests is: 'Does the test measure what it claims to measure?' The answer to the question is complex and depends on the combination of reliability and validity evidence, in relation to a test's underlying theory. An ideal and comprehensive set of standards are the criteria set forth by the european Federation of Psychologists' Associations (eFPA; Lindley, Bartram, & Kennedy, 2005). Also part of the answer is the quality of available norm groups, which concern sample size, demographic composition (gender, age, ethnicity, etc.), and year of collection of the data. The key guiding question here is whether the available norms represent the target population.
Figure 1 summarises the proposed test selection process. note that the sequence of steps is aimed at maximising efficiency - it has no bearing on the relative importance of the steps.
Availability of information
Complicating the selection of tests is the fact that the information necessary to make informed decisions is often not readily available without possession of the test manual. However, several alternative sources of information are worth exploring: Independent test reviews, primary and secondary literature, and test publishers themselves - who should be able to provide detailed information supporting quality of their measures.
Test reviews exist in various forms. The British Psychological Society's (BPS) Psychological Testing Centre provides its 'Certificate of Test registration', along with ratings and comprehensive reviews aligned to the eFPA criteria. This information can be accessed online (via www.psychtesting.org.uk) and is freely available to members of either the register of Qualifications in Test use or the BPS. Alternatively, they are available for a fee. The existence (or absence) of a BPS certificate, which indicates an 'adequate' standard, should be interpreted cautiously. many good quality tests do not get submitted for BPS review (it is the up to the publisher/author to decide on the submission of a test for review). Conversely, many tests meet the minimum criteria for registration, but not all achieve a high rating.
Another organisation that provides test reviews is the Buros Center for Testing (http://buros.org/). It offers descriptions of over 3500 commercially available tests, the majority of which include more extensive reviews. These can be purchased and accessed online. The organisation also publishes its Mental Measurements Yearbook (Carlson, Geisinger & Jonson, 2014), containing all new and updated test reviews, and its Tests in Print (Murphy, Geisinger, Carlson & Spies, 2011), which aims to provide descriptions of all commercial tests in the english language. Both are typically available via university libraries.
Several specialist journals publish reviews of commercially available tests. on a systematic basis, these can be accessed via academic databases (eBSCo, PsycInFo) typically subscribed to by universities or otherwise available for a personal fee. Published articles are increasingly available open-access or via authors' personal webpages, such as research Gate or Academia.edu profiles. It is well worth searching the internet (e.g. Google Scholar) if one has no access to the relevant databases. Furthermore, some practitioner- and academic-focused books include reviews of commercially available tests (e.g. Boyle, Saklofske, & matthews, 2015; maddox, 1997).
Less formal types of review may be found online. examples of other outlets are assessmentpsychology.com or even online communities, for instance on LinkedIn. General advice: Before investing in a general resource, it is wise to ascertain that it contains information on the measure(s) of interest. It also is important to consult reviews that refer specifically to the relevant version or edition. Tests may have updated versions as well as editions in multiple countries of the same language (e.g. uK and uS) that vary considerably in test content and norms.
Test manuals should include references to research reported in journal articles. A lack of such references should raise some concern. Ideally, one should see studies on the test's reliability, validity, and factor structure, as well as cross-cultural comparisons and measurement equivalence studies. Whilst of specific focus, consulting individual studies can be highly informative. For instance, studies may show whether a test has utility in a specific context or provides psychometric properties for a specific population or context. reviews of the literature or meta-analyses focused on individual tests can be particularly informative.
On a systematic basis, this type of information is available via academic databases, such as Health and Psychological Instruments (HAPI) and PsycInFo. Without access to these, it is worth searching the internet for open-access articles.
Publisher websites often provide a good starting point for finding basic information, along with administrative and commercial details. Publishers are also happy to provide further information about their products, which might not be available elsewhere. They can for instance provide information on updated editions, pricing plans, updated norms, discounts, etc. For example, Hogrefe Ltd operates its own in-house consultancy team of occupational psychologists, who provide expert advice on the use of company products.
Dr Alex Siegling is Psychometrician at Hogrefe Ltd and research Associate at university College London.
Boyle, G., Saklofske, D. & matthews, G. (2015). Measures of personality and social psychological constructs. London, united Kingdom: Academic Press.
Carlson, J.F., Geisinger, K.F. & Jonson, J.L. (2014). The nineteenth mental measurements yearbook. Lincoln, ne: Buros Center for Testing.
Lindley, P., Bartram, D. & Kennedy, n. (2005). eFPA review model for the description and evaluation of psychological tests. european Federation of Psychologists' Associations.
Maddox, T. (1997). Tests: A comprehensive reference for assessments in psychology, education, and business (4th ed.). Austin Tex.: Pro-ed.
Murphy, L.L., Geisinger, K.F., Carlson, J.F. & Spies, r.A. (2011). Tests in print VIII. Lincoln, ne: Buros Institute of mental measurements.
By Dr Alex B Siegling