A Test for Health Plan Translators
In California, private and public health plans produce translated documents and websites to meet the language needs of their members. In an ideal world, all of their documents would be certified, culturally competent, and have benefitted from the input of community reviewers. With communication needs on the rise, however, many health plans have turned to their bilingual staff to produce some translations that do not require certification. But health plans need an objective method to determine if their staff is qualified to produce these simple translations.
It is for this level of translation that the Health Plan Translator Test (HPTT) has been developed. In the course of developing the HPTT, we looked at existing tests in California and the nation and research related to language testing, which is summarized below.
The best-known translator test available in the United States is the American Translator Association’s (ATA) accreditation test. It is administered yearly in different regions of the country to highly competent professional translators in 15 language pairs. The ATA test measures a translator’s ability to create a legal, technical/scientific, and literary translation of parallel register, style, and syntax and with near perfect punctuation, orthography, and clarity. The ATA test requires the linguist to have excellent knowledge of the legal, scientific, and literary jargon, as well as an ability to produce a well written document that is legally, commercially, and/or socially viable. In sum, it is, perhaps, the highest level of certification available in this country.
This level of certification, however, is not in line with the more routine non-certified translation needs of California’s health plans. The disparity between the demands of a typical health plan translation and the skills required to attain certification make the ATA test a poor predictor of the translator’s ability to perform the day-to-day tasks. Health plan administrators need a test that will be a useful and valid way of determining their translators’ ability to produce appropriate text. These more mundane tasks require knowledge of:
- Basic grammar, orthography, and writing conventions,
- Medical, insurance, and administrative terminology, and
- Legal terms commonly used in plan documents.
What is needed is a testing instrument that would evaluate the non-certified translator’s ability to produce translations of the fairly simple documents they typically handle. It was our goal to create a test that would test the specific skills and areas of knowledge that health plan translators would need to demonstrate to prove basic competency. We started by studying the most commonly administered Spanish language proficiency instruments in California, summarized below.
Transcend reviewed a number of language proficiency instruments (in English and Spanish), including:
- ATA test (described above)
- CPS test (written portion of medical/administrative interpreter test)
- BSM, Spanish Idea Proficiency Tests (IPT), LAS 2
- Woodcock Muñoz Language Survey
- NCIHC Guide
Some of these are online instruments; others are administered at testing centers or on-site. Most assess oral language skills. Despite their different modalities, they share important similarities:
- They are fairly short tests (15 minutes – 2.5 hours)
- They look at a variety of language skills, including grammar, usage, and lexicon
- They set fairly accessible passing scores, typically 80%
The Health Plan Translator Test (HPTT) is the first of its kind in California. It tests the specific skills health plan translators would need to perform their job at a minimum level of competence, i.e., as a starting translator. Examinees are not expected to possess the same level of language expertise as certified and highly skilled translators. The HPTT posits that a minimally competent health plan translator would need to demonstrate:
- Strong writing skills, including mastery of common writing conventions, such as capitalization and punctuation (which are not the same as English).
- Command of spelling and correct use of accents.
- Key and common terms in simple health plan documents.
The test has 6 sections, each with different types of questions that have been designed to test different skills, including those listed above. Some sections provide instruction in English; others in Spanish. Three sections are in multiple-choice format. In the remaining 3 sections, the examinee must fill in a blank to produce the missing term. The “fill-in” question format was used instead of multiple-choice to ensure that examinees can produce appropriate responses rather than simply recognize them. The scoring procedures recognize more than one correct fill-in answer. Each test is comprised of 50 questions that are randomly pulled from a database of similarly structured questions, so individual tests vary. The level of difficulty of each question is reasonably constant. Based on feedback from a cohort of subject matter experts and passing scores of competent health plan translators, the passing score for the HPTT is set at 74%. This test was designed to be culturally competent for the Southwestern U.S., where Spanish-speaking health plan members are primarily from Mexico and Central America. As such, the test favors the Spanish spoken in the southwestern U.S.
To protect the integrity of this test, it was created without outside funding sources and participation. We were guided by a PhD student/statistician in the Psychology Department at the University of California, Berkeley, who has helped us to produce, test, validate, and revise the HPTT to meet generally accepted standards of reliability and validity. We relied principally on the following resources to guide us: Brown JD & Hudson T (2002). Criterion-Referenced Language Testing. Cambridge University Press, Cambridge, UK. Cizek GJ (2001). Setting Performance Standards: Concepts, Methods, and Perspectives. Laurence Erlbaum Associates, Manwah, NJ. Hurtz GM & Auerbach MA (2003). A Meta-Analysis of the Effects of Modifications to the Angoff Method on Cutoff Scores and Judgment Consensus. Educational and Psychological Measurement, 63(4): 584-601. Impara JC & Plake BS (1997). Standard Setting: An Alternative Approach. Journal of Educational Measurement, 34(4): 353-366.
To address the health industry’s standards for privacy, accuracy, and cultural competence, we also included test questions on ethics and best practices in translation. Working with a testing consultant, we devised a test that health plans would find useful and cost-effective, that health plan translators would find fair, and that testing experts would deem valid.
The test is comprised of 6 parts. Click here for sample questions from the HPTT.