Threats to validity of a research design can lead the results of said research to be thrown into doubt. Both external and internal validity, in addition to construct validity, are factors that need to be considered when forming a research design. In this paper, the various threats to validity are compared and contrasted, and applied to my current research design involving second language interventions for teaching Arabic to military officers.
External validity involves the ability to link the current research in the sample to outside settings and situations. Threats to external validity include inferences that are wrongly inferred from sample data to these outside forces, often due to “the characteristics of individuals selected for the sample, the uniqueness of the setting, and the timing of the experiment” (Creswell, p. 162).
Internal validity, unlike external validity, involves the factors within an experiment that impede the ability of the researcher to correctly measure the sample population. Many different threats to internal validity exist, from the various problems inherent in the participants to the way in which the experimental treatment is handled and manipulated, and issues with the specific procedures used as well (Creswell, p. 162).
Construct validity deals with “the adequacy of the operational definition of variables” (Cozby, p. 97). The primary concern with construct validity is whether or not the thing that was measured was truly measured. It can be determined primarily through face validity, where researchers judge if the measured content is an accurate measure of the variable. Face validity itself is not a quality indicator of validity, as appearance can mask the low quality of a measure. Further tests are required to ensure that the actual measure taken in the study is valid. This type of validity differs from the other types of validity threats in that, instead of threatening the validity of the experimental factors, they threaten the validity of the results by not being thorough enough in their investigation of what the experiment comes up with.
Threats to internal validity for my research design include selection; the fact that I am picking students who are attending the Defense Language Institute Foreign Language Center means that every member of the sample meets the criteria for attending that specific school. This can predispose them to particular outcomes based on the level of intelligence required to attend the school, as well as their specific interest in learning languages. However, as this is the overall goal of the study (to examine interventions in a military language learning context), the beneficiaries of the data would be that same subset of students, and so it remains valid.
Another potential threat is diffusion of treatment, as it will be difficult to prevent students from talking to each other between the various groups as they are taking the same courses. Helping each other with their language studies can reveal aspects of the intervention methods as they pass them along to other students. However, this can be combated through stipulating in the selection process that participants are meant to not disclose their interventions and help other students in the other groups with their studies.
Compensatory demoralization is also possible in the event the control group (the group receiving no intervention) becomes resentful of receiving no help in their language studies, and therefore receiving lower grades than their peers. A proper response to this may be to offer one of the effective intervention models for their studies once the experiment ends – this provides them with the same assistance, and will encourage participation in the study.
External threats to validity for the intended research design include the interaction of the setting (the Learning Center) and treatment (the learning intervention models used). Because this experiment is central to the specific learning center in the experiment, it can be difficult to apply these intervention models to other settings, such as other language institutes. Another difficulty can come from the idea of taking these intervention models to a language learning context outside of the military, where the motivation to learn the language is different (from military and defense applications to economic or personal betterment). This threat could be combated by performing other experiments in a non-military context, or in other settings – however, the problem statement is specific to military language learning, so its validity to the individual situation is still high.
Construct validity could be threatened in terms of content validity; there may be other factors that lead to the results received in the OPI test scores that are independent of the effectiveness of the intervention model. This could be combated by having a sufficiently high sample to eliminate anomalies in learning within the intervention model – the more people in the sample, the more of a chance that the test scores will be indicative of how effective the intervention may be.
Validity issues can impact the research in a number of ways – first, internal validity threats will have the most direct interference with the validity of the experimental factors and the population. External validity threats can diminish the ability to apply the research to other settings or situations, and construct validity threats can lead to doubt in the results, as the measurements are brought into question.
References
Cozby, P. C. (2008). Methods in behavioral research. Mountian View: Mayfield Pub. Co.
Creswell, J. W. (2009). Research design: qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks: SAGE Publications.
Trochim, W. M. (2001). Research methods knowledge base (2nd ed.). Cincinatti: Atomic Dog Pub.
