Independent and Dependent Variables
Measuring the current competency in the provision of vital public health services and training is crucial to enhancing competencies. The effort helps to ensure reliable leaders in the provision of health services. Hence, the Council on Linkages has undertaken the task of establishing various Core Competencies for public health personnel (Council on Linkages, 2001). In addition, the Council on Linkages (COL) has established a set of competencies for assessing the staff in public health (Council on Linkages, 2005).
The COL Core Competency set has eight substantive areas:
• Communication
• Cultural Competency
• Policy Development and Program Planning
• Community Dimensions of Practice
• Basic Public Health Science
• Financial Planning and Management
• Analytic Skills
• Leadership and Systems Thinking
The COL competency set appears in the IOM report (2003) as an appropriate standard for public health workers. Besides, one may argue that the COL set has become a tool of choice due to its widespread use. During the 2004 American Public Health Association meeting, examples of the utilization of the COL competency set included 26 separate cases for projects. Such cases included the creation of training programs for developing a state health plan. The database of public health training programs provides examples of programs and courses based on the COL set. The Council on Linkages core competencies is related to ten crucial public health services and helps to assess the public health workforce (Council on Linkages, 2005).
The IOM report (2003) lists in excess of sixty different competency sets that might be relevant to various professional positions within the public health workforce. The area of competency development is an area of substantial activity while the validation of competencies has relatively less robust activity. Hence, the present investigation addresses the measurement issue.
Important levels of measurement
Competencies must be validated using the traditional tools of analysis in order to ensure the credibility of the main stakeholders. Since my primary aim is to investigate the measurement properties of the Core Competencies, I would like to address the issue through principal components analysis. By analyzing the underlying factor structure, one can determine the level that each of the items measures the intended parameter, as well as the items that might need elimination or revision. Thus, specific questions to be investigated in my study include:
1) Does the solution that emerges from the principal components analysis coincide with the eight-domain structure of the COL competency set?
2) Are there individual items that do not appear to be good discriminators? That is, do they exhibit factor loadings that suggest they are measuring more than one underlying construct or not measuring the domain of interest accurately? If so, these items could be candidates for revision or elimination.
Ensuring Construct Validity, Content Validity, and Empirical Validity
Ensuring reliability of measurement
Guaranteeing the reliability of the study’s measurement instrument has been characterized as an on-going process (Streiner & Norman, 1995). It is also a “matter of degree” (Nunnally, 1998) that requires the use of multiple methods and information sources. Thus, the study would utilize various methods, as well as several sources of information.
The limitations and strengths of the selected measurement instrument
Validity can be viewed as comprising of content, construct, or criterion validity. The content validity involves the level at which an instrument adequately samples the full domain of phenomena under investigation and is often established through expert review. Criterion validity assessment involves determining the level at which the results obtained with the instrument being validated correlate with results from a previously validated tool that measures the same phenomena. For example, when establishing a new measurement tool to assess depression, the obtained scores should correlate with scores from a previously validated instrument (Streiner & Norman, 1995). Under the Trinitarian view, construct validity is the point at which a measurement tool measured the construct under investigation.
Recent work has resulted in a more unitary model in which criterion and content validity have become subsumed under the overarching term of construct validity. In the model, evidence of criterion and content validity are seen as elements in establishing a case for the construct validity (Schultz, Riggs, & Kottke, 1998; Moss, 1992). Moss (1995) also suggests that there is universal acceptance of this conceptualization. Adcock and Collier (2001) have stated that the new model of validity identifies criterion and content validity as necessary but not sufficient to establish a more expansive notion of construct validity. In the unitary model, the term construct validity has become almost synonymous with the term measurement validity. However, Adcock and Collier (2001) use the term nomological validity.
The establishment of construct validity requires the use of multiple methods, using both empirical study and the testing of theoretical implications based on the phenomena under investigation. The construction of a “nomological network” (Campbell, 1990; Cronbach & Meehl, 1995) composed of evidence from multiple methods and sources provides the strongest case for the validity of a measurement instrument. This network can consist of “observables” or empirical data about the constructs that the instrument under investigation attempts to measure.
Appropriate Scale for Use
Factor analysis or principal components analysis can provide evidence regarding the efficiency of the measuring items. For example, the Core Competency set has a number of items intended to measure a respondent’s perceived ability regarding Communication. Factor analysis of such items can provide statistical evidence regarding the interrelationships between items and assess their effectiveness in measuring an underlying construct, which can be termed as communication.
Justification of the Scale’s reliability and validity
Studies of the validity (Beaulieu & Scutchfield, 2002) and reliability (Beaulieu, Scutchfield, & Kelly, 2003) of the National Public Health Performance Standards are examples of efforts to validate the measurements used in public health system research. Beaulieu Scutchfield and Kelly (2003) assessed content validity through a survey of state and local staff. Consequently, the respondents rated each of the model standards used in the National Public Health Performance Standards. The authors achieved a 45% response rate (n =33) from the local health department staff. They also calculated descriptive statistics on the proportion of the sample that responded Yes, Partially or No to questions about the model standards for completeness and achievability. For each of the model standards, the majority of respondents answered yes for both completeness and achievability. Responses regarding the importance of the model were much lower with only 33 to 45% of the sample answering yes. Further, the authors used health department staff as external judges and compared the score by the judges to the self-assessed score by the local health departments. The authors used the two-tailed t-tests to calculate the percentage of the external judges’ scores that differed significantly with the self-assessed scores. Only 35% of the external judges’ scores agreed with the self-assessment scores across all the model standards. When stratified by the size of the health department, the percentage of agreement ranged from 27% to 58%.
The study by Beaulieu, Scutchfield, and Kelly (2003) partially responds to the call for the assessment of the validity and reliability of measurements used in public health systems. However, in its investigation of content and criterion validity, their study stops short of assessing the more encompassing idea of construct validity. Although the authors can be praised for their attempts at validation, measures are often developed and put into use prior to rigorous assessment of their validity and reliability.
The Appropriate Test for the Plan
Specifically, the work tests are appropriate for my project since the measurement qualities and the underlying conceptual model implied by the 8- domain structure discriminates among the various known groups within the workforce.
The statistical analysis includes principal components analysis to explore the factor structure of the competencies. Correlation is used to compare the orthogonality of the domains in their initial form to any revisions suggested by the principal components analysis. Alpha coefficients are calculated on the domains in their original form, as well as after any suggested revisions to measure internal consistency. Finally, analysis of covariance is utilized to estimate adjusted means for the various known-groups.
Known-groups comparisons provide support for construct validity when distinct groups score as one might hypothesize given the underlying construct of interest. For example, an instrument measuring depression would be expected to produce different scores for persons known to be diagnosed with depression compared to others not so diagnosed. In the case of the Core Competencies, we can hypothesize that groups, such as the administrators, would score more highly on items meant to measure the underlying construct of leadership or policy development. Therefore, empirical evidence that corroborates a priori theory might support the construct or an instrument’s nomological validity.
References
Adcock, R. & Collier, C. (2001). Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review, 95, 528-546.
Beaulieu, J. E. & Scutchfield, D. F. (2002). Assessment of validity of the national public health performance standards: The local public health performance assessment instrument. Public Health Reports, 117, 28-117.
Beaulieu, J. E., Scutchfield, D. F. & Kelly, A. N. (2003). Recommendations from testing the national public health performance standards instruments. Journal of Public Health Management and Practice, 9, 188-198.
Campbell, D. T. (1990). Recommendations for APA testing standards regarding construct, trait, or discriminant validity. American Psychologist, 15, 546-553.
Council on Linkages (2001). Core competencies for public health professionals. Washington, DC: National Association of County and City Health Officials.
Council on Linkages (2005). Prologue to the core competencies for public health. Retrieved September 26, 2014 from http://www.phf.org/Link.prologue.htm
Cronbach, L J. & Meehl, P. E. (1995). Construct validity in psychological tests. Psychological Bulletin 52, 281-302.
Moss, P. A. (1992). Shifting conceptions of validity in educational measurement: Implications for performance assessment. Review of Educational Research 62, 229-258.
Moss, P. A. (1995). Themes and variations in validity theory. Educational Measurement: Issues and Practice 14, 5-13.
Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw Hill.
Shultz, K. S., Riggs, M. L. & Kottke, J. L. (1998). The need for an evolving concept of validity in industrial and personnel psychology: Psychometric, legal, and emerging issues. Current Psychology, 17, 265-86.
Streiner, D. L. & Norman, G. R. (1995). Health measurement scales: A practical guide to their development and use (2nd ed.). Oxford, UK: Oxford University Press.
