Evaluation research
Evaluation research is a form of applied research that has been around for ages but which has gained significant prominence over the last few decades. The aim of this paper therefore is to define evaluation research and compare it with basic social research. In addition, it will describe the various types of methodologies applied in evaluation research as well as the different types of evaluation research according to the timing of the evaluation. The final part of the paper will constitute a review of a published evaluation research study published. This review will focus on the intent of the study, methods employed for data collection, data analysis and the conclusions of the authors of the study. Last but not least, I will provide and justify my opinion on the conclusions arrived at by the authors of the study.
What is evaluation research?
Evaluation research is a form of applied social research which employs scientific research methods to plan and monitor the implementation as well as the operations of both new and existing programs. It also entails evaluating the effectiveness of programs as well as clinical practices in achieving their goals.
Comparison of evaluation research and basic social research
Evaluation research is similar to basic research in that evaluation research applies the general techniques of basic social research albeit in a special manner. However, some important differences between the two do exist. For one, the findings of evaluation research have relatively immediate practical uses as far as the assessment of the operation of programs is concerned. On the other hand, basic research focuses more on hypothesis testing and information gathering. It is geared more towards the acquisition of knowledge and furthering of learning. Secondly, basic researchers have more control over the issues they would like to investigate. This is not so for evaluation researchers because they have to take into consideration the input of the sponsors of their studies especially in regard to the form and content of their research. Moreover, the very nature of evaluation research imparts on it a judgmental quality which is not part and parcel of basic research. Further, whilst the findings of basic research are likely to be made public, those of evaluation research are more often than not are only disclosed to the organizations and persons who sponsored the studies or published in books or journals. Last but not least, while basic research adheres to the principles of producing a scientifically sound study, the same cannot be said for evaluation studies. This is because evaluation research takes place within the context of ongoing programs whose demands may be contrary to those of sound scientific practices (Monette, Sullivan & DeJong, 2010, p.321).
Types of evaluation research
Evaluation research can be conceptualized as either being formative or summative in respect to the timing of the study. Formative evaluation is conducted before or during the implementation phase of programs. Summative research is on the other hand done at the end of a program. Formative evaluation research is purposed to provide information that can be used to guide the process of planning, development and implementation of a particular program (Monette, Sullivan & DeJong, 2010, p.323). To this end, it critically assesses and examines important program or object attributes like the organizational context, inputs, procedures, personnel, delivery of the program or the quality of its implementation. In essence therefore, formative evaluation aims at improving or strengthening the programs or objects being evaluated. Examples of types of formative evaluation include needs assessment, process evaluation and process evaluation (William, 2006). In contrast, summative evaluation focuses on the outcome or effects of a program. It therefore concerns itself with the examination of the effects of a program on target populations as well as on non-target populations as well as the relative costs of the program or object (Monette, Sullivan & DeJong, 2010, p.323). In a nutshell therefore, summative evaluation is geared towards assessing the efficiency and effectiveness of programs or objects under study. Examples of forms of summative evaluation include outcome evaluation, impact evaluation and cost-benefit analysis (William, 2006).
Evaluation research methods
Contrary to popular notions, evaluation research is not a new form of research because it conceivably employs the scientific research methods used in basic research. These methods are normally labeled in accordance with their primary research designs such as survey and experimental methods as discussed below. Survey research method is defined as a research strategy in which data is collected from a sample or the whole population for the purposes of assessing the relative distribution, incidence and relationships of naturally occurring variables. Experimental research on the other hand refers to the method whereby the researcher manipulates one independent variable, controls other relevant variables and then observes the effect of the independent variable on one or more dependent variables. Experimental research designs can further be divided into factorial, covariance and blocking designs (William, 2006).
Historical research refers to the strategy whereby data relating to past occurrences is collected systematically and evaluated in an objective manner with an aim of testing hypothesis regarding the effects, causes and trends of those specific events that may help elucidate on current events as well as predict future events. Operations research on the other hand refers to the application of scientific method to the operations of management for the purpose of aiding managerial decision making. Modeling otherwise known as simulation is another research method employed in evaluation research, it entails observing the behavior of an analogous or representative system so as to determine the performance of the real system. Systems analysis utilizes the same concept as operations research. It however differs from operations research in that it emphasizes more on the total system as well as the manner in which the various components that as a whole make up the system interact (Connaway & Powell, 2010, p.81).
Delphi technique is a procedure that uses sequential questionnaires to obtain the opinions of experts on issues that are by their very nature non-factual. Content analysis is a systematic analysis of the frequency of occurrence of concepts, words or phrases in a body of communicated material such as a book or film. Other basic research methods that can be used include metanalysis which is a systematic review of a number of studies, quasi-experimental designs which are similar to experimental methods but do not use random sampling amongst others (Connaway & Powell, 2010, p.81).
Analysis of a published evaluation research journal article
“Traditional Chinese medicines in the treatment of hepatocellular cancers: a systematic review and meta-analysis” was a meta-analytic study by Wu, Dogoua, Eyawo & Mills conducted in 2009. It was first published on August 12 in the Journal of Experimental and Clinical Cancer Research. The study sought to review in a systematic manner data from a number of randomized clinical trials (RCTs) in order to establish the efficacy of traditional Chinese medicine (TCM) products as far as the treatment of liver cancer is concerned. The researchers searched for RCTs from six English and two Chinese language databases. Initially, 45 RCTs were identified but after application of the inclusion criteria only 37 studies met the requirements for the study. To be eligible for inclusion in the meta-analytic review, the RCTs had to have randomly allocated the patients to either an active TCM formulation treatment group or a control group on placebo treatment or no treatment. It also had to have enrolled patients with liver cancer who were more than 18 years of age. Data extraction was done by three reviewers based on a standard pre-piloted form.
Data analyzed included clinical outcome measures and laboratory values such as the stage of the disease, the Child-Pugh Score, Karnofsky performance, survival rate and the response evaluation criteria in solid tumors (RECIST). Priori Data on priori covariates for the TCM formulations was also corrected. The latter group included data pertaining to the quality of the trials, protocols and outcomes assessed.
Regarding the statistics employed in the study, both descriptive and inferential statistics were used. Findings from the meta-analysis were reported in terms of median, relative risk, odds ratio, statistical level of significance, confidence intervals. Other statistical tests conducted included the two-tailed 5% alpha, Begg-Mazmundar test, Egger’s test and Horlbond-Eggers test.
The authors of the study concluded that based on their evaluation; there was compelling evidence that TCM might provide potentially new and effective therapeutic options for the treatment of hepatocellular cancers. They however recommended that their findings be confirmed via high-quality, transparent clinical trials in Western settings as opposed to China. I agree with the conclusions of the study based on my critical appraisal of the study methodology, data analysis, results, strengths and limitations of the study. The authors have elucidated upon these aspects in an expansive manner. Additionally, the authors adhered to the principles of research and thus the study is scientifically sound.
Conclusion
In conclusion therefore, evaluation research is a form of applied social research that employs basic research methods like survey and experimental in a special way to plan, monitor and evaluate the effectiveness and efficacy of programs and clinical practices. It differs from basic social research on certain attributes namely, the uses of the findings, the amount of control researchers have as far as decision making on issues to investigate are concerned amongst others. Evaluation research can be classified into formative and summative evaluation research depending on the timing of the study.
“Traditional Chinese medicines in the treatment of hepatocellular cancers: a systematic review and meta-analysis” was a meta-analytic study by Wu, Dogoua, Eyawo & Mills conducted in 2009. Data was collected by a systematic search of RCTs from English and Chinese databases. In particular, data relating to the clinical outcomes and laboratory values of the patients in these studies was collected. Both descriptive and inferential statistics were employed in reporting the findings of the study. The study concluded that based on the RCTs reviewed; there was compelling evidence in favor of TCM as far as the treatment of hepatocellular cancer is concerned. They therefore recommended that there was need for other studies to be done on the same. I concur with the conclusions of the study mainly because they are congruent with the findings and the authors of the study adhered to the principles of scientific research.
References
Connaway, L.S., & Powell, R.R. (2010). Basic research methods for librarians. Santa Barbra,
California: Greenwood Publishing Group.
Monette, D.R., Sullivan, T.J., & Dejong, C.R. (2010). Applied social research: A tool for the
human services (8th ed.). Belmont, CA: BROOKS/COLE Cengage learning.
William, M.K. (2006). Research methods knowledge base. Retrieved from
http://www.socialresearchmethods.net/kb/index.php
Wu, P., Dugoua, J.J., Eyawo, O., & Mills, E.J. (2009). Traditional Chinese medicines in the
treatment of hepatocellular cancers: A systematic review and meta-analysis. Journal of
Experimental & Clinical Cancer Research, 28, 112. doi: 10.1186/1756-9966-28-112
