Ashley, Kerr, Banwell, MacPherson and Heron (2016: 125) remark that work-integrated learning coordinators should evaluate in order to “plan and improve programming to better meet stakeholders’ needs (e.g., student, mentor, institutional and societal needs) and to continually improve the educational quality of the work-integrated learning experience”. They site Scriven (1967) who defined evaluation as ‘judging the worth or merit of something’ (p. 124) and also site Mertens and Wilson (2012) who qualified that programme evaluation “uses formal methodologies to provide useful empirical evidence about public entities (such as programs, products, performance) in decision-making contexts that are inherently political and involve multiple often-conflicting stakeholders, where resources are seldom sufficient, and where time-pressures are salient.”
Chapter 7, ‘Evaluating your WIL programme’, of the ‘HEQCO Practical guide for work-integrated learning’ (Ashley et al 2016: 123 to 149) outlines a six-step evaluation process, reflected in the following diagram (p. 129):
Step 1: Develop and evaluation question
According to Ashley et al (2016: 127) “There are three common purposes for evaluation: to gain a better understanding of the needs within a particular context (needs assessment), to identify ways to improve the implementation of the programme (implementation), and for the purposes of reporting the degree to which the programme achieves its intended outcomes (evaluation of programme effectiveness).” illustrated as follow (p. 130):
With regard to implementation six types of implementation evaluation are mentioned (p. 132) and with regard to assessment of effectiveness four types (p. 134)—these are tabled as follow:
Step 2: Choose an evaluation paradigm
Ashley et al (2016: 127) identify “four primary paradigms that are applied to programme evaluation: postpositivist, constructivist, transformative and pragmatic.” and outline (p. 135) the four as follow:
With regard to the postpositivist evaluation paradigm the Kirkpatrick Four Levels Model for the evaluation of training programmes is documented (Ashley et al 2016: 135-6). The levels are (1) Reactions or participant satisfaction; (2) Learning or improved knowledge or skills of participants; (3) Behaviour as in the performance improvement of participants; and (4) Results, which pertains to the achievement of objectives. In the social sciences the postpositivist paradigm is regarded as means of exploring human behavioural laws with the view of improvement.
The pragmatic evaluation paradigm considers common sense and practical thinking. The Shufflebeam CIPP Model—Context, Input, Process and Product—illustrated first below; and the RE-AIM Framework—Reach, Efficacy, Adoption, Implementation and Maintenance—second below, are advocated (Ashley et al 2016: 138-141). The pragmatic paradigm rejects the discovery of truth by means of scientific methods.
“The constructivist approach to evaluation attempts to understand meaning from the perspective of the persons who have the experiences” states Ashley et al (2016: 142). They present both the Scriven’s goal-free approach to evaluation and the case study approach. The intent of the constructivist approach is to make visible the understandings of stakeholders.
The transformative paradigm focuses on perspectives of marginalised groups and aims to address power inequalities versus privilege and achievement of social justice. Ashley et al (2016: 143) present a number of transformative theories portrayed in the diagram below. A participatory transformative approach to evaluation is recommended.
Ashley et al (2016: 146) encourage the consultation of institutional codes of ethics prior to engaging in evaluation. Although ethical considerations are important throughout, it is of particular importance with regard to sampling. Informed consent, confidentiality (meaning collection, analysis, storage and reporting in such ways that data cannot be traced back to origins), and anonymity (absence of any uniquely identifiable information). Ashley et al (2016: 147) outline five guiding principles with regard to evaluation of work-integrated learning programmes, namely:
Ashley, S., Kerr, G., Banwell, J., MacPherson, E. & Heron, A. (2016). A practical guide for work-integrated learning — Effective practices to enhance the educational quality of structured work experiences offered through colleges and universities. Toronto, Ontario: Higher Education Quality Council of Ontario. Electronically available from: http://www.heqco.ca/en-ca/Research/ResPub/Pages/A-Practical-Guide-for-Work-integrated-Learning.aspx