Melanie Powell

  • Statistical analysis of the impact of computer based formative assessment on summative assessment and skills for business students

    Melanie Powell, Derby University, UK

    Derek Fry, Derby University, UK

    Session 3h, Tuesday 10.10

    Research paper

    Theme addressed:

    • Teaching methods

    Statistical analysis of the impact of computer based formative assessment on summative assessment and skills for business students.

    Failure rates for first year economics and statistics modules in business education have been high because they require practice and progression over short time spans. A progressive set of computer based formative learning and assessment materials were created with extensive feedback to address this problem by develop self-regulating learning skills amongst first year students. The formative materials were designed to follow recent theoretical models linking practice and structure to formative assessment, and formative assessment to self-regulated learning and positive impacts on retention and learning outcomes. This paper presents the theoretical reasoning and results of a statistical analysis of the impact of computer based formative assessments on summative assessment and skills.

    Recent literature on learning in higher education calls for a more theorised base for learning and assessment strategies. (McAlpine, 2004; Scanlon & Issroff, 2005; Yorke, 2003). In particular, the role of formative learning and developing self-regulated learning has been emphasised, (Black and Wiliam, 1998; Heikkila and Lonka, 2006; McAlpine, 2004; Nicol and Macfarlane-Dick, 2006; Yorke, 2003), alongside the potential for computer based assessments (Gipps, 2005). These models were used to aid the design of the computer based formative learning and assessment materials with a view to improving learning and reducing failure rates.

    The formative assessment was designed using activity theory (Scanlon & Issroff, 2005). Computer assessment is the tool mediating between subject (student) and object (learning). Student information is the rule mediating between subject and community (the university), and the criteria determine the division of labour mediating between community and object. The formative system aimed to maximise effect taking into account potential conflicts around mediating factors between the subject, object and community as Scanlon and Issroff (2005) suggest.

    Computer based assessment was chosen as progress could be structured and explicit, student activity could be monitored, and instantaneous feedback provided with multiple chances to repeat and practice. The materials contained both learning activities and formative assessment with extensive feedback loops, advice on improvement, multiple attempts, staged development and continuous formative grades. Students could progress at their own speed and assess their own development. Students who failed to engage or progress could be identified quickly and were given additional personal and clinic support. The materials were explicitly related to class materials and summative assessment. These features aimed to meet the seven principles of good practice in the model of self-regulated learning and feedback in Nicol and Macfarlane-Dick (2006). They also reflect the critical role of practice, structure and formative feedback identified in McAlpine's (2004) model for instruction based on theories of learning.

    Data was collected on participation, progress and outcome measures, as well as self-recorded skills and demographic details using experiment design. The control group comprised a first year HND cohort with similar modules but without the formative assessment. Multivariate statistical analysis was used to analyse the effect on summative grade achievement and self-reported skill acquisition and comparison with prior years. The results indicate significant effects of formative assessment on outcome measures.

    References

    • Black P & Wiliam D (1998), Assessment and classroom learning. Assessment in Education, Vol 5, No 1, pp7-74.
    • Gipps C (2005), What is the role for ICT-based assessment in universities?. Studies in Higher Education, Vol 30, No 2, pp171-180.
    • Heikkila A & Lonka K (2006), Studying in higher education: students' approaches to learning, self-regulation, and cognitive strategies. Studies in Higher Education, Vol 31, No 1, pp99-117.
    • McAlpine L (2004), Designing learning as well as teaching. A research-based model for instruction that emphasizes learner practice. Active Learning in Higher Education, Vol 5, No 2, pp119-134.
    • Nicol D and Macfarlane-Dick D (2006), Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, Vol 31, No 2, pp199-218.
    • Scanlon E & Issroff K (2005), Activity Theory and Higher Education: evaluating learning technologies. Journal of Computer Assisted Learning, Vol 21, pp430-439.
    • Yorke M (2003), Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. Higher Education, Vol 45, pp477-501.