A Region-based Approach to the Automated Marking of Short Textual Answers

Raheel Siddiqi

doi:10.33317/ssurj.74

Authors

Raheel Siddiqi

DOI:

https://doi.org/10.33317/ssurj.74

Keywords:

Automated assessment, short textual answers, automated free-text marking, region, Question Answer Language

Abstract

Automated marking of short textual answers is a challenging task due to the difficulties involved in accurately “understanding” natural language text. However, certain purpose-built Natural Language Processing (NLP) techniques can be used for this purpose. This paper describes an NLP-based approach to automated assessment that extends an earlier approach [1] to enable the automated marking of longer answers as well as answers that are partially correct. In the extended approach, the original Question Answer Language (QAL) is augmented to support the definition of regions of text that are expected to appear in a student’s answer. In order to explain the extensions to QAL, we present worked examples based on real exam questions. The system’s ability to accurately mark longer answer texts is shown to be on a par with that of existing state-of-the-art short-answer marking systems which are not capable of marking such longer texts.

References

R. Siddiqi and C. J. Harrison (2008) “On the Automated Assessment of Short Free-Text Responses”. Paper presented at the 34th annual conference of the International Association for Educational Assessment (IAEA), Cambridge, UK.

R. Siddiqi and C. J. Harrison (2008) “A Systematic Approach to the Automated Marking of Short-Answer Questions”. Proceedings of the 12th IEEE International Multi topic Conference (IEEE INMIC 2008), Karachi, Pakistan, pp. 329-332.

M. C. de Marneffe, B. MacCartney and C. D. Manning, “Generating Typed Dependency Parses from Phrase Structure Parses,” Proc. The fifth international conference on Language Resources and Evaluation, pp. 449-454, 2006.

R. Siddiqi, C. J. Harrison and R. Siddiqi, “Improving Teaching and Learning through Automated Short-Answer Marking,” IEEE Transactions on Learning Technologies, vol. 3, no. 3, pp. 237-249, July-September 2010.

R. Siddiqi, “Improving Learning and Teaching through Automated Short-Answer Marking”, PhD Thesis. The University of Manchester (UK), pp. 40-54, 2010.

M. S. Powell, “An Input/Output Primitive for Object-Oriented Systems”, Information and Software Technology, vol. 30, no. 1, pp 44-56, January/February 1988.

T. Mitchell, T. Russel, P. Broomhead and N. Aldridge (2002) “Towards robust computerized marking of free-text responses”. Proceedings of the Sixth International Computer Assisted Assessment Conference, Loughborough, UK: Loughborough University pp. 231-249.

C. Leacock and M. Chodorow (2003) C-rater: Automated Scoring of Short-Answer Question. Computers and the Humanities, 37 (4), pp. 389-405.

J. Z. Sukkarieh and S. G. Pulman (2005) “Automatic Short Answer Marking”. Proceedings of the 2nd Workshop on Building Educational Applications Using NLP, June 2005, Association for Computational Linguistics, pp. 9-16.

J. L. Fleiss, Statistical Methods for Rates and Proportions. New York: John Wiley & Sons, pp. 212-236, 1981.