Maths CAA Series: November 2003
Page Guide: Home > Articles > maths-caa-series >
Abstract
This short article describes the efforts being made to extend the existing Question and Test Interoperability (QTI) specification to support numerical assessment as used in the Mathematics community. The article reports on a meeting held at Heriot-Watt University for interested parties and describes some of the critical issues and the steps taken to address them by the community.
1 Introduction
Although exploited to their full potential in research, many practitioners feel that computers are not utilised as effectively in the delivery of education. In general, this failure can be attributed to two factors:
Firstly, small scale content development is inextricably tied to progress in technology - most effort is devoted to developing new approaches and materials for cutting edge technology, rather than refining existing approaches. The consequence of this is that enormous effort is wasted re-creating the same content over and over: content that is incompatible with older hardware and whose educational validity is untested.
Secondly, large scale content creation has been driven by quantity and cost not quality - so eLearning content is typically text-rich and lacking opportunities for interaction, as this is the easiest and cheapest to produce. Once content exists, there is little incentive to improve it.
In light of these problems, there has been an international drive to underpin future provision of eLearning with standards. Widely adopted standards would reduce wasted effort and facilitate re-use and interoperability of content and systems. This in turn will result in an increase in the quantity of eLearning content available and promote eLearning to the mainstream. Educational Technology standards and specifications cover all aspects of online delivery of education, from enterprise level interoperability of different services, right down to the storage and description of content. Within UK Higher Education, significant investment from JISC (Joint Information and Systems Committee) has been used to ensure an active role in developing these new standards. The JISC funded Centre for Educational Technology and Interoperability Standards (CETIS (1)) provides the community with representation on international standards bodies, and funding programmes such as X4L (2), a JISC initiative, have created a community of practitioners using these standards which is both committed to the goals and aware of the challenges.
One of the key specifications for the academic community is the Question and Test Interoperability (QTI) Specification (3). QTI is one of a number of specifications developed by the Instructional Management Specification (IMS) Global Organisation (4), a US based body comprised of key players from both business and academia whose goal is to provide standards across the whole domain of e-Learning. QTI is concerned with developing standards for tests delivered electronically - and attempts to specify standard methods for describing questions, answers, marking schemes, test databases and feedback. Interoperability between systems would allow questions written for one system to be used in another system or for questions to be stored in neutral question banks, not tied to a particular assessment system.
Now is a key time in the QTI development lifecycle. After an initial release in 2000, the specification has previously undergone minor revisions, but has now been put forward for major revision to version 2.0. The main need for a version 2.0 is to harmonise QTI with other closely related specifications produced by IMS - specifically Metadata, Content Packaging and Simple Sequencing. It is hoped that version 2.0 will also provide an opportunity to make the specification a little more robust and rectify the omissions in the original specification.
A key failing of the original specification was its poor support for numerical assessment. For instance the QTI specification will handle straightforward multiple choice questions, but has no support for expression evaluation, randomisation within questions and precision checking. This lack of support for numerical assessment was unfortunate, as numerical disciplines such as mathematics and engineering could benefit greatly from well written online assessment and there are examples of such projects creating these types of materials around the UK (5,6,7). Nevertheless, a few groups have started to use the QTI specification for numerical assessment and have tried to extend the QTI specification to support their needs.
This article reports, in part, on a meeting held at Heriot-Watt University on 27th August 2003 (8). The meeting was called to bring together people trying to use or considering using the IMS Question and Test Interoperability (QTI) specification to deliver computer based assessment in mathematics and other numerical disciplines. It was hoped that the meeting would:
· Foster the establishment of a focus group of practitioners;
· Help the group move towards a consensus on key needs of the community;
· Provide a forum to debate the key issues; and
· Generate a summary of these key issues for presentation to IMS as they consider updating the QTI specification.2 Presentations
The meeting was informal, with discussion based loosely around presentations from Niall Barr (Strathclyde), Graham Smith (Leeds), Dick Bacon (Surrey) and Manolis Mavrikis (Edinburgh).
2.1 QTI Basics
As an introduction to the day, Niall Barr provided a quick discussion of the QTI specification, using as his basis the QTI interactive documentation pages he has prepared at the University of Strathclyde (9). Niall showed the hierarchical structure of elements and explained that little if any work had been done with most of the QTI specification (the Objectbank, Assessment and Selection elements), with most practitioners concentrating on the Item element - i.e. on individual question types.
Niall gave a brief description of the main response types currently supported by the QTI specification (for more details see the QTI Interactive Documentation site (9)
- Response_num (numeric)
- Response_lid (local id)
- Response_xy (x,y pair)
- Response_str (string)
- Response_grp (group)
- Response_extension (extension - the catch all).
Each response type has an associated cardinality:
- one: for multiple choice questions,
- multiple: for multiple response, and
- ordered: for lists.
Each response also has an associated render type:
- render_choice (checkbox or radio),
- render_hotspot (on an image, graph etc.),
- render_slider (likert scale type),
- render_fib (fill in blanks).
This provides many different potential combinations, most of which have never/will never be tried, and which highlight pedagogical and implementation issues. For example in a multiple choice question the default is to use radio buttons - but radio buttons trap the user into providing an answer - no option to un-select. Check boxes would be better. Further elements (concerned with response processing) were also discussed briefly.
Using these elements, the test author (or authoring system) can specify a question, responses, marking schemes and rendering rules in XML which can be read by any QTI based system. Many companies now offer systems which claim to be QTI compliant and exchange of questions within these systems is possible. Over time, a best practice consensus will emerge. Further details of the IMS QTI Specification are available in a recent MathSTORE article (10).2.2 Implementing QTI and Supporting Numerical Question Types
Dick Bacon then gave a talk summarising the progress he has made implementing QTI (and QTI like) elements in his SToMP system (11). Dick has tried to achieve the same functionality with QTI (and QTI like elements) as he had when the SToMP tool did not use a QTI based approach for assessment. As SToMP has been used extensively with students, the shortcomings of the QTI specification for serious numerical assessment have been clearly demonstrated and practical solutions developed.
Dick began by listing some of the ways in which the current QTI specification is ill equipped for numerical assessment. There is:
- no support for variables - this means that you can't set randomised questions;
- no expression language - either for answer evaluation or user input.;
- no support for precision checking - this means there is no reliable way to check for significant figures or decimal places;
- no support for accuracy checking - either relative or absolute;
- no support for non base 10 numbers; and
- no support for rendering of graphs.
Dick Bacon has written extensions to QTI elements to address all of these requirements. A document was passed around on the day which summarises this work (12) and this work has also been presented at the 7th International CAA Conference (13). Dick showed each of his extensions in turn and highlighted those issues (e.g. lack of algebraic manipulation, further work on graphs) which remain unresolved. Dick's current set-up uses the MathInput tool written by John Appleby and included in the DIAGNOSYS software (14) to allow the user to enter expressions.
2.3 An Online QTI Implementation
Graham Smith then talked about some of the work he has been doing for CETIS, working through the QTI specification and creating reference examples of each of the question types - in an effort to clarify issues and highlight errors in the current specification. Graham uses a web based Assessment Engine which renders QTI content as HTML (15). In addition to creating reference questions, Graham has also undertaken to implement the QTI extensions written by Dick Bacon and has so far managed to recreate most of the initial numerical question types. No work has been done on the graphing and expression types. Implementing the QTI Extensions in another system is a key step in the process of confirming the robustness of these extensions.
2.4 The WALLIS System
After lunch, Manolis Mavrikis gave a talk based around the work he has been doing on creating feedback rich environments for Mathematics teaching (WALLIS (16) as discussed in a recent MathSTORE article (17).
The WALLIS system is centred on the provision of rich (synchronous) feedback with support for partial credit, and adaptive testing based on responses. At present, the WALLIS system uses MAPLE to provide support externally for random numbers. Manolis discussed his work in using OpenMath as an expression language, which is then rendered in Presentational MathML.3 Discussion
After the four presentations were finished, the afternoon session took the form of a discussion of the major issues. Firstly, however, Steve Lay (University of Cambridge Local Examinations Syndicate), who represents the UK HE community within IMS gave an overview of the current position.
3.1 Current Status of QTI
Steve Lay started off with a summary of the current status of QTI and its possible evolution over the next few months. Steve reported that IMS have become interested in QTI (they have a Special Interest Group on Interactive Content which is looking at harmonisation of the QTI, Content Packaging and Simple Sequencing specifications which do not currently integrate well, but are seen as complementary).
There is an acknowledged need for a QTI 2.0, based on the realisation that much of the original specification has remained unused as it is too complex, whilst the simple elements have been found inadequate for many applications (e.g. numerical assessment as discussed here). The working group of QTI 2.0 has been set up and includes representatives from Harcourt, Thomson ETS, CETIS (Steve Lay himself), Texas instruments and Question Mark Computing. The timescale for the group was to have a proposal document tabled by mid-September 2003 which will then go before IMS for voting. If accepted, the specification will be written to a short timescale (complete within 6 months by Easter 2004). The scope of the new specification will be the <item> element and below - as in reality these are the only elements that have so far been used. In that respect, IMS have opted for the practical solution - to improve what is being used. (Update: The proposal was accepted and work has now started on version 2.0 due for public draft release in early 2004).
After the meeting, Steve Lay summarised the issues raised in this meeting in a document to be tabled during the QTI v2.0 specification process and the main points are reproduced below. Further details are available (18).
Those using assessment for mathematics and science in HE have use cases that aren't covered by the existing specification. In particular:
- They want to score student responses on the basis of precision and accuracy.
- They need to be able to use additional variables that are assigned when a question is 'instantiated' for a candidate and that can be assigned randomly according to given constraints. The variables need to be displayed in the question and used in processing the student response and require some form of expression language to enable relative values to be assigned, displayed and tested.
- They need to be able to display more complex mathematical expressions as part of questions, ideally not as embedded images.
- They need to be able to accept more complex mathematical expressions as responses from the student and to be able to test these in response processing using notions such as "equivalence".
3.2 Priorities
Some priority discussion areas were identified and these formed the basis for most of the afternoon discussion. These were:
- Expression Processing
- Expression Display (input and output)
- Variable handling
a. EXPRESSION LANGUAGE
The choice of a suitable expression language is key. At present, Dick has one implementation, but other options would include Maple, CMath (the expression library for C programming language), MathML or OpenMath (semantically clearer than MathML). It was decided that the best options were MathML as it could be the root of many things (expression evaluation, display etc). and OpenMath, on the basis that it works well with MathML. Dick Bacon was worried about the possible overhead of using a language such as MathML - much harder to parse than a traditional programming languages' Math expression language. One of the failings of MathML is that it does not have in-built support for units, though solutions to this have been discussed (19).
It was agreed among the group that it would be best to try and create some examples using MathML and OpenMath. The examples should be challenging (Summation, Matrices) to stretch the candidates.
Since QTI is an attempt to encourage interoperability, then MathML is the obvious language for exchange, even if individual tools use different formats internally. Using MathML has the added bonus of using something which is an existing standard in related domains. MathML also lends itself to opportunities for making a core set of supported expression types which could be extended at a later stage.
b. EXPRESSION PROCESSING
It was recognised that another area of importance was expression processing, and in particular the issues on the simplification of expressions, including use of double negatives, brackets, term collection, fractions, and factorisation.
It was suggested that generating examples within the group and circulating these would be a good way of identifying the specific issues.
c. GRAPHS
Not enough work has been done on graphs yet to be of use, but identifying core properties for exchange would be useful and possible. Other systems (e.g. JeLSIM tools (20) and GnuPlot (21)) could be examined in this regard.
3.3 Other Issues
A number of other issues were identified and discussed briefly
a Steps and Partial Credit
There was some discussion of what an Item was, with definitions being governed by environment (a screen full, a HTML page) or pedagogy (the smallest unit which can be marked independently). If an Item is scored all at once, then a Section could be a question with a number of parts, but this interfered with the traditional view of an item. Steve Lay said that the simplistic, view taken at the time of writing the original QTI specification was that an Item would be the granularity of question that sits within an item bank and that because item banks were largely filled with MCQs this was convenient - but fitted less well with multi-part questions which were really more like selections.
b Simple Sequencing
Although QTI contains some in-built sequencing functionality (the ability to progress through content based on conditional rules), it was recognised that with the advent of a separate IMS specification on Simple Sequencing that this functionality might be deprecated (no one is using it yet anyway). Simple Sequencing is seen as a key to adaptive and formative assessment, where the distinction between assessment and learning is blurred. For this reason, greater integration of QTI and other IMS specifications is seen as a good thing.
c How might the Mathematics Extensions eventually fit into QTI?
There was some debate on what the eventual fate of this work would be. On one level, any move by the community to work together would be of benefit to the community, even if no formal extension to the specification emerged. IMS may be reluctant to take all the extensions developed on board as this would mean anyone seeking compliance would need to satisfy quite specialised criteria. Rather, it might be an idea to try and specify a set of extensions in addition to the core IMS QTI specification. It was generally agreed that better documentation of existing work would be helpful in presenting the views of the group.
4. Next Steps
The group took away a number of actions, mainly to produce examples and test the options presented. In addition further development of SToMP and WALLIS will continue to refine the needs of the community. There are plans to implement QTI functionality in the CUE Assessment Engine in the near future (22). CUE already has strong support for numerical expressions and randomisation and so should provide an ideal system in which to attempt implementation of the QTI-Maths extensions. The group have agreed to use the 'QTI-Implement' JiscMail group (23) to conduct further discussions on the subject. The progress of the group can be followed by visiting the 'QTI Extensions for Mathematics and Science' page on the SCROLLA (24) web site (8).
References
1 CETIS: The Centre for Educational Technology and Interoperability Standards, http://www.cetis.ac.uk/
2 X4L: The JISC eXchange for Learning programme, http://www.jisc.ac.uk/index.cfm?name=programme_x4l
3 QTI: Question and Test Interoperability, http://www.imsglobal.org/question/index.cfm
4 IMS: The Instructional management Specification Global Learning Consortium, http://www.imsglobal.org/
5 AIM: Alice Interactive Mathematics, http://web.mat.bham.ac.uk/C.J.Sangwin/aim/, from http://aimmath.sourceforge.net/
6 CALM: Computer Aided Learning in Mathematics, http://www.calm.hw.ac.uk/
7 CALMAT, http://www.calmat.gcal.ac.uk/
8 QTI Extensions for Mathematics and Science, http://www.scrolla.hw.ac.uk/qti/
9 QTI Interactive Documentation, http://ford.ces.strath.ac.uk/QTI/
10 IMS Question and Test Interoperability, Bacon, R, LTSN MathSTORE CAA Series August 2003, http://ltsn.mathstore.ac.uk/articles/maths-caa-series/aug2003/index.shtml
11 SToMP: Software teaching of Modular Physics, http://www.stomp.ac.uk/
12 Suggestion for QTI support for question variables, expressions and graphs , http://www.scrolla.hw.ac.uk/qti/stmp_qvarv1p3.doc
13 Assessing the use of a new QTI tool within Physics, Bacon, R, 7th International CAA Conference, http://www.lboro.ac.uk/service/ltd/flicaa/conf2003/pdfs/bacon.pdf,
14 DIAGNOSYS, http://www.staff.ncl.ac.uk/john.appleby/diagpage/diagindx.htm
15 http://ford.ces.strath.ac.uk/QTI/, then follow link to 'Graham Smith's CETIS test engine'.
16 WALLIS: Web-based Assistant for Learning on-Line Intelligent System, http://www.maths.ed.ac.uk/~wallis/
17 Incorporating assessment into an Interactive Learning Environment for mathematics, Manolis Mavrikis and Antony Maciocia, LTSN MathSTORE CAA Series June 2003, http://ltsn.mathstore.ac.uk/articles/maths-caa-series/june2003/index.shtml
18 http://www.scrolla.hw.ac.uk/qti/slnotes.rtf
19 http://lists.w3.org/Archives/Public/www-math/2001Apr/0013.html
20 JeLSIM, Java eLearning and Simulations, http://www.jelsim.org/
21 GnuPlot, http://www.gnuplot.info/
22 CUE, as used in PASS-IT, http://www.pass-it.org.uk/
23 QTI_Implement, http://www.jiscmail.ac.uk/lists/QTI-IMPLEMENT.html
24 SCROLLA: Scottish Centre for Research into Online Learning and Assessment, http://www.scrolla.ac.uk/