Evaluation of a Web-based Language Learning Environment

Steve Cushion & Dominique Hémard

London Guildhall University

The CALL Project at LGU

Design Brief: Develop a Web-based, hypertext environment accommodating:

interactive grammar exercises
an authoring tool for easy inputting of language learning material

Main Characteristics

Expertise: Java programming & User Interface Design

Context of Use: curriculum-based, formal classroom teaching & open access

Users: Language students & language staff (at LGU initially)

User Interaction: directed & exploratory navigation (sequential & relational links)

High Level Design Objectives to be Evaluated

Students:

Usability & attractiveness of interface

Applicability (utility)

Accessibility

Teachers:

Facilitate & help generate input of new language material by teachers with little ICT experience

Help teachers to conceptualize a structural model (Home page) within which to produce a complete package of exercises.

Evaluation: Methodology

Emphasis on qualitative data gathering

Formative evaluation:

Expert reviews / discussions

Summative evaluation:

Questionnaires

Workshops

User walkthroughs in laboratory conditions

Expert Reviews

Identification: cross-departmental expertise in language teaching, multimedia, Java programming.

Formal and informal discussion on aspects of interface, content & planned interaction.

Recommendations: technical reports identifying problems & suggesting solutions.

Questionnaires

Student profile (ICT experience & confidence)

CALL exposure & access

Usability, appropriateness, appeal of interface

Comments on functionality and usefulness

User Walkthroughs

Appropriateness of method:

Focus on learnability and usability of a system.
Good for obtaining qualitative data related to the interface, interaction and exploratory learning.
Good for collaborative work with staff and students

Aims of the User Walkthrough

To consider the design of the interface: Is it clear, uncluttered, consistent, unambiguous, appealing, intuitive, and robust? Does the way it looks adapted to match your expectations?
To consider the user interaction: Is it easy to use? Do you know what to do? Are the instructions explicit enough? Do you understand the structure of the designed environment? Can you navigate through it without getting disoriented? Can you access the program easily? Do you have control over the interaction or not? Is the interaction meaningful?
To consider the functionality: Are available functions appropriate for the tasks to be carried? Are the navigational commands clear enough? Are the links easy to use? Is there feedback on actions taken? Is the feedback for given tasks appropriate?
To consider the multimedia impact: Does it correspond to your idea of what a multimedia interface should look like? Is the multimedia presentation appropriate and useful? Does it provide additional information stimulating the interaction and the learning process?
To consider the language learning experience: Are the learning objectives clear enough? Should grammar be contextualized? Do you see the role of the computer as a support? A convenient tool? A tutor which should instruct and assess? Is the material provided appropriate for self-access? Distance learning? Is the language learning environment authentic enough? Is the language immersion satisfactory?
To further debug the program (if appropriate): Spot and identify errors, faulty links, typos, etc.

Examples of Qualitative data from User Walkthroughs

Interface design:

Students were positive about the design: simple and recognisable

But:

Critical about the display of feedback: would have liked more systematic scoring and display of answers.
Critical about the display of the interactive space: would have liked length and difficulty of exercises better displayed.

Interaction:

Overall, the interaction was linear and exercise-based.
Inexperienced students adopted a structural model and only partly explored the space.
Experienced students adopted a more task-based, cognitive approach and made fuller use of the interactive space.

Functionality:

The students liked:

The navigational commands.

The checking device.

But would have liked:

A more flexible and customisable feedback mechanism (greater tutoring element).

More systematic control over available functions

Multimedia:

The students liked the sound facility, but would have liked greater control over it.
Inexperienced students never explored the supplementary content-based material.
Experienced students looked at the supplementary content-based material but their interaction remained passive.

Language learning experience:

Interaction: conditioned by learning context.
Learning pathways: student-controlled.
Role of computer: support mechanism.
Added value: recognition & facilitation.
Learning experience: sustained linguistic student / student interaction.

Main Design Issues

Tasks: finite and manageable entities with greater use of sound.
Interaction: greater multimedia links and student control.
Support: systematic feedback and scoring.
Usability: clearer instructions with access to glossary

Lessons?

Increase scope, range of interaction and multimedia potential of authoring tool with sound facility.

Further enhance the links between language and related content within the interactive space (grammar, multilingual links etc.).

Further develop integration of CALL material into language curricula.