MOTIVATION

A completely well-structured problem is a problem in which the starting statement contains all relevant

information, and there exist a limited number of relatively easily-formalized rules used to reach an

unambiguously correct or incorrect solution [Jonassen, 1999]. These problems are often found in mathematical

domains; to an expert, the simple arithmetic question “26+38=?” has a clear set of steps from the initial state to

the goal state. Most real-world problems are not so well-structured. In particular, domains like design and

cultural education contain problems that are not so tidy. When the start state, rules, or goals of a problem are not

the conditions in which they are applied are difficult to formalize, it is also difficult to form a model of student

and expert performance for that domain.

domain. A cognitive tutor is an intelligent tutoring system that compares student action to a model of correct and

incorrect behavior and provides context-sensitive feedback and problem selection. Cognitive tutors have been

effective at increasing student learning in real-world settings by as much as one standard deviation over

traditional classroom instruction [Koedinger, 1997]. However, most successful tutors have been limited to well-

defined domains like algebra and physics. Ill-defined domains may also benefit from cognitive tutoring, but this

area is only beginning to be explored. Cognitive tutors rely on the existence of a formal domain model as the

basis for other stages of tutor development like identifying problem content or designing tutor feedback, thus the

We discuss the difficulties of applying traditional modeling techniques in ill-defined domains as well as our

solutions for adapting these techniques. Finally, we examine the model created using this procedure and outline

future plans for developing a tutor. Our results suggest that while traditional modeling techniques are inadequate

for ill-defined domains, adaptations in knowledge representation, problem presentation, and experimental design

lead to effective solutions.

ASPECT IN FRENCH LANGUAGE LEARNING

Successful cognitive tutors based on student models of observable behaviors have been implemented in well-

2005]. Mastering the distinction between these is a difficult task for both beginning and advanced French students

and is reviewed often throughout programs of French instruction. This distinction is acquired by learning and

understanding the concept of aspect.

know the role aspect plays in a sentence in order to both understand temporal qualities of actions and be able to

accurately produce novel utterances. When speaking in the past in French, aspect is conveyed through the use of two

tenses: the passé composé and the imparfait. The passé composé involves a completed action, as if viewed from

an external perspective, while the imparfait involves an ongoing action, as if viewed from an internal perspective

[Salaberry, 1998]. Examples of uses of the passé composé, translated into English, are “I went to the store on

Tuesday” and “I stopped at the store,” while uses of the imparfait include “I went to the store every Tuesday” and “I

are extremely hard to infer, implicitly or explicitly, from the input” [DeKeyser, 2005]. Aspect has a relatively

clear goal state: Experts might disagree on the aspect of a sentence, but only in certain ambiguous cases where

the intention of the speaker is not clear. Difficulties in identifying aspect tend to arise because apsect is a

problem with ill-structured operators: rules are hard to formalize, the conditions under which they apply are too

specific and numerous to be described, and rules may be applied in parallel. It is difficult to find a formal

description of the rules for identifying aspect, and descriptions often do not correspond to one another.

Additionally, instructional texts often confuse rules, which will always return a correct answer when applied

correctly (e.g., “If the action occurs a single time the tense is passé composé”), with heuristics, which may be

easier to apply but are not always correct (e.g., “If the sentence contains a word like ‘once’ the tense is passé

where multiple rules apply, and the student must perform conflict resolution. For example, the sentence, “All of a

sudden, the sky was blue,” appears to be both an ongoing description of circumstances and a completed change

of state. The student has to know that this sentence is in fact not a description, but an event (as signified by all of

a sudden) and the passé composé should be used. Because aspect is an ill-defined domain which is challenging

and important for students to master, it is an ideal candidate for our attempts to model ill-structured problems.

TRADITIONAL MODEL-BUILDING AS APPLIED TO ASPECT

A cognitive model is a formal description of a problem-solving process. It includes both expert and novice

behavior, and correct and incorrect actions. It is required when building a cognitive tutor so that the tutor can

provide contextual feedback on problem-solving. Cognitive models are generally developed using a combination

of theory-driven and data-driven approaches. Although a theory-driven approach can identify what is relevant

the telephone _________ (to ring)” and asked to select the appropriate tense of the sentence. We first performed a

rational task analysis and then enhanced our model through think-aloud protocols from experts and students.

In a rational task analysis, a formal specification of the task is combined with consultation with experts to

produce a model of ideal performance and identify places where novices may make errors. The result is

generally a set of production rules (see Anderson et. al., 2004) arranged in a sequential and deterministic

structure. For example, this approach was used by Siegler (1976), who proposed a decision tree representing

children’s ideal performance on balance scale problems, identified subsets of the tree representing novice

indicates a one-time action, and a one-time action indicates the passé composé tense, then the tense of the

sentence is the passé composé.” Because the instructional texts and experts we consulted tended to disagree on

the formalization of the rules, we chose a set of five that were agreed upon by the majority of sources and

covered the full problem space. We then attempted to identify how these rules might be applied in sequence to

efficiently reach a correct result. The full decision tree is shown in Figure 2. In our model, students ask

themselves a sequence of increasingly abstract and more difficult questions. Notice that the default question is

not a yes or no question but a catch-all which covers the full problem space. It requires students to make a high-

level judgment about the action in the sentence without resorting to the heuristics targeted in previous questions

that are easy to answer but not always accurate. The model predicts that experts (and novices) go through a

sequential process of querying the sentence for features and use the first positive response to arrive at a decision.

Our next step was to collect data using a think-aloud procedure (Ericsson & Simon, 1984). In a think-aloud,

individuals are asked to verbalize all thoughts and actions while solving a problem. Participants are not asked to

explain or justify what they are doing nor are questions posed by the experimenter during the process. The result

is a stream of conscious record of the actions being performed and is thus valuable for collecting data to build or

refine a model. Think-alouds have been shown to not significantly interfere with the problem-solving processes

branch of the decision tree. Secondly, the first three rules of the tree were never mentioned. Finally, their

explanations were holistic, often using phrases such as "This sentence is ..." rather than referring to individual

parts of the sentence.

We used three subjects for the novice think-aloud. All had completed elementary French instruction so in

theory should have been able to complete the task. However, their level of expertise made a great difference in

performance. Our first subject had one semester of French and didn't remember conjugations other than the

present tense. The second subject had the most French experience, and completed the task with a high level of

automaticity. He could not explicitly articulate the process he followed when making the aspectual decisions.

The third subject had sufficient French experience, but since time had elapsed since her last instruction she had

difficulty completing some of the sub-steps (e.g. understanding the sentence, identifying the tense, conjugating

unlike the previous model which made no mention of implicit knowledge, now implicit knowledge is accounted

for by the existence of evidence gathering methods at different levels of abstraction. Expert knowledge can be

represented as a single evidence gathering method (“Identify the completeness of the action”), rather than the

more specific methods we have described. Moreover, rules can be applied in any order, represented by

specifying a list of evidence gathering methods than can be used, rather than a sequence of actions that must be

performed to solve the problem. Finally, the approach handles uncertainty by specifying a threshold for

accepting a given hypothesis, and allowing different evidence gathering methods to be weighted based on

characteristics of the problem and the skill of the problem-solver. This uncertainty also allows heuristics, or

answer itself. When students are required to explain their decisions, the tutor learns the rule the student is trying

to apply. We experimented both with rule-based and freeform explanations. Further, self-explanation, both free-

form and rule-based, has been shown to increase student learning in intelligent tutoring systems [Aleven, 2002].

of the sentence that lead students to initially apply the rule. For this information, we asked students to identify a

sentence comparable to the problem sentence with respect to the features for choosing aspect. This process

highlights the features students use to make their decision. With the added scaffolding, not only is the student

required to actively process the text but insight into the process being used to derive the answer is now available

to the tutor; they are no longer simply making a binary decision. Forcing students to make comparisons between

had recently completed at least the first semester of college French (or equivalent) but had not continued beyond

the second semester. Participants had differing levels of exposure to the passé compose and the imparfait.

Students were given a pretest, instruction regarding the use of aspect, and then asked to think out loud as they

solved the task. Students completed this activity for four different paragraphs with four different types of

scaffolding (no scaffolding, comparison example, self-explanation, combined comparison example and self-

explanation). During this phase, students were given immediate feedback on their performance. Each session

concluded with students completing a post-test and transfer assessment.

minutes) might suggest that our model doesn’t deviate much from students’ existing models. We also found

some direct evidence for the idea that students were in fact conducting evidence gathering when making their

decisions. When presented with conflicting evidence, students would verbalize the conflict and look for other

features to support one aspect or the other. Even when conflicts were absent, some students were reluctant to

make a decision based on limited information. For example, P1’s behavior and statement “[the sentence] doesn’t

say exactly when it happened, no specific time, but it’s an event” suggest that she was trying to gather more

evidence that the sentence was in the passé compose before making a final decision.

We proposed four main areas where novice behavior may differ from expert behavior. Each of these are re-

examined and supported by samples of student behavior:

previous model. A table of the heuristics including the participants who used them follows:

Method Passé Composé Imparfait

Identify the type of verb Action verb (2, 4, 6) State verb (1, 3, 7)

Identify the specificity of the

action

Identify the suddenness of the

activity

Identify a mention of times Time keyword (2, 3, 6) No mention of exact time (1, 2)

approach is a decent representation of student behavior, and an improvement over the previous model. The new

model is based on a representation that is suitable for ill-defined domains: It incorporates holistic sentence

processing, the application of rules in any order, and allows both rules and heuristics to be employed by students.

otherwise have been clear. Ultimately, the data collected fits the model. Students show an evidence gathering

approach to identifying aspect, and use the rules and heuristics that we have described to help them arrive at the

correct answer. It is important to note that this model is preliminary, and serves as a plausible interpretation of

the data. Further validation and specification of the model will be necessary.

In particular, there are still unanswered questions with respect to the completeness of the model. Because we

Additionally, we did not do a full analysis of the “helping” strategies that students used to solve problems such

as translation of the sentence or looking for context in surrounding sentences. In the future, we intend to

incorporate those types of heuristics into our model. Finally, we limited our analysis to a high level of

abstraction, and did not look at how students parse the sentence to identify features that may be relevant for the

conditions of the rules in our model. We believe that scaffolding and tutoring mechanisms can be built from our

model which do not require such a fine grain of analysis to be effective.

FUTURE DIRECTIONS

Developing an accurate student model marks the beginning of full tutor development. With our current model,

we plan to create a full cognitive tutor to be deployed and evaluated in an online French course. The tutor will

likely incorporate the scaffolds used during the think-aloud procedure, but future studies are also planned to

Skogsholm for their help with this project.

REFERENCES

Aleven, V., & Koedinger, K. R. (2002). An Effective Meta-cognitive Strategy: Learning by Doing and

Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y . (2004). An integrated theory of

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. W. (2002). Learning from examples: Instructional

Cadierno, T. (1995) Formal Instruction from a Processing Perspective: An Introduction to the French Past Tense.

Gamper, J. & Knapp, J. (2002). A review of intelligent CALL systems.Computer Assisted Language Learning,

Ishida, M. (2004) Effects of Recasts on the Acquisition of the Aspectual Form te i(ru) by Learners of Japanese as

Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999). Task analysis methods for instructional design. Mahwah,

Koedinger, K. R.; Anderson, J. R.; Hadley, W. H.; and Mark, M. A. (1997). Intelligent Tutoring Goes to School

Mayo, M., & Mitrovic, A. (2001). Optimising ITS behavior with Bayesian networks and decision theory.

Robert M. DeKeyser. (2005) What Makes Learning Second Language Grammar Difficult? A Review of Issues.

Salaberry, R. (1998). The development of aspectual distinctions in L2 French classroom learning. The Canadian

von Stutterheim, C. (1991). Narrative and description: Temporal reference in second language acquisition. In