Coupling Between Metacognition and Emotions During STEM Learning With Advanced Learning Technologies: A Critical Analysis, Implications for Future Research, and Design of Learning Systems

Twenty-first century skills are key to learning, performance, academic achievement, problem solving, and future careers in science, technology, engineering, and mathematics (STEM) fields (National Research Council [NRC], 2012). Learning and innovation skills (e.g., creativity, critical thinking, communication, and collaboration) are increasingly being recognized as abilities that separate students who are prepared for the increasingly complex life and work environments in the 21st century from those who are not. We argue that the ability to acquire, retain, internalize, and use these skills requires learners to accurately and effectively monitor and regulate their cognitive, affective, metacognitive, and motivational (CAMM) processes. One approach has been the use of advanced learning technologies (ALTs) such as intelligent tutoring systems, adaptive hypermedia, serious games, and augmented reality environments designed to track, model, foster, and support these complex skills (see Figure 1 for an example). Although we acknowledge the critical roles of cognition, affect, metacognition, and motivation for learning with ALTs (Azevedo, Taub, Mudrick, Farnsworth, & Martin, 2016; Efklides, 2011; Pekrun & Linnenbrink-Garcia, 2014; Schunk & Greene, in press; Winne & Azevedo, 2014), our article focuses exclusively on the coupling between metacognition and emotions during STEM learning of 21st-century skills with ALTs.

Figure 1. A screenshot of MetaTutor, an ALT designed to foster effective self-regulated learning about the human circulatory system

__

Metacognition and emotions are key for monitoring and regulating learning of STEM content while using ALTs (e.g., intelligent tutoring systems, adaptive hypermedia, simulations, serious games). These processes unfold in real time as learners dynamically monitor and attempt to regulate them to enhance their learning of complex STEM materials. Although the literature is inundated with frameworks, models, and theories that focus on cognition, emotions, and cognition with emotions, there is no comprehensive model that binds emotions with metacognition (e.g., D’Mello & Graesser, 2012; Boekaerts & Pekrun, 2015; Efklides, 2011; Winne & Azevedo, 2014). As such, we argue that the coupling of metacognition and emotions is needed to fully comprehend learners’ self-regulated learning (SRL), including when they use ALTs for STEM learning. For example, negative emotional responses (e.g., frustration) following a pedagogical agent’s (PA’s) prompts and scaffolding could be based on miscalibrated metacognitive judgments from previous learner–PA interactions and the learner’s perceived utility of the scaffolding; a learner’s prolonged negative emotional state (or transition between negative emotional states) when inspecting STEM content could be triggered by a failure to correctly judge the content’s relevance in relation to his or her current learning goal (i.e., misappropriated stimulus evaluation check); a learner could also experience prolonged frustration when attempts to use an optimal learning strategy continuously lead to suboptimal results; and so forth. These are just a few examples that highlight the need to develop a theoretical framework that binds metacognition and emotions.

As such, our article will provide a succinct review of two leading theories of metacognition and emotions (i.e.,  Scherer, 2009; Winne & Hadwin, 1998, 2008) seldom used by ALT researchers. In our opinion, these theories are critical to advancing our conceptual and theoretical bases for ALT research, and they have implications for designing metacognitive and affective-sensitive ALTs. More specifically, our chapter will focus on the following: (1) presenting a concise review of two leading models and theories of metacognition (Winne & Hadwin, 1998, 2008) and emotions (Scherer, 2009); (2) presenting and critiquing Azevedo and colleagues’ extension of the information processing theory (IPT) of SRL by articulating the assumptions as well as describing the advantages and disadvantages of including the macro-level, micro-level, and valence of metacognitive processes; and (3) proposing future directions and presenting instructional implications for the design of metacognitive and affect-sensitive ALTs for STEM.

BACKGROUND, THEORETICAL FRAMEWORK, AND RELATED WORK

SRL is critical to learning, problem solving, reasoning, and understanding complex topics and domains when using ALTs, such as ITSs, to learn, acquire, retain, and effectively use key 21st-century skills (Azevedo & Aleven, 2013; Winne & Azevedo, 2014). However, several major issues continue to impede the ability to learn these key skills and excel academically when using ALTs to improve the educational and training needs of learners of all ages (Azevedo, 2015). First, learners are not being adequately prepared to meet the needs of 21st-century STEM jobs (NRC, 2012). Deficiencies include critical thinking, communication, and collaboration skills. Second, success in STEM careers requires the deployment of CAMM regulatory processes across complex STEM topics, which students lack training in and frequently fail to deploy effectively. Third, the use of ALTs to enhance students’ learning and problem-solving skills in STEM fields, for the most part, has not been guided by theories of how people learn (Azevedo, 2014, 2015; Corno & Anderman, 2015; Mayer, 2014; Van Merriënboer & de Bruin, 2014).

Research suggests that students rely heavily on the use of ineffective strategies (e.g., rote memorization, copying information verbatim) instead of effective strategies, such as summarizing content in their own words, hypothesizing, and making inferences (Azevedo et al., 2013). In addition, they fail to accurately monitor their use of learning strategies, their emerging understanding of the content, and the adequacy and relevance of multiple representations of information afforded by the learning environment. They also fail to relate the informational sources with prior knowledge. Motivationally, they show little interest in science, do not value the task, misattribute their failure to external factors, and lack self-efficacy in their ability to use effective strategies (Boekaerts & Pekrun, 2015).¹ Emotionally, learners frequently show prolonged frustration, boredom, and confusion, which they are ineffective at regulating and resolving to reengage in learning (Calvo & D’Mello, 2015; D’Mello & Graesser, 2012; Pekrun & Linnenbrink-Garcia, 2014; Sabourin & Lester, 2014). Finally, despite results on the relevance and importance of CAMM processes, there is a strong need for systematic research focusing on all four CAMM processes simultaneously. For example, recent efforts have focused on cognition and metacognition (Aleven, 2013; Azevedo, Taub, & Mudrick, 2015; Greene & Azevedo, 2009; Winne & Azevedo, 2014), cognition and emotions (D’Mello & Graesser, 2012), affect and engagement (Sabourin & Lester, 2014), and motivation and emotions (Del Soldato & du Boulay, 2016), but none have explicitly addressed the conceptual and theoretical link between metacognition and emotions (cf. Efklides, 2011).

We argue that this is a major flaw that needs to be addressed by interdisciplinary researchers working in the area of ALTs. More specifically, we argue there is a need for the following: (1) conceptually and theoretically link assumptions, levels of description, and explanations between metacognition and emotions given their respective fundamental assumptions about metacognitive monitoring and emotional appraisals; (2) provide an initial process-oriented model/framework between metacognitive processes (including judgments) and appraisal mechanisms (e.g., stimulus-evaluation checks) and their respective control processes (including cognitive processes and emotion regulation strategies); (3) employ trace methodologies to detect, track, and model temporally unfolding metacognitive and appraisal mechanisms during learning with ALTs; and (4) make inferences about the processes in (3) to understand fundamental mechanisms and then embody them in ALTs to provide real-time individualized metacognitive and emotional support to learners.

ASSUMPTIONS OF THE INFORMATION PROCESSING THEORY OF SELF-REGULATED LEARNING: AN OVERVIEW

Winne and Hadwin’s (1998, 2008) IPT model of SRL is the leading process-oriented model currently used by interdisciplinary researchers studying SRL-related issues and designing ALTs (Azevedo, 2015; Harley, Bouchet, Hussain, & Calvo, 2015; Molenaar & Järvelä, 2014; Taub, Azevedo, Bouchet, & Khosravifar, 2014; Winne & Azevedo, 2014). The IPT model of SRL (Winne & Hadwin, 1998, 2008) has several theoretical assumptions regarding the use of SRL processes during each phase of learning. First, there are four distinct phases: task definition, goal setting, studying tactics, and adaptation. However, these phases might not be mutually exclusive, such that students can monitor and adapt their current study tactics (e.g., reading content) to ones more effective for learning, cutting across multiple phases (e.g., goal setting and adaptation). For example, if students are reading a page and determine that it is not relevant to their current subgoal, the students should change the page and thus adapt their plans and studying tactics. This implies that the students are engaging in phases two, three, and four all at the same time, not independently.

However, this model assumes that students are aware of these different stages and processes involved in information processing prior to beginning the task. In fact, the model postulates that monitoring and control are the hubs of SRL. Therefore, students should already know that they should set subgoals, plan to use the appropriate and most effective cognitive and metacognitive SRL strategies, and know how to adapt the use of these strategies as well. The IPT model posits that throughout each stage of learning, students are constantly monitoring and controlling how their learning is unfolding such that they are in control of the learning processes they are using, and are monitoring how effective these processes are in contributing to learning, information processing, and, thus, task completion. Furthermore, the model suggests the dynamic nature of SRL behaviors, yet does not outline predictions regarding the temporal unfolding of and transitions between these processes. Although the IPT model outlines general SRL behaviors, it does not identify specific SRL processes that contribute to greater understanding and effective SRL. Additionally, the IPT model was originally conceptualized to examine studying behaviors; however it should be noted that Winne and Hadwin’s (1998, 2008) model has been extensively used by several research groups to provide the theoretical bases for analyzing SRL behaviors during learning with various ALTs across topics and domains related to 21st-century STEM content (e.g., Azevedo, 2015).

MACRO- AND MICRO-LEVEL INFORMATION PROCESSING THEORY PROCESSES

Greene and Azevedo (2009) developed a model of SRL by expanding on Winne and Hadwin’s (1998, 2008) IPT model and Pintrich’s (2000) four-phase SRL model. Greene and Azevedo’s model expands on these original frameworks by proposing specific micro-level valenced SRL processes within five macro-level processes. The five macro-level processes included in their model are listed as planning (e.g., activating prior knowledge, setting subgoals), monitoring (e.g., feelings of knowing, content evaluation, self-questioning, judgments of learning, monitoring progress toward goals), strategy use (e.g., drawing, coordinating informational sources, knowledge elaboration, maintenance rehearsal, hypothesizing, making inferences), handling of task difficulty and demands (e.g., help-seeking behavior), and interest activities (e.g., interest in content domain; Greene & Azevedo, 2009). Additionally, 35 specific micro-level SRL processes discovered through extensive research using concurrent think-aloud methods have been categorized within these macro-level SRL processes. Next we describe three specific metacognitive monitoring processes that are particularly relevant to different types of learning 21st-century skills with ALTs.

Three distinct yet critical metacognitive monitoring processes are feeling of knowing (FOK), judgment of learning (JOL), and content evaluation (CE). For example, FOK is a micro-level SRL process within the macro-level domain of monitoring (Azevedo, 2014; Azevedo, Taub, & Mudrick, in press). This micro-level process consists of students’ realization that they have preexisting familiarity with the content presented within the learning environment. This typically involves the awareness that the material currently being read or inspected has previously been learned. Thus, this monitoring process involves activation of prior knowledge stored in long-term memory and potential activation of this knowledge during learning with the ALT. Another metacognitive judgment, JOL, occurs when learners question whether they do (+) or do not (–) understand or comprehend something that has just been read or inspected. JOLs are similar to FOKs in that both processes monitor correspondence between a learner’s domain knowledge (i.e., cognitive condition) and the learning resources (i.e., task condition). However, JOLs allow students to monitor their emerging understanding of the content, whereas FOKs do not. Another example is CE, whereby a learner assesses the relevancy (+) or irrelevancy (-) of the material to their current subgoal. Therefore, Azevedo, Greene, and colleagues (Greene & Azevedo, 2009) focus on the macro-level, micro-level, and valence of metacognitive processes that extend and address some of the challenges found in Winne and Hadwin’s model.

The integration of Winne and Hadwin’s IPT model with Azevedo, Greene, and colleagues’ model, based on the hierarchical nature of the macro-level, micro-level, and valence of metacognitive judgments, offers several conceptual, theoretical, methodological, analytical, and design issues. Adding micro-level processes and valence allows for detailed feedback mechanisms to describe the temporally unfolding metacognitive processes (in real time) and the associated adaptive control strategies, based on the valence. For example, while monitoring the relevance of diagrams to one’s goals, the learner judges they are not relevant (i.e., CE–) and subsequently switches to the table of contents searching for potentially relevant representations of information available in the ALT. However, there is a possibility that the learner might have persisted inspecting the diagrams despite their lack of relevance to the current subgoal. In the next section, we present Scherer’s component process model (2009) by describing its assumptions and then presenting the strengths and weaknesses of the model.

SCHERER’S COMPONENT PROCESS MODEL

Scherer’s component process model (CPM; 2009) is a contemporary appraisal theory of emotions based on the conceptualization of emotions as processes rather than states. In its current form, the CPM is one of the most comprehensive models that aim to describe emotions as dynamic processes that unfold over time, including interactions among multiple components and processing on several levels (Sander, Grandjean, & Scherer, 2005; Scherer, 2001). According to the CPM, emotions arise from different appraisals of a relevant event. The appraisal sequence includes the following four steps: (1) relevance, (2) implications, (3) coping, and (4) normative significance. These appraisal steps consist of discrete sets of stimulus evaluation checks (SECs) that define different appraisal outcomes. For example, evaluation of the relevance of an event includes assessments of its novelty, intrinsic pleasantness, and goal conduciveness. The appraisals are postulated to occur in a fixed order, but Scherer (2009) highlighted that the emotional process is recursive by nature, which means that some appraisal outcomes might be revisited and updated during an emotional episode and that some of those processes even occur simultaneously. Furthermore, the appraisal process both influences and is influenced by other components of the emotional process, including the subjective feeling component, motor expressions, action tendencies, and physiological changes, as well as other psychological constructs and processes such as attention, memory, or motivation. These interactions not only shape the ongoing emotional process but also can feed forward into future emotional processes. Ultimately, emotions then lead to motivational changes and prepare action readiness and action tendencies. However, they are not solely sufficient causes for the execution of actions; rather, the execution of actions is determined depending on other factors (e.g., volition or cognitive control; Scherer, 2009).

The core assumption of the CPM is that the emotion a person experiences is critically defined by the appraisal of an event. The appraisal is a component of an emotional episode that is caused by a significant event. The emotion-eliciting event can be either internal (e.g., remembering an unpleasant interaction with a PA) or external (e.g., receiving feedback from a PA during learning with an ALT). Furthermore, the appraisal of an event is completely subjective and can therefore be inaccurate in regard to the objective situation (Scherer, 2009). See Figure 2 for examples of potential emotions that students might experience during learning with an ALT.

Figure 2. Example of emotions typically experienced during learning with ALTs

The appraisal criteria described above can be processed on different levels. Processing can range from effortless unconscious low-level neural mechanisms (e.g., pattern matching for attention/perception) to complex conscious considerations involving calculations in prefrontal cortical areas that can include propositional knowledge and cultural meaning systems (Scherer, 2009; for a detailed description of the levels of processing, see Sander et al., 2005). The level of processing can also vary throughout the appraisal process, with early appraisals (e.g., relevance check) often occurring on a low unconscious level and later appraisals (e.g., normative significance) requiring high conscious levels of processing.

Even though emotional episodes are defined by the interaction of several components, the CPM postulates that the nature of an emotion is exclusively characterized by the pattern of appraisal outcomes and their development over time (see Scherer, 2009). This implies that the CPM does not rely on a limited set of emotions. Instead, this model leaves room for an infinite number of different emotional episodes without any categorical limitations (for a different perspective, see Gross, 2015). However, the CPM can also offer predictions regarding distinct emotions based on appraisal outcomes that occur frequently.

CONCLUSIONS, FUTURE DIRECTIONS, AND IMPLICATIONS FOR THE DESIGN OF ADVANCED LEARNING TECHNOLOGIES

This article has afforded us the opportunity to briefly describe two of the most comprehensive models of metacognition and emotions found in the literature. We have specifically chosen to describe and present the assumptions of each one to familiarize the ALT community with their comprehensiveness. We hope to start a discussion about the relative importance and contributions of these models when considering their role in learners’ ability to acquire, retain, internalize, and deploy them as part of 21st-century skills. Because of space limitations, we raise several key conceptual, theoretical, methodological, and design issues that will serve as a major source of discussion for various interdisciplinary communities. In this section, we focus on the following four specific issues: conceptual/theoretical issues, analytical issues, design of external regulating agents, and their interventions in supporting metacognitive and emotional self-regulatory processes.

Some major conceptual and theoretical issues stem from the overlapping nature of assumptions, constructs, mechanisms, feedback mechanisms, and predictions in Winne and Hadwin’s IPT (1998, 2008) model, with the addition of Azevedo, Greene, and colleagues’ (2009) macro-level, micro-level, and valence approach to cognitive and metacognitive processes, as well as Scherer’s (2009) CPM theory. First, several issues arise when comparing these models and theories. For example, are metacognitive judgments similar to emotional appraisals? Is this an issue of terminology? They both seem to rely on evaluation of an event that triggers a response that leads to a metacognitive judgment or a multicomponential response that subsequently necessitates the use of a control strategy (e.g., making an inference) or an emotion regulation strategy (e.g., cognitive reappraisal; see Gross, 2015, for a detailed explanation).

Second, although both theories provide a comprehensive set of constructs, mechanisms, and so forth, it is important to consider amalgamating them in order to include a comprehensive SRL model that cohesively integrates metacognition and emotions because they are integral components required to explain complex learning. Other open issues to be discussed by researchers, educators, trainers, and system designers will include the following: (1) How does context (including type of ALT and instructional material within the ALT), as well as internal (e.g., individual differences, SRL knowledge and skills) and external (e.g., provision of adaptive scaffolding and feedback, instructional resources) conditions, impact the quality and quantity of metacognitive judgments, emotional appraisals, and subsequent self-regulatory behaviors? Do emotional appraisals and metacognitive judgments rely on the same internal and external conditions? (2) Do internal standards for monitoring cognition differ from the ones for SECs? If so, how? Do they differ by quality, quantity, complexity, and so forth? How are they used (for monitoring), revised, updated, and so on, prior to, during, and following learning? (3) What factors influence the sequence of processes and mechanisms specified in both models and theories? What does the effective regulation of both metacognitive and emotional processes look like, both behaviorally and physiologically? To what extent do emotions influence metacognitive processes and vice versa? Are there instances in which emotional appraisals and metacognitive judgments directly conflict? (4) Both the IPT and CPM only make assumptions about the individual. As such, how can they be extended to account for multiple agent (e.g., human–human, human–artificial agent) interactions during learning in complex contexts involving some type of computerized system(s) (e.g., high-fidelity mannequins, human–robot interactions, tangible landscapes)? (5) Do feedback mechanisms align during monitoring and regulation of cognitive and emotional processes? Or are they parallel systems that “communicate” with one or several SECs? Or are they managed and coordinated by a complex conflict resolution mechanism? Do they influence each other? If so, to what extent? These are a few of the questions that need to be addressed by interdisciplinary researchers from the cognitive, learning, affective, social, and computational sciences.

Several analytical issues can be raised, assuming researchers move toward using multimodal, multichannel data (see Azevedo, 2015; Azevedo et al., in press). For example, the following questions can be researched: (1) Can metacognitive and emotions data be analyzed for distinct metacognitive and emotional signatures within channels (e.g., is the gaze pattern of frustration different from the gaze pattern of confusion? Does the gaze pattern precede, occur concurrently [or with some short latency] with, or follow the gaze behavior patterns?), and across data channels (e.g., is the electrodermal activity boredom pattern also observed when the boredom gaze pattern is detected? Are there short corresponding verbal or human–computer interactions that accompany the gaze behavior patterns? What does it mean if they do not match?)? (2) Can metacognitive and emotions data be analyzed to assess which pattern(s), both within and across channels, is/are most reliable and predictive of metacognitive and emotional (and emotions predicting metacognitive) processes and performance meas­ures? (3) Can metacognitive and emotions data be analyzed for indications of students’ ability to adaptively monitor and regulate their metacognitive and emotional processes and other external regulating agents (e.g., PA, intelligent virtual human, peer, teacher, trainer)? (4) Is a longer time frame required to measure emotions than metacognitive processes (or the opposite)? Are the actual lengths of these processes the same? Typically, emotions are short lived, so we can also have multiple emotions during one instance of a metacognitive process (within the given timespan). (5) Which process takes precedence during regulation? Are emotions typically regulated before metacognitive processes and vice versa? (6) Can metacognitive and emotions data be analyzed to assess the interactions and temporal sequences among SRL processes across different contexts and phases of problem solving, conceptual understanding, comprehension, and so forth? These data can be analyzed using both traditional statistical methods (e.g., multilevel modeling) as well as data mining and machine learning techniques, including hidden Markov model analysis and hierarchical and differential sequence mining algorithms (Kinnebrew, Segedy, & Biswas, in press). The community should adopt principles of data fusion, feature fusion, and decision fusion proposed in several leading publications (e.g., Grafsgaard et al., 2014).

These issues have further important implications for designing ALTs with PAs or intelligent virtual humans that can detect, model, track, and foster students’ cognitive, affective, and metacognitive SRL processes during learning, problem solving, and so forth. When designing these external (monitoring and) regulating agents, we must address critical issues regarding timing, because the agent must be provided with the appropriate threshold in order to detect the student’s real-time monitoring (e.g., onset of a metacognitive judgment, and its associated micro-level classification and valence; for example, “I do not understand how this diagram is related to my current learning goal”²) and determine how to scaffold and provide helpful feedback and individualized instruction. For example, are there multichannel data that preceded this verbalization to indicate metacognitive monitoring, based on sampling of previous behaviors (e.g., prolonged fixation on the diagram, unequal time spent reading text and less time spent inspecting the corresponding diagram, gaze behavior patterns of a specific duration indicative of Winne and Hadwin’s SMART [e.g., selecting, assembling, translating], and so forth)?

A further complication that can arise is that the verbalization is somewhat different—for example, “I know that this diagram is not related to my current learning goal.” How does this verbalization (metacognitive judgment) differ from the previous example? How do these differences impact the type of external regulation needed to address this student’s specific learning needs, at a particular point in time, during learning with a particular ALT? Last, this issue can be further complicated by a verbalization such as “I do not understand how this diagram is related to my current learning goal, and I am getting very confused because I am not sure what to do next.” This utterance seems to indicate a (it is hoped) correct metacognitive judgment related to the relevancy of the representation of information (i.e., diagram) provided by the ALT as well as an appraisal of the corresponding affective state with a rather nebulous statement about whether there is a lack of cognitive strategy, emotion regulation strategy, or both. Which process would the agent provide feedback on first? Would it first focus on the emotions and then help regulate the student’s cognitive strategy? In sum, besides using natural language processing (NLP) to capture these utterances in real time to make inferences about metacognition and emotions, multichannel data would be necessary to provide contextual cues, history of prior SRL knowledge, behaviors, performance, learning, assessments, and so forth.

Last, challenges exist regarding the specific type of intervention (e.g., prompts, pumps, scaffolds, feedback) an agent can provide. The agent can intervene by prompting the student to engage in a particular cognitive, affective, and/or metacognitive process or by confirming that it has correctly identified the process the student is engaging in. We consider this a challenge because to create agents that are capable of intervening by prompting, we need to understand student behavior; and if we have trouble doing so as humans, due to the complexity and unpredictability of human behavior, how do we expect to program agents to be capable of doing it using algorithms that are not respondent to individual human behavior? Thus, instead of making the inferences based solely on data, the agents can be programmed to intervene by engaging in dialogues (through NLP) with the students to confirm if what they are detecting in the data is correct. In addition, instead of waiting for the right amount of data, the agent can intervene to obtain the ground truth about the student’s use of cognitive, affective, and/or metacognitive processes and establish a rapport with the student to gain this ground truth. For example, when a student facially expresses confusion when inspecting a diagram, the agent could intervene by saying, “I see that you are confused when reading this diagram, is this correct?” The more information the agent can obtain, the greater the likelihood it will be able to make accurate inferences regarding student behavior.

Ideally, artificial agents should have access to multichannel data and be able to understand and reason from the data while determining how to adapt their own behavior to support learners’ SRL. In conclusion, we argue that conceptual, theoretical, empirical, and analytical investigation on the coupling of metacognition and emotions is necessary to provide a comprehensive understanding of their roles in supporting 21st-century knowledge and skills related to learning, problem solving, and conceptual understanding with ALTs.

Acknowledgments

This study was supported by funding from the National Science Foundation (DRL 1431552). The authors would also like to acknowledge the contributions of Joseph Grafsgaard, Garrett Millar, and Megan Price.

Notes

1. We acknowledge the role of motivation, but it is beyond the scope of this article to cover it here.

2. This assumes that verbal data are collected and that the human is capable of producing language (e.g., coded utterance based on concurrent think-alouds) that is codable.

References

Aleven, V. (2013). Help seeking and intelligent tutoring systems: Theoretical perspectives and a step towards theoretical integration. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 311–336). Amsterdam, The Netherlands: Springer.

Azevedo, R. (2014). Issues in dealing with sequential and temporal characteristics of self- and socially-regulated learning. Metacognition and Learning, 9, 217–228.

Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical issues. Educational Psychologist, 50, 84–94.

Azevedo, R., & Aleven, V. (Eds.). (2013). International handbook of metacognition and learning technologies. Amsterdam, The Netherlands: Springer.

Azevedo, R., Harley, J., Trevors, G., Duffy, M., Feyzi-Behnagh, R., Bouchet, F., & Landis, R. (2013). Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 427–449). Amsterdam, The Netherlands: Springer.

Azevedo, R., Taub, M., & Mudrick, N. (2015). Technologies supporting self-regulated learning. In M.  Spector, C. Kim, T. Johnson, W. Savenye, D. Ifenthaler, & G. Del Rio (Eds.), The SAGE encyclopedia of educational technology (pp. 731–734). Thousand Oaks, CA: SAGE.

Azevedo, R., Taub, M., & Mudrick, N. V. (in press). Understanding and reasoning about real-time cognitive, affective, metacognitive processes to foster self-regulation with advanced learning technologies. In D. Schunk & J. A. Greene (Eds.), Handbook of self-regulation and performance (2nd ed.). New York, NY: Routledge.

Azevedo, R., Taub, M., Mudrick, N., Farnsworth, J., & Martin, S. (2016). Interdisciplinary research methods used to investigate emotions with advanced learning technologies In P. Schutz & M. Zembylas (Eds.), Methodological advances in research on emotion and education (pp. 231–244). Amsterdam, The Netherlands: Springer.

Boekaerts, M., & Pekrun, P. R. (2015). Emotion and emotion regulation in academic settings. In L. Corno & E. M. Anderman (Eds.), Handbook of educational psychology (pp. 76–90). New York, NY: Routledge.

Calvo, R., & D’Mello, S. K. (Eds.). (2015). New perspectives on affect and learning technologies. New York, NY: Springer.

Corno, L., & Anderman, E. M. (Eds.). (2015). Handbook of educational psychology (3rd ed.). New York, NY: Routledge.

Del Soldato, T., & du Boulay, B. (2016). Implementation of motivational tactics in intelligent tutoring systems: 20 years on. IJAIED, 26, 170–182.

D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22, 145–157.

D’Mello, S., & Graesser, A. (2014). Confusion and its dynamics during device comprehension with breakdown scenarios. Acta Psychologica, 151, 106–116.

Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46, 6–25.

Grafsgaard, J., Wiggins, J., Vail, A., Boyer, K., Wiebe, K., & Lester, J. (2014). The additive value of multimodal features for predicting engagement, frustration, and learning during tutoring. Proceedings of the Sixteenth ACM International Conference on Multimodal Interaction, pp. 42–49.

Greene, J. A., & Azevedo, R. (2009). A macro-level analysis of SRL processes and their relations to the acquisition of sophisticated mental models. Contemporary Educational Psychology, 34, 18–29.

Gross, J. J. (2015). Emotion regulation: Current status and future prospects. Psychological Inquiry, 26, 1–26.

Harley, J. M., Bouchet, F., Hussain, M. S., & Calvo, R. (2015). A multi-componential analysis of emotions during complex learning with an intelligent multi-agent system. Computers in Human Behavior, 48, 615–625. doi:10.1016/j.chb.2015.02.013

Kinnebrew, J., Segedy, J., & Biswas, G. (in press). Integrating model-driven and data-driven techniques for analyzing learning behaviors in open-ended learning environments. IEEE Transactions on Learning Technologies. http://doi.ieeecomputersociety.org/10.1109/TLT.2015.2513387

Mayer, R. (2014). Cognitive theory of multimedia learning. In R. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 43–71). Cambridge, England: Cambridge University Press.

Molenaar, I., & Järvelä, S. (2014). Sequential and temporal characteristics of self and socially shared regulated learning. Metacognition and Learning, 9, 75–85.

National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century (Committee on Defining Deeper Learning and 21st Century Skills, J. W. Pellegrino & M. L. Hilton, Eds.). Board on Testing and Assessment and Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academies Press.

Pekrun, R., & Linnenbrink-Garcia, L. (Eds.). (2014). International handbook of emotions in education. New York, NY: Routledge.

Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). San Diego, CA: Academic Press.

Sabourin, J., & Lester, J. (2014). Affect and engagement in game-based learning environments. IEEE Transactions on Affective Computing, 5, 45–56.

Sander, D., Grandjean, D., & Scherer, K. R. (2005). A systems approach to appraisal mechanisms in emotion. Neural Networks, 18, 317–352.

Scherer, K. R. (2001). Appraisal considered as a process of multilevel sequential checking. In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 92–120). New York, NY: Oxford University Press.

Scherer, K. R. (2009). The dynamic architecture of emotion: Evidence for the component process model. Cognition & Emotion, 23, 1307–1351.

Schunk, D., & Greene, J.A. (Eds.). (in press). Handbook of self-regulation and performance (2nd ed.). New York, NY: Routledge.

Taub, M., Azevedo, R., Bouchet, F., & Khosravifar, B. (2014). Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners’ levels of prior knowledge in hypermedia-learning environments? Computers in Human Behavior, 39, 356–367. doi:10.1016/j.chb.2014.07.018

Van Merriënboer, J., & de Bruin, A. (2014). Research paradigms and perspectives on learning. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 21–30). Amsterdam, The Netherlands: Springer.

Winne, P., & Azevedo, R. (2014). Metacognition. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (2nd ed., pp. 63–87). Cambridge, England: Cambridge University Press.

Winne, P., & Hadwin, A. (1998). Studying as self-regulated learning. In D. Hacker, J. Dunlosky, & A. Graesser (Eds.), Metacognition in educational theory and practice (pp. 227–304). Mahwah, NJ: Erlbaum.

Winne, P. H., & Hadwin, A. F. (2008). The weave of motivation and self-regulated learning. In D. H. Schunk & B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). New York, NY: Lawrence Erlbaum.