Advait Sarkar’s “Constructivist Design for Interactive Machine Learning” review

In “Constructivist Design for Interactive Machine Learning”, Advail Sarkar argues that the Interactive Machine Learning (IML) loop facilitates constructivist learning, as it maximises the interaction between end-users’ experience of the Machine Learning (ML) model and their ideas about the status of the ML model. Sarkar proposes a set of objectives and characterisation for designing IML systems by drawing parallels with constructivist learning environments (CLEs).

Sarkar sets the definition and context for IML, as an approach to enable people to build custom ML models for personal use and in different scenarios, by lowering the entry barrier to ML non-experts through the application of interaction design techniques. Sarkar classifies existing IML systems into three categories, according to the objectives or tasks they enable: a) model-building, b) analytic and c) hybrids, that exist between the first two categories.

In the model-building category of IML systems the main concerns are building a reusable model capable of robust predictions, where both user intervention and system feedback are used in the IML loop to improve the model iteratively. Examples of IML systems in this category are the canonical “Crayons” by Fails and Olsen’s, and Amershi’s “CueT”. In the analytic category the main concerns are analytic insights and data interpretation. Behrish et al. IML system is fitted into this category given it provides multiple visualisations to assess model quality, histograms, etc. “BrainCel” is fitted in the hybrid category as an IML system for applying machine learning within spreadsheets, which Sarkar et al. developed in Cambridge. It enables the creation of models from spreadsheet rows’ data for predicting empty cells values, and also provides coordinated views for assessing model quality.

In order to support the argument, Sarkar provides a brief contextualisation of constructivism in Computer Science (CS) education. Constructivism (Piaget, Papert, etc.) theorises that human knowledge results from an interaction between the person’s mental model and their experiential perceptions. In contrast, Instrucionism (Heider) is about transferring information more effectively. Sarkar gives examples of CS pedagogical systems as Constructivist Programming Environments (CPE), such as Logo, Alice, Scratch and DrawBridge. In these systems, novice programers “directly experience interaction between their ideas and experiences, which facilitates the construction of new or better mental models”.

Sarkar’s main argument is that the “interaction loop of interactive machine learning systems facilitates constructivist learning, as it maximises the interaction between the end-user’s experience of the model, and their ideas regarding the model status.” (p. 4). The core of Sarkar’s argument is in restating the objectives of IML systems as learning outcomes (i.e., critically learning about model-building, about the structure and properties of data, about algorithms, training and testing workflow, etc.)and highlighting how the IML loop can provide the users with experiential learning. Sarkar has the care to acknowledge, however, that learning outcomes are not explicit outcomes, but rather implicit and intermediate outcomes of IML systems which primarily build models and assist in data analysis.

Based on the interpretation of core texts about CPE design, Sarkar identified a set of design principles that he proposes for asking critical questions about the design of IML systems:

• Task ownership - IML systems should facilitate end-user tasks and real-world problems, as they can intrinsically motivate users and make them goal-oriented in their resolution. This principle is quite straightforward and necessary to all task-supporting systems;
• Ill-defined problem - Sarkar affirms that IML systems can facilitate the engagement with the ill-defined nature of model-building and analytics tasks. Ill-defined problems “allow aspects of the problem to be emergent and have users making defensible judgments”. IML systems should assist users with ill-defined questions to make defensible judgments, (e.g., related to model accuracy, overfit, concept acquisition and evolution, etc.), support the user to take reasonable actions in assessing and managing them (e.g., by providing effective data wrangling and labelling techniques), and enabling the negotiation of meaning and understanding (e.g., is this a reasonable accuracy/error, has the model learned the concept, etc.). Sarkar makes interaction design incumbent of providing effective metaphors and techniques for mixing objectivist requirements (e.g., workflow, algorithm and parameter configuration, labelling, etc.) with the open-ended and constructivist nature of the tasks (i.e., model-building and data analytics) to enable the user to form gradually stable notions of learning concepts.
• Perturbation - Sarkar refers to perturbation as a stimulus that “gently subverts” the user’s expectations or mental model. IML system should use these stimuli to encourage the user to address and explore errors strategically. However, he notes that designers should be aware of “the impact of errors on learners’ motivations, and the potential for the misattribution of poor instructional outcomes”. It’s not clear to the reader whether the cause for misattribution relates to shortcomings in the user’s instructional efforts to train the model, or to shortcomings of the system to enable the users to do so.

Sarkar considers that the previous three principles are already characteristic of IML systems and discusses how the following four core CLE aspects can provide new design insights to IML systems:

• Reflexivity - IML systems as constructivist environments should support critical awareness and reflection about the process of knowledge building. This includes accounting for knowledge provenance and its manipulation in the historical space. How to design interaction in a IML system for capturing, making accessible and displaying the interaction history? How use and design this to promote critical reflection?
• Collaboration - how can the design of IML systems incorporate collaborative activities that either to capture the social construction of meaning and collaborative analytics for making sense of the process?
• Task in context - how should the design of IML systems account for different types of context — social, historical, technical, professional, institutional, etc. —in the process of knowledge construction?
• Tool mediation - tools can influence and make a transformational impact on the practice and culture they emerge from. The set of assumptions that are built into IML systems — from epistemological, to ontological, to data, through to ideologic— should be made explicit and clear to the user.

Sarkar acknowledges the infancy of IML system design practices and concludes by calling out for thick descriptive evidence and characterisation of the design process of IML systems, recommending that they should be grounded in a defensible theoretical framework, informed by ethnography and activity-theory analysis, and validated iteratively through different implementations.

References:

Advait Sarkar (2016) “Constructivist Design for Interactive Machine Learning” In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘16). ACM, New York, NY, USA, 1467–1475. DOI: https://doi.org/10.1145/2851581.2892547