YC Huang

Artificial Intelligence (AI) will change how designers design, what they design, and why they design. Recent efforts have extended design education with AI and related machine learning (ML) concepts and technologies. Often, these courses are grounded in other fields such as Computer Science, Electrical and Mechanical Engineering, and Social Sciences. However, few courses teach AI concepts and technologies in combination with creativity, aesthetics, and speculation to leverage the expertise of design. We explored this combination to allow students to creatively design AI exemplars. In a nine-week design activity, students envision and prototype AI exemplars that are based on their personal vision and aesthetic values. This paper reports on our vision, course design, educational context, and learning activities with a focus on developing and integrating different areas of design expertise with AI technology. We contribute insights into 1) what was designed: six design cases, resulting in design speculations, prototypes, and critical reflections as well as our design critiques of these AI exemplars. Then, 2) how to design with a vision-based approach to AI design education in comparison to other approaches, and 3) a futuring landscape with three horizons unpacking why to design with AI.

Best Poster Honorable Mention Award

Artificial Intelligence (AI) is increasingly integrated into our everyday practice. However, limited research has explored users’ perceptions on AI outputs, especially for AI mistakes. Unexpected outputs from AI might lead to frustration and negative emotions. Thus, this work aims to investigate how AI outputs influence people’s perceptions and experiences in their everyday practice, particularly when they are given the opportunity to correct AI mistakes. In this work, we developed an AI-powered system capable of recognizing six types of beverages based on a photo and generating a data report summarizing people’s drinking behaviors. We conducted a study with 14 participants to understand user perception, experience and trust on AI predictions. Results showed that participants had higher trust on the system when given ability to modify AI outputs. Interestingly, incorrect predictions made by AI sparked people’s curiosity, leading them to speculate on the underlying reasons behind the AI model.

Artificial intelligence (AI) has been widely adopted daily, and much academic effort has been directed to investigate the influencing factors of human acceptance and collaboration with it. Anthropomorphism can effectively foster positive evaluations of inanimate objects, and AI is no exception. The current study investigates how naming and personalizing an AI prototype’s appearance affect the human decision in a recipe ingredient recommendation system and finds that people are more likely to accept AI’s recommendation if they could name and customize its appearance than if they did not have the options.

Good explanations help people understand hateful memes and mitigate sharing. While AI-enabled automatic detection has proliferated, we argue that quality-controlled crowdsourcing can be an effective strategy to offer good explanations for hateful memes. This paper proposes a Generate-Annotate-Revise workflow to crowdsource explanations and presents the results from two user studies. Study 1 evaluated the objective quality of the explanation with three measurements: detailedness, completeness, and accuracy, and suggested that the proposed workflow generated higher quality explanations than the ones from a single-stage workflow without quality control. Study 2 used an online experiment to examine how different explanations affect users’ perception. The results from 127 participants demonstrated that people without prior cultural knowledge gained significant perceived understanding and awareness of hateful memes when presented with explanations generated by the proposed multi-stage workflow as opposed to single-stage or machine-generated explanations.

Designers have used everyday photos as stimuli to speculate alter- native scenarios about the future. Many studies have developed diverse ways of re-designing roles of an actor to stimulate scenario speculation; however, a lack of understandings of how the presence of an actor influences the scenario speculation process. Therefore, this work conducted a study that investigates how everyday photos with and without human actors affect people to speculate possible scenarios. We recruited 29 crowd workers from Amazon Mechanical Turk to generate 80 scenarios for two groups of photos (i.e., with-actor and non-actor photos). We analysed these scenarios and found that with-actor photos led people to speculate consistent scenarios around human actors and their activities. By contrast, non-actor photos allowed people to speculate the scenarios flexibly and diversely. These insights suggest a new design opportunity of using non-actor photos as potential materials to encourage people to generate diverse scenarios.

Designing future IoT ecosystems requires new approaches and perspectives to understand everyday practices. While researchers recognize the importance of understanding social aspects of everyday objects, limited studies have explored the possibilities of combining data-driven patterns with human interpretations to investigate emergent relationships among objects. This work presents Thing Constellation Visualizer (thingCV), a novel interactive tool for visualizing the social network of objects based on their co-occurrence as computed from a large collection of photos. ThingCV enables perspective-changing design explorations over the network of objects with scalable links. Two exploratory workshops were conducted to investigate how designers navigate and make sense of a network of objects through thingCV. The results of eight participants showed that designers were actively engaged in identifying interesting objects and their associated clusters of related objects. The designers projected social qualities onto the identified objects and their communities. Furthermore, the designers changed their perspectives to revisit familiar contexts and to generate new insights through the exploration process. This work contributes a novel approach to combining data-driven models with designerly interpretations of thing constellation towards More-Than Human-Centred Design of IoT ecosystems.

Voice Conversational Agents (VCAs) are increasingly becoming part of our daily life. Calling them by their names can solve more than just a problem of efficiency in interaction between users and VCAs, but also can cast them as actors capable of playing a role in a relationship. However, due to the immature state of the technology and its related services, such relationships between users and VCAs can be limited in practice. To broaden the scope in designing different relationships, this study explores VCAs depicted in sci-fi movies. Sci-fi is not purely based on fantasy, but also is a social reflection on technology, which in turn can inspire design researchers to understand and speculate the complexity of VCAs. Through community sourcing with sci-fi enthusiasts, 43 Sci-fi VCAs were deduced. A movie event was also organized to discuss the role of VCAs. Finally, this paper presents several possible design insights for designing the role of VCAs.

Online videos provide rich materials for people to learn conceptual knowledge, but novices often fail to transfer acquired knowledge into practice. To support learners to apply high-level concepts to the task at hand, this work aims to explore several approaches to enhance learning experience and task performance in the domain of creative tasks (e.g. writing or design). With insights drawn from reflection theories and models, we attempt to integrate reflection and practice into video learning process to enhance learning experience and work performance. Furthermore, we explore the design space of reflective prompting on video learning and the subsequent performance in creative tasks. This research will contributes findings to online video learning and creative tasks.

To support the acquisition of drawing skills, this research explores a learnersourcing approach to generating personalized learning points. These are annotations containing a clip of a drawing process, a description, and an explanation. This paper presents ShareSketch, a web-based drawing system that enables learners to practice drawing, review the drawing process, and share their works with others. In particular, we propose the before/after-practice reflection workflow that allows learners to generate learning points before or after each short practice. We evaluated our reflec- tion workflows with eight self-motivated drawing learners. The results showed that our reflection workflow can guide learners to generate high-level subgoal or concept labels, low-level steps, and personalized coping strategies.

In this research, we propose feedback orchestration that uses a categorical structure to orchestrate feedback for facilitating effective revision process sequentially or concurrently. Moreover, we demonstrate ReviseO, a system that implements the concept of feedback orchestration and support writers to solve global or local issues by adjusting feedback presentation.

StructFeed is a crowd-powered system that supports ESL writer to improve the unity of writing by generating structural feedback. The structural feedback consists of summary of writing issues, suggested steps, and aggregated crowd annotations of topic sentences and relevant keywords. Interesting findings is that people received StructFeed outperformed people received feedback from a single expert or a crowd worker.

This research attempts to explore whether structuring paired reviewers to generate feedback collaboratively can benefit feedback geneation process. A between-subjects experiment was conducted to explore three collaborative strategies for a feedback generation task, including individual work, sequential collaboration, and simultaneous collaboration.

CrowdButton is a situated crowdsourcing platform which is accessible to everyone and allow any people to make a small contribution by clicking one button in a short time. It is designed to collect room status data from passersby by answering a micro-QA task in a physical location.

This work presents a crowdsourcing approach to generate explanations for Internet memes. With insights drawn from the linguistic humor theories, we propose a three-stage workflow to generate template-based explanations that illustrate the incongruity between normal situations and the punchlines in jokes.

We present a gaze-based drawing assistant system. Eye gaze patterns of the observation are projected onto the canvas to provide visual aids on what to draw, which allows different interpretation and drawing styles.

Seeing Aural - An Installation Transferring the Materials You Gaze to Sounds You Hear

We present an audio installation allowing participants to create sounds by looking at different parts of surfaces or objects in various color and texture. By hacking a desktop eye tracker to track gaze points in 3D space, and deploying a webcam to pick up the hue, saturation and brightness of gazed material, we create an experimental musical instrument that can be played by eye gaze. Users have the freedom of choosing different raw materials and explore ways of seeing as a mean of self-expression to create a unique audio experience.

This installation showcases an office desk, a wonderland where a group of diligent kitties live happily and work industriously to make artifacts react to users’ eye gaze. Kitties are shy about being seen, but determined to keep everything in the wonderland moving after they hide behind.

The desk environment consists of two paper objects augmented by video projections and two devices with digital displays as well as other office stationaries. One desktop eye tracker was carefully setup in order to track gaze in the 3D physical space. Users sit in front of the desk and work as a normal office worker while objects and devices behave in different ways upon the presence and absence of eye gaze of users.

User interface design involves selecting proper UI components and their layout on each screen. We apply crowdsourcing approach for collecting UI annotation from mobile mockup image to build up a rich design gallery. Designers can explore design examples from gallery for inspirations.