"How about writing a Sci-Fi story with AI!"
About
This is an AI-HCI project that researched human-AI interactions in fiction writing. The paper "AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing" was published in Joint Proceedings of the ACM IUI Workshops 2022.
Collaborating with 4 other scholars, I
- Designed and developed the GPT-2 based Sci-Fi story generator;
- Designed the interactions and interfaces of the generator;
- Designed and conducted experiments with 9 participants;
- Coding and analyzed transcripted data;
- Wrote the Intro, Methods, Result, Discussion, and Conclusion sections of our paper.
Research Questions
- What patterns of interactions are taken up by humans when they interact with machines in collaborative writing?
- How does the ability to select, edit, and cut out machine-generated text affect the human-machine co-writing process?
- How do humans perceive the role of the machine in the editable vs. non-editable interaction modes?
Methods
For our study, we prototyped a web-based collaborative writing tool where the user can co-write a short sci-fi story with a GPT-2-based text generation model. The tool uses a ``turn-taking'' approach where the user starts with writing the beginning of the story. The model then continues the story by generating a section that follows the user's previous one. The user and the model continue each writing a section in turns until the end of the story. The user may also edit the AI-generated section or regenerate a section when they are not satisfied with the AI-generated result.
The architecture of the prototype human-AI co-writing tool used in this study.
The web interface.
The study uses a within-subject design, where each user had to use both the "Edit Mode" (where editing the AI-generated text was enabled) and the "No Edit Mode" (where editing the AI-generated text was disabled) when writing with the tool. The order of the two conditions was random. We recruited nine participants (n=9) for our study. Participants were all graduate students who were interested in human-machine co-writing
Results - statistic
In both modes, the stories were about 600 words. And both the user and the machine write half of the story. There are just slight differences in the words written in the two modes.
Results - within-mode reactions
From the think-aloud and interview transcripts, we found that users have different expectations of machine-generated texts. We divided users into two categories according to whether they had a clear and explicit intent about what they wanted to write before the experiment.
Regardless of the differences between the Edit and the No Edit mode, users who have a clear mind tend to have higher expectations of machine-text coherence. They regenerated a lot of times and tried to find something that logically fitted into their story. Also, they would be more excited if some unexpected items that logically fitted their story were found in the generated text.
By contrast, users who did not have an explicit intent would have lower expectations of machine-text coherence. They almost did not regenerate the texts if the texts contained any useful information. And any new characters, events, and locations were good for them to continue the story.
Results - Between-Mode Reactions
Also, due to the difference in the interactions between the two modes, users have different expectations of coherence and fluency in the two modes.
Since users could not edit the texts, they tended to have higher expectations of machine-text coherence and fluency. They usually did not accept the texts even if half of the texts were good. However, users could accept texts to be just partially useful in Edit Mode, even if the texts just contain some useful words or ideas.
Since users could not edit the texts, they tended to have higher expectations of machine-text coherence and fluency. They usually did not accept the texts even if half of the texts were good. However, users could accept texts to be just partially useful in Edit Mode, even if the texts just contain some useful words or ideas.
Results - General Reactions
We also found that Users were amused when unexpected texts appeared, even if they presented random events or characters that had no relationship to what users had written.
For more detailed results and discussion, please read the paper.