AI Studio provides a flexible open-source framework for integrating Unity games and applications with AI systems that enable real-time, speech-driven character dialogue and actions.This framework is a byproduct of a case study exploring a step-by-step integration of OpenAI's GPT-3.5 model into Unity 3D for the purpose of enhancing non-playing characters (NPCs) in video games and interactive applications. By leveraging GPT-3.5's natural language processing capabilities, we aimed to create NPCs capable of dynamic real-time interactions with players and each other! Our methodology involved integrating GPT-3.5, implementing speech-driven dialogues, and incorporating humanoid avatars with lip-syncing and emotive animations.To run the sample scenes in AI Studio, several third-party licenses and packages must be acquired and installed by the user. These packages are not included in the AI Studio package and until installed the sample scenes will contain objects with missing component scripts. Once the required packages are installed, these missing component script errors will be resolved. Instructions for acquiring and installing these packages are included in the README and are also documented in a set of YouTube videos.To run the “Simple” sample scenes, the following dependencies must be satisfied:An OpenAI API key.Integration for the OpenAI API in Unity. This is an open-source Unity package hosted in GitHub governed by the MIT license.A subscription key for the Microsoft Azure Speech Service.The Microsoft Cognitive Services Speech SDK for Unity package.To run the “City” sample scenes except for “Avatar Wars”, these additional packages must be installed:The Unity Cinemachine package from the Unity Registry.The UMA 2 Unity Multipurpose Avatar package from the Unity Asset Store.The CITY package from the Unity Asset Store.The SALSA Lip Sync Suite from Unity Asset Store.The SALSA Lip Sync Suite OneClick UMA DCS package from the Crazy Minnow Studio website.A package of several Mixamo animations.To run the “City” sample scene “Avatar Wars”, this additional dependency must be satisfied:The Ready Player Me Unity SDK package.Key features of AI Studio include:Core action-driven architecture implementing multiple character dialogues which feed asynchronous events into a central queue for processing by the main Unity threadBasic Scriptable Object framework for representing characters and character dialogues and providing AI model prompt engineering inputsAbstracted interfaces to key implementation boundaries including speech recognition and synthesis, natural language dialogue generation, and dialogue character animation and movementFeature extensions including streaming and non-streaming audio, continuous and discrete player voice recognition, conditional dialogue triggering, player synthetic voice option, programmatic chat injection, and basic non-player character movementExample scenes demonstrating integration with Microsoft Cognitive Services Speech SDK, OpenAI's chat completion models (e.g. GPT-3.5), Unity Multipurpose Avatars (UMA), SALSA LipSync Suite, and Ready Player Me avatarsYouTube videos documenting installation and example scenes
