Moshi, released on September 17, 2024, by Kyutai, a French AI research institute, is an open-source, bidirectional voice dialogue foundation model designed to precisely simulate real-time human voice interaction. Similar to how humans detect sounds in the auditory cortex and formulate responses in the frontal lobe almost simultaneously during a conversation, Moshi adopts a multi-stream approach that combines listening and speaking independently yet closely, without mediating the input voice with text. Its internal backbone features a 7-billion parameter (7B) Helium large language model, which serves as the core inference engine, and is equipped with Mimi, a custom-developed neural audio codec that encodes 24kHz audio into a 12.5Hz ultra-low frequency band for efficient compression and restoration of audio information. Through the organic combination of these two core modules, the input and output are connected as parallel token streams, allowing for the smooth expression of the intricate flow of sounds that occur during a conversation without interruption. Conventional AI-based voice assistants have primarily adopted a cascade pipeline structure, which sequentially processes voice recognition (ASR), large language models (LLM), and speech synthesis (TTS). This conventional approach has limitations in real-time interaction due to the cumulative data processing delays at each stage, and it has been extremely difficult to smoothly detect and respond to interruptions, such as when a user interrupts or interjects during a conversation. Moshi completely overcomes this serial structure and proposes a complete end-to-end, multi-modal architecture that directly learns and infers audio tokens instead of text, significantly reducing the real-time response latency to approximately 160ms to 200ms. Thanks to this ultra-low latency design, it provides flexibility by allowing appropriate acknowledgments while the user is speaking, or by immediately canceling the previous utterance and switching to a new response if the user modifies the question in the middle. These innovative features can be effectively utilized to assist with sophisticated work workflows in the fields of biotechnology and healthcare. In sterile laboratory environments where biological hazardous materials are handled, researchers perform complex DNA extraction protocols while holding micropipettes in both hands or wearing goggles. In such situations, they can ask questions to a voice assistant equipped with Moshi and receive immediate real-time voice feedback on reagent concentration calculations or the next experimental procedure. In medical education, Moshi can be applied to virtual patient personas with various symptoms, allowing medical students to repeatedly practice medical interviews and initial screenings in an environment similar to a real clinical setting. By leveraging the strengths of the architecture, which can process variations in voice tremor, speed, and tone, it simulates changes in tone based on the patient's psychological state or pain level, greatly contributing to the improvement of healthcare service quality and communication training.

Moshi, an open-source interactive voice dialogue foundation model, was released on September 17, 2024, by Kyutai, a French AI research institute. It is designed to precisely simulate real-time human voice interaction. Just as humans detect sounds in the auditory cortex and formulate responses in the frontal lobe almost simultaneously during a conversation, Moshi adopts a Multi-Stream approach that combines listening and speaking independently yet closely, without mediating the input voice with text. Its internal backbone consists of a 7 billion parameter (7B) Helium model.