Meta unveils AI suite that makes speech translation more seamless and expressive

In August, Meta introduced its comprehensive AI translation model, SeamlessM4T, that facilitates translation in around 100 languages for text and 36 for speech. Boosting this tool with an upgraded “v2” architecture, Meta seeks to make conversational translations more spontaneous and expressive, a factor essential to genuine cross-language conversations.

The first of two newly introduced features is “SeamlessExpressive,” which as the name suggests, transfers your expressive elements into the translated speech. These elements encompass pitch, volume, emotional tone (from excitement and sadness to whispers), speaking rate, and pauses.

Given that translated speech has traditionally been somewhat robotic, this development could be a revolutionary one, impacting both everyday interactions and content creation alike. Supported languages include English, Spanish, German, French, Italian and Chinese, although Italian and Chinese are absent from the demo page at the time of writing this article.

The other feature, “SeamlessStreaming,” initiates the translation process while the speaker is still talking, thus enabling others to hear the translated version more quickly. Although there’s a brief latency of just under two seconds, it allows for rapid translation without having to wait until the speaker finishes the sentence.

Meta elucidated that the obstacle here was different sentence constructs in various languages, necessitating the development of an algorithm specifically designed to analyze incomplete audio inputs, to determine whether adequate context exists to begin producing a translated response, or if continued listening is needed.

Meta’s recent progress on its “Seamless Communication” suite appears to be substantial, potentially more so than mobile interpreter tools offered by competitors like Google and Samsung. While no specific timeline has been provided for public access to these new features, it’s possible to envision Meta incorporating them into future products like smart glasses, improving their practicality even further.