Gates Foundation

gatesfoundation

A workspace for the Gates Foundation DPI, FairFoward and Gooey teams focused on evals for low-resource languages plus the home of our Agriculture advisory work e.g. https://gooey.ai/ageval

489 Public Workflows
10 Members

Which of the latest models best understand Swahili questions (as WhatsApp audio notes) and can provide an English? GPT-realtime, GPT4o realtime, Jacaranda(ASR) + GPT5 (LLM as MT) and Jacarandra + GPT5 + Google Translate are compared.

Removed extra rows and columns

1y ago

Public

Here we compare the top speech recognition, LLM and machine translation models for Kinyarwanda. In short, unfortunately the realtime OpenAI models (GPT4o-Audio and GPT-realtime) perform badly while leveraging dedicated ASR models (MMS, Sunbird, MBaza) in conjunction with GPT5 and Google 2.5 Pro appears to work reasonably well.

1y ago

Public

This workflow is designed specifically to measure end-to-end latency for voice-based AI interactions, focusing on benchmarking system response times rather than providing full conversational answers. Incoming audio samples (in Kikuyu or other selected languages) are transcribed, processed by an AI assistant with a maximum output of 10 tokens, and then synthesized back to audio. The workflow compares two different Kikuyu audio-to-audio (A2A) translation pipelines by processing input samples from a Google Sheet and logging runtime, price, and output URLs for each. This setup enables reliable benchmarking and optimization of transcription, AI processing, and text-to-speech components, helping teams evaluate latency and cost across different ASR models for the Kikuyu language.

🦾

1y ago

Public

Here we compare the top speech recognition, LLM and machine translation models for Kinyarwanda. In short, unfortunately the realtime OpenAI models (GPT4o-Audio and GPT-realtime) perform badly while leveraging dedicated ASR models (MMS, Sunbird, MBaza) in conjunction with GPT5 and Google 2.5 Pro appears to work reasonably well.

Which of the latest models best understand Swahili questions (as WhatsApp audio notes) and can provide an English? GPT-realtime, GPT4o realtime, Jacaranda(ASR) + GPT5 (LLM as MT) and Jacarandra + GPT5 + Google Translate are compared.

1y ago

Public