Comment on page
Gooey.AI's Open Source Vision
Shared AI Workflows as Public Innovation Infrastructure
by Sean Blagsvedt - Founder, Gooey.AI
How does every org become an AI organization (so they don’t get displaced by another org that does)? How can we leverage the constant advances in both open and private AI models, making them cheaply available and ready for impact evaluation in an organization’s specific use case? How do organizations discover and apply the hard-won AI lessons of their field’s peers to their own problems?
We propose that an open source AI orchestration and hosting service would greatly accelerate the deployment, iteration and testing of AI-based solutions for enterprises and development organizations. The system should expose most of the world’s prominent public and paid AI services; provide a hosting infrastructure to run open source AI models; foster an ever-growing collection of simple, reusable AI workflows; provide meta-tools such as feedback, LLM analysis and billing systems; and be both hosted as a reliable cloud service or available to run on private, enterprise or government compute systems. With such an infrastructure, one-off AI investments become new assets for the public to reuse, thereby enhancing the speed of innovation.
“If you get into this (AI) space, the most important thing is that you share what you are learning.”
“If you get into this (AI) space, the most important thing is that you share what you are learning.”
We must improve the innovation infrastructure of human beings, to survive the climate crisis and to solve virtually any problem we imagine. In letting each of us more efficiently build on the work of each other, we increase the leverage of our collective efforts. This is the theory of change we employ and how we hope to accelerate progress by improving the innovation infrastructure of all organizations - including development organizations - who wish to leverage AI.
“It’s not that AI will replace lawyers; it’s that the lawyers who use AI will replace those that don’t.” - Superlegal
“Every business will become an AI business.” -Satya Nadella
It is clear to us and many others that the productivity enhancements made possible first with software - with its feature that humans can reuse and modify prior investments at near- zero marginal cost - and now modern AI tools such as OpenAI’s ChatGPT will likely cause a transformation in how most processes in organizations function. We go further with the belief that the SuperLegal adage above will apply to almost every organization and job function; namely that those organizations and people that best leverage AI - as a super-set of all reusable collective human work and knowledge - will outperform those that do not. As Bill Gates stated in his April 2023 memo:
The development of AI is as fundamental as the creation of the microprocessor, the personal computer, the Internet, and the mobile phone. It will change the way people work, learn, travel, get health care, and communicate with each other. Entire industries will reorient around it. Businesses will distinguish themselves by how well they use it.
But how can we help people and organizations make this transition to a hyper-competitive and productive world? What tools do we need when “thinking” jobs are ones for AI prompt writers - trying to wrangle the AI to our desires, and/or API stitchers - connecting non-obvious or custom sets of data and functionality to build novel and useful new things? How do we specifically help organizations - such as development organizations - learn from each other’s investments?
“Every business will become an AI business.”
To better understand why an infrastructure like Gooey is needed, we need to understand the needs of organizations that could benefit from it. Here we present three organizations, who are all attempting to build AI chatbots.
Digital Green is an NGO that helps over 4 million smallholder farmers increase their productivity. For the last 20 years, they have sought out and filmed best practices from these farmers and then distributed their expertise through village-level screenings and more recently, 80M+ YouTube views. They have recorded over 10,000 videos and operate in 10 states in India, Ethiopia and Kenya. Like many established organizations, they’ve created an incredible repository of wisdom. Ideally, we’d like this wisdom to be available instantly to every farmer and the extension agents who help mentor farmers and convince them to take a risk on a new best practice and/or grow a more climate change resistant crop. Such a solution would ideally cost almost nothing on a per user basis, be available whenever the agent or farmer needed help, incorporate all of the latest agricultural research, science and real-time weather and soil sensor data, give fluent advice in any language, dialect or literacy level and have a robust feedback system so we could measure the quality of its advice, whether it was being followed and how much it impacted farmers.
Z is a US startup that provides home repair and HVAC services via technicians. They too have collected a large repository of institutional wisdom in the form of vetted, relevant training videos, repair decision trees and hundreds of manuals. Their aim is to make this wisdom available in multiple languages via speech interfaces via slack, so their technicians can immediately address hard problems in the field.
Noora Health works in 400 hospitals across India, Indonesia and Bangladesh and provides training for families who are just leaving the hospital after giving birth or undergoing a surgery. They built a successful BPO of nurses and doctors who provide WhatsApp based advice to family members and patients on a wide variety of topics. They’ve answered over 30,000 questions (about 200 / day) but now want to scale their services at near-zero marginal cost to 70M people.
Investing in AI to provide better & cheaper and scalable services in healthcare
Each of these organizations is investing in LLMs and AI chatbots to provide better & cheaper services for their users. This process today is difficult and expensive on several fronts.
- 1.Technology understanding: There is an ever increasing set of AI technologies that organizations could leverage to solve their problem. DigitalGreen, Z and Noora have all hence tasked their senior engineers to explore the potential solution space - including LLMs like OpenAI’s GPT, vectorDBs for storing larger knowledge bases, translation, speech recognition and text-to-speech AI models to support low-resource language users. But the AI space is vast, constantly changing and growing in capability every day as both the largest tech companies and best funded startups release new tools. It’s incredibly difficult to keep up with these innovations, let alone understand their relative trade-offs without actually building a prototype. Hence, DigitalGreen, Z and Noora all funded internal prototyping projects - often stretching into months. These prototypes often require technical AI knowledge and these developers are among the most expensive right now.
- 2.Time to test: Given the complexity and newness of these systems, prototypes that are testable with real users often take months to build.
- 3.Cost of deployment: AI chat prototypes that use just an LLM such as GPT-3.5 are fairly cheap to deploy today but also have limited functionality. For example, if orgs want to employ a fine-tuned, open source AI model that promises to offer better speech recognition in a low-resource language, these models often require the most expensive computers available today. For example, Gooey rents these machines (e.g. 80GB A100 GPUs) from Google for approximately $12,000 per computer per month. Running these efficiently then requires more specialized and very expensive Dev and MLOps engineers.
- 4.Cost to prove user value: Simply building an AI demo or prototype doesn’t prove it’s useful to help users achieve their goals. Hence, feedback, cohort and usage analysis systems must be also created and then the prototype must be deployed at reasonable scale inside apps or via integrations such as WhatsApp or Slack.
As we can see with just the chat use cases here, building viable systems with proven user value is hard and expensive and we believe a platform like Gooey can help.
Building viable AI systems with proven user value with Gooey
Together with our influences (see Appendix), we hold these principles in mind as we design the platform.
- 1.Learn from others
Most organizations don’t have or can’t afford AI researchers on their staff but they could certainly benefit from knowing which AI initiatives of their peers are working best and ideally, can apply those initiatives quickly and cheaply to their own particular domains.
- 2.Keep Abstracting
We can offer the greatest leverage to our customers by building on top of the constantly expanding foundational AI ecosystem. All of the tech biggest players - MSFT/OpenAI, Google, AWS - are competing for developers to integrate with their respective technologies while the open source is also constantly releasing new innovations. With Gooey, it should be easy for organizations to build on the best of private + open source models. Furthermore, as new innovations are made available in the market, organizations should be able to “swap” in the latest technology components and compare their relative price vs performance, without large up-front investments to experiment or deploy a potentially a game-changing model or API.
- 3.Encourage Sharing + Reuse
Libraries, the scientific peer review system, GitHub, open source and the Mosaic browser’s “View Page Source” all enhanced learning and innovation ecosystems by encouraging innovators to share their work and for viewers to understand it deeply and quickly. Hence, like GitHub, Gooey.AI workflows are public by default for others to discover and reuse them as we attempt to grow the ecosystem of creators building and sharing AI workflows. Being the website where great AI workflows are discovered increases our network value and hence, encourages more creators to join and strengthen the ecosystem.
- 4.Include Everyone - especially non-coders + non-English speakers
For organizations that work with marginal or non-English populations, they must be able to quickly run and assess the effectiveness of tools in resource-poor languages, often spoken by millions (not billions) of people whose documents do not dominate the content of the Internet. We will facilitate this by making the private and public models & APIs for low-resource languages available with numerous examples of how others use them and evaluation frameworks for organizations to easily benchmark which models perform best for their particular users’ data sets.
An open source API orchestration layer of simple, shared workflows, with unified billing to access the entire AI universe.
(Starting from the diagram’s center, then clockwise starting at 11 o'clock)
- These are the core metaphor of Gooey - small collections of LLM prompts that weave functionality together via API calls.
- Each workflow has a credit cost, used to pay for the API calls that a given workflow runs.
- Apps (including applications created by client organizations)
- Via our APIs, orgs can expose workflow functionality in their own applications, websites, etc. This allows them to deploy and/or white-label any workflow as part of their own solution.
- We support WhatsApp, Facebook, Instagram, Slack, IVR/Telephony and embeddable Web widgets today as communication platforms, with Telegram and Discord support expected in Q4 2023.
The Gooey.AI Website
- Any workflow can be immediately altered and run on the site or via an API call
- Future: Up / Down votes, better search, trending workflows, top creators, etc
Shared Workflow Services
- These services supplement the value of workflows and today include:
- Automated, comparative workflow analysis, enabling orgs to swap in new models or make changes to their workflows and immediately re-assess, compare and score outputs
- Built in feedback, translation and audio services on most communication platforms
- The ability to push structured data back to data stores of apps
- Connections to dashboards and external data stores enable orgs to understand usage
- Future: Create and edit workflows with natural language
Model Abstraction and Unified Billing
- We abstract the most popular AI models and make them “hot-swappable” depending on the particular needs of each workflow
- Each model has a per-API call fee which is deducted from the user’s credits when they run a workflow
Paid / Private APIs
- Gooey buys a key and then deducts credits from the workflow caller’s account
- New Paid APIs can be made available for experimentation and integration of private non- source partners
- Future: Organizations can use their own paid private keys (e.g. OpenAI keys)
Open source AI models
- Gooey can host and dynamically scale any open source AI model on our cluster of A100 GPUs
- Future: For organizations with their own compute resources, they’ll be able to host the Gooey AI model orchestration code on their own machines.
- Future: Developers can contribute code to host new models
Inbuilt Data providers
- Google Drive - we can connect and authenticate to any google drive link.
- AI OCR providers - often important knowledge exists in old scanned PDFs, filled with complicated tables and visuals. We’ve built in advanced OCR AI providers to transform complex documents into well parsed tables and structured sheets for inclusion in vectorDBs so the LLM can correctly reason over their data.
Profile data stores
- Especially for chat applications, apps will want to seed interactions with a user before a chat begins - e.g. if DigitalGreen knows the location of a farmer from their phone number, we can fetch this data to better inform the LLM prompt to advise the farmer with location specific information..
- Soon, we’ll enable workflow authors to easily pull data from an external source (via OpenAI Functions as described below) and insert the returned data into its LLM prompt.
- Additionally, we’ll enable workflows to push data back to Profile data stores too. E.g. If the user mentions their location in a conversation, the organization should be able to easily deploy a simple script that pushes that particular user ’s location data back to their own profile data store.
- Example Data Stores: FarmStack, Sunbird
Any Data Source (via OpenAI Functions)
- Workflows need the ability to read external data sources in real time. For example, in order to properly give advice on how a Bihar farmer should plant his crop, a weather forecast is often crucial.
- We see OpenAI’s Functions becoming an industry standard for how LLMs can selectively integrate data from any source, with Function support being the basis of ChatGPT and Bing Extensions. Hence, we plan to implement this protocol as a standardized and re-usable method to integrate external data sources.
We’ve partnered with DigitalGreen to integrate 400+ of their videos, 100+ documents and best practice FAQs to create a multi-lingual, audio-capable WhatsApp chatbot for farmers and the agriculture extension agents who are employed by the government to mentor farmers.
To solve the problem of how to make the vetted documents, URLs and videos of DigitalGreen accessible in local languages to farmer extension agents, we’ve been evolving the https://gooey.ai/copilot workflow. By building this as a re-usable recipe, it’s allowed us to leverage the advancements we added to it since April 2023 (e.g. feedback mechanisms, conversation analysis, conversational summarization to improve vector DB searches, synthetic data creation from video transcripts, etc) and extend those new features to expand Farmer.CHAT to 5 geographies. The work to expand to a new geography consists of:
- 1.Update the knowledge base documents
- 2.Gather “Golden” questions and answers that act as the dataset to measure whether changes to the workflow actually improve its output
- 3.Updating the conversational analysis prompts
- 4.Setting the language
- 5.Provisioning new WhatsApp numbers
- 6.Evaluate feedback from users and iterate the knowledge documents
This process is repeated as move to additional domains and build new copilots in entirely different fields; the process has changed from one focused on coding new features to one focused on content curation, usability testing, impact measurement and iteration with users to make a valuable service. Here’s initial feedback from the first 100 users of Farmer.CHAT.
Much of what’s been described above is available on Gooey.AI today. For example, Gooey.AI users can:
- Use the hosted instance and immediately tweak and iterate an existing workflows
- Save their LLM prompt, document and model parameters as reusable, shareable urls for others
- Call all workflows as via REST APIs, meaning organizations can integrate or whitelabel any service into their own apps.
- Leverage Search Serp (i.e. the ability to search the web), YouTube videos (and the ability to transcribe them), Google Docs as data services
- Run analytics, feedback and conversation analysis on /copilot
- Connect /copilot workflows to a WhatsApp, Slack, Facebook or Instagram
Furthermore, the market appears to be reacting positively to our hypothesis that it’s compelling to find and fork AI workflows; we’d had ~190,000 unique users since the start of 2023 and ~900,000 workflow runs. We’ve had over a dozen development organizations approach us in the last quarter to use our AI workflows: DigitalGreen, NooraHealth, IPRD Solutions, Quicksand, TheNudge, AllIn, PrecisionDevelopment, Jhatkaa, plus many private sector clients including Fandom.com, Zephyr and MyHeritage. These organizational engagements range from 10s to 100s of thousands of dollars in both consulting and API level revenue to Gooey.AI.
Several of our clients have specific open source asks:
- 1.Run the Gooey workflow business logic on their own servers, with the ability to inspect and edit the code like other open source projects.
- 2.Exclusively use open source AI models (rather sending their user data to Google, OpenAI or other private companies for processing)
- 3.Host workflows - and the open source AI models they depend on - on their own computing infrastructure. This is vital to governments as they consider scaling these services to 100s of millions.
- 4.Use their own keys for private paid API calls (while continuing to optionally use the Gooey service to provide unified billing to models and services for which they don’t have private API keys)
- 5.Contribute new code modules to host new open source hosted models or connect to paid API services
- 6.Contribute new data service connectors eg look up weather before running a script
Fulfilling these open source requests presents significant risk to Gooey.AI as an organization, given that our ability to charge for our workflows and hosting infrastructure will be affected. In particular, much of our revenue today comes from enterprises running workflows in the cloud on Gooey.AI and once we are open source, those organizations could choose to download their particular workflows, use their own direct API keys and/or run their own GPUs and pay nothing to us.
That said, we believe the ecosystem benefits of open source could be extremely worthwhile and that we can navigate the business risk, albeit with support. Building a thriving open source community would by definition imply more organizations using and contributing to the Gooey.AI codebase and should create more opportunities for our strategic consulting business and cloud hosting business.
- 1.With sufficient support, Gooey makes a public commitment to become open source.
- 2.Gooey workflows can be hosted on organizations’ own servers by downloading their workflow and our orchestration runtime.
- 3.Our analytics DB and visualization tools are open-sourced and locally hostable.
- 4.Orgs (such as governments) can host large open-source AI models on their own GPUs
- 5.Orgs can specify their own paid API keys and call Gooey’s AI abstraction cloud service if they lack any key.
- 6.Other orgs can contribute code to the codebase e.g. create a communication service adaptor to connect our copilot workflows to a Kenyan IVR solution such as AfricasTalking.com.
Just as GitHub hosts both public and private code repositories on GitHub.com, we expect the public discovery workflow experience to remain on our website. However, our core workflow runtime and AI model orchestration cluster will be open sourced.
For example, if you want to see how others in your field are using AI, and then discover and modify their workflows, the Gooey.AI website will be your goto destination. Once there, you can choose to keep your workflows hosted on Gooey.AI or choose to download the workflow’s prompts, settings and code to your own server, along with the orchestration run-time required to execute the workflow and the AI model cluster code if your servers are capable of running it.
Here’s an overview of the primary components:
Hence organizations can choose to run their workflow in 3 ways:
- Development orgs can prototype, test, measure impact and iterate faster at much lower cost
- The latest AI innovations from the ecosystem get deployed in real systems faster
- Every org gets to reuse and tweak the best performing AI interventions of others
- Greater sharing of tech and AI components such that new innovations are published as components in Gooey and then adopted across many of its ecosystem users.
- Shared measurement infrastructure
- Insights and data on how research based knowledge is being translated by AI systems into advisory that’s being adopted or not.
An organization wants to create a local-language IVR front-end for ChatGPT.
Imagine a Kenyan NGO proposes to BMGF a great potential innovation - allow non-literate, non-smartphone users to call a phone number, ask any question in their local language and receive back an audio answer, leveraging the OpenAI GPT LLM’s knowledge base via the API that powers ChatGPT.
Under the current funding system, this organization would have to build (or use private models) to create all the components. They would:
- 1.Get their own API keys to various LLMs like OpenAI
- 2.Determine which speech recognition and synthesis models worked best for their users and then host that infrastructure on their own GPUs.
- 3.Build a robust connections among LLMs, their speech recognition/synthesis components and whatever IVR system they choose
- 4.Build feedback and analysis systems to determine usage and retention patterns and do cohort analysis.
- 5.The code would likely not be structured for easy reuse by other organizations.
If this were a Gooey ecosystem funded project, the execution would significantly differ.
- 1.Faster time to validation. By re-using components already in Gooey, the NGO can perform large-scale usability testing faster to determine if a wider roll out is merited. E.g. They could modify the LLM script of http://gooey.ai/copilot, select their user's language and then connect the workflow via an API to WhatsApp or their IVR service.
- 2.Leveraged investments in components - eg if the org choose to use a new IVR provider e.g. AfricasTalking.com in Kenya, they would not connect IVR system endpoint just to their code but to the Gooey.AI/copilot AI workflow, so that any other user of the /copilot could now also connect to AfricasTalking.com as a communication provider in the future. Importantly, this implies that each BMGF or Ekstep funded tech project ideally ends up growing the collective open source codebase that all future projects can re-use.
- 4.The code components and LLM scripts are inspectable and reusable in a standardized format for other organization in the ecosystem to learn from and re-use, just as the Farmer.CHAT workflow (including its LLM scripts, WhatsApp connections, knowbase documents and speech recognition models) is publicly viewable and re-usable today.
- 5.Feedback: By using our /copilot workflow, the organization would get qualitative and 👍🏾 👎🏽 feedback user interface, storage and analysis systems with no additional development cost.
- 6.Measurement: If the org chooses, they can leverage the standardized conversation database components already available in Gooey. Doing so and then sharing this data would enable comparison of conversational usage patterns across multiple funded conversational projects.
Like Linux or other large scale open source projects, we fully expect similar needs to appear across organizations deploying GenAI solutions. If we can get more organizations building on a shared platform like Gooey, as each organization solves similar problems in code (and pushes those solutions back to a shared code base), the ecosystem’s pace of innovation will accelerate.
As an open source digital public good, the Gooey ecosystem will hold itself to the following metrics.
There’s a long history and discipline of ideas, tools and institutions that accelerated innovation - from public education as a societal investment to unlock the intellectual potential of every citizen to open source software. Germain to this project, there are several from which we borrow ideas.
Last modified 20d ago