Wikipedia Article Sections

📝 Instructions

🧠 Language Model

ChatGPT (openai)

📄 Knowledge

Add documents or links to give your copilot a knowledge base. When asked a question, we'll search them to generate an answer with citations.

Submit Links in Bulk

Capabilities

🗣️ Text to Speech & Lipsync

Text-to-Speech Provider

Google Text-to-Speech

Voice name (Google TTS)

hi-IN-Wavenet-A (Female)

Please refer to the list of voice names here

🫦 Add Lipsync Video

🔠 Translation & Speech Recognition

🩻 Photo & Document Intelligence

⚙️ Settings

🗣️ Google Text-to-Speech Settings

Speaking rate

1.0 is the normal native speed of the speaker

Pitch

Increase/Decrease semitones from the original pitch

📄 Knowledge Base

👩‍🏫 Search Instructions

How should the LLM interpret the results from your knowledge base?

Citation Style

Plain Text / WhatsApp Numbers + Footnotes

🔗 Shorten Citation URLs

Create Synthetic Data

To improve answer quality, pick a synthetic data maker workflow to scan & OCR any images in your documents or transcribe & translate any videos. It also can synthesize a helpful FAQ. Adds ~2 minutes of one-time processing per file.

———

Advanced Settings

In general, you should not need to adjust these.

👁‍🗨 Conversation Summarization

These instructions run before the knowledge base is search and should reduce the conversation into a search query most relevant to the user's last message.

⌥ Variables

🔑 Keyword Extraction

Instructions to create a query for keyword/hybrid BM25 search. Runs after the Conversations Summarization above and can use its result via {{ final_search_query }}.

Embeddings Model

Text Embedding 3 Large (OpenAI)

Dense Embeddings Weightage

Weightage for dense vs sparse embeddings. 0 for sparse, 1 for dense, 0.5 for equal weight.
Generally speaking, dense embeddings excel at understanding the context of the query, whereas sparse vectors excel at keyword matches.

Max Citations

The maximum number of document search citations.

Max Snippet Words

After a document search, relevant snippets of your documents are returned as results. This setting adjusts the maximum number of words in each snippet. A high snippet size allows the LLM to access more information from your document results, at the cost of being verbose and potentially exhausting input tokens (which can cause a failure of the copilot to respond). Default: 300

Overlapping Snippet Lines

Your knowledge base documents are split into overlapping snippets. This settings adjusts how much those snippets overlap. In general you shouldn't need to adjust this. Default: 5

🔠 Language Model Settings

Avoid Repetition

Answer Outputs

How many answers should the copilot generate? Additional answer outputs increase the cost of each run.

Max Output Tokens

The maximum number of tokens to generate in the completion. Increase to generate longer responses.

Creativity (aka Sampling Temperature)

Higher values allow the LLM to take more risks. Try values larger than 1 for more creative applications or 0 to ensure that LLM gives the same answer when given the same user input.

🛠️ Tools

Give your copilot superpowers by giving it access to tools. Powered by Function calling.

Save JSON as PDF

Run cost = 4 credits

Breakdown: 1 (ChatGPT (openai)) + 3/run

By submitting, you agree to Gooey.AI's terms & privacy policy.

Show Raw Output

Assistant

To remove toxicity from a dataset, one approach is to perform dataset cleaning. This involves removing toxic passages from the dataset, discarding low-quality data, and de-duplicating the content [1] . By applying these steps, a cleaned and high-quality dataset can be obtained.

It's worth noting that as large language models (LLMs) generate more content, future datasets may become increasingly "contaminated" by LLM-generated contents themselves [1] . Therefore, ongoing efforts to improve dataset cleaning techniques will be crucial in maintaining the quality of datasets used for training LLMs.

If you have any other questions, feel free to ask! 😊

Large language model - Dataset cleaning https://gooey.ai/2/9lNn

User
How to remove toxicity?

💁‍♀️ Sources

1. Large language model - Dataset cleaning

2. Large language model - Reinforcement learning from human feedback (RLHF)

3. Large language model - Probabilistic tokenization

Using a modification of byte-pair encoding, in the first step, all unique characters (including blanks and punctuation marks) are treated as an initial set of n-grams (i.e. initial set of uni-grams). Successively the most frequent pair of adjacent characters is merged into a bi-gram and all instances of the pair are replaced by it. All occurrences of adjacent pairs of (previously merged) n-grams that most frequently occur together are then again merged into even lengthier n-gram repeatedly until a vocabulary of prescribed size is obtained (in case of GPT-3, the size is 50257).[6] Token vocabulary consists of integers, spanning from zero up to the size of the token vocabulary. New words can always be interpreted as combinations of the tokens and the initial-set uni-grams.[7]

A token vocabulary based on the frequencies extracted from mainly English corpora uses as few tokens as possible for an average English word. An average word in another language encoded by such an English-optimized tokenizer is however split into suboptimal amount of tokens.

tokenizer: texts -> series of numerical "tokens" may be split into:

Generated in 8.2s on

...

ℹ️ Details

Have you ever wanted to create a bot that you could talk to about anything? Ever wanted to create your own https://dara.network/RadBots or https://Farmer.CHAT? This is how.

This workflow takes a dialog LLM prompt describing your character, a collection of docs & links and optional an video clip of your bot’s face and voice settings.

We use all these to build a bot that anyone can speak to about anything and you can host directly in your own site or app, or simply connect to your Facebook, WhatsApp or Instagram page.

How It Works:

Appends the user's question to the bottom of your dialog script.
Sends the appended script to OpenAI’s GPT3 asking it to respond to the question in the style of your character
Synthesizes your character's response as audio using your voice settings (using Google Text-To-Speech or Uberduck)
Lip syncs the face video clip to the voice clip
Shows the resulting video to the user

PS. This is the workflow that we used to create RadBots - a collection of Turing-test videobots, authored by leading international writers, singers and playwrights - and really inspired us to create Gooey.AI so that every person and organization could create their own fantastic characters, in any personality of their choosing. It's also the workflow that powers https://Farmer.CHAT and was demo'd at the UN General Assembly in April 2023 as a multi-lingual WhatsApp bot for Indian, Ethiopian and Kenyan farmers.

👣 Steps

Final Search Query

References

[3 Items

{

…

}4 Items

{

…

}4 Items

{

…

}4 Items

]

Final Prompt

<|im_start|>system
You are an expert on LLMs<|im_end|>
<|im_start|>user
Search Result: [1]
Title: """Large language model - Dataset cleaning"""
Snippet: """
Removal of toxic passages from the dataset, discarding low-quality data, and de-duplication are examples of dataset cleaning.[10] Resulting, cleaned (high-quality) datasets contain up to 17 trillion words in 2022, raising from 985 million words, used in 2018 for GPT-1,[11] and 3.3 billion words, used for BERT.[12] The future data is, however, expected to be increasingly "contaminated" by LLM-generated contents themselves.[13]
"""

Search Result: [2]
Title: """Large language model - Reinforcement learning from human feedback (RLHF)"""
Snippet: """
Reinforcement learning from human feedback (RLHF) through algorithms, such as proximal policy optimization, is used to further fine-tune a model based on a dataset of human preferences.[14]
"""

Search Result: [3]
Title: """Large language model - Probabilistic tokenization"""
Snippet: """
Using a modification of byte-pair encoding, in the first step, all unique characters (including blanks and punctuation marks) are treated as an initial set of n-grams (i.e. initial set of uni-grams). Successively the most frequent pair of adjacent characters is merged into a bi-gram and all instances of the pair are replaced by it. All occurrences of adjacent pairs of (previously merged) n-grams that most frequently occur together are then again merged into even lengthier n-gram repeatedly until a vocabulary of prescribed size is obtained (in case of GPT-3, the size is 50257).[6] Token vocabulary consists of integers, spanning from zero up to the size of the token vocabulary. New words can always be interpreted as combinations of the tokens and the initial-set uni-grams.[7]

tokenizer: texts -> series of numerical "tokens" may be split into:

n-grams:	token	izer	:	texts	->	series	of	numerical	"	t	ok	ens	"
numbers as "tokens":	30001	7509	25	13399	4613	2168	286	29052	366	83	482	641	1
Probabilistic tokenization also compresses the datasets, which is the reason for using the byte pair encoding algorithm as a tokenizer. Because LLMs generally require input to be an array that is not jagged, the shorter texts must be "padded" until they match the length of the longest one. How many tokens are, on average, needed per word depends on the language of the dataset.[8][9]
"""
**********
[Guidelines]
- Never give advice to search the Internet.  
- Always use appropriate emojis to emphasize your point.
- Make sure to use only the Search Results to produce a coherent answer
- If the Search Results provide insufficient information, refuse to answer the question
- If the Search Result's Title or Snippet is unrelated or irrelevant to the question, don't respond and just say you don't know.
- Cite the Search Results using [${number}] notation in your answer.
**********
How to remove toxicity?<|im_end|>

Raw Text Response 1

Final Response 1

Generated Audio 1

How to Use This Recipe

🙋🏽‍♀️ Need more help? Join our Discord

Related Workflows

Lip Sync with TTS

Add your text prompt, pick a voice & upload a sample video to quickly create realistic lipsync videos. Discover the ease of text-to-video AI.

Search any document with GPT4

Add your PDF, Word, HTML or Text docs, train our AI on them with OpenAI embeddings & vector search and then process results with a GPT3 script. This workflow is perfect for anything NOT in ChatGPT: 250-page compliance PDFs, training manuals, your diary, etc.

AI Animation Generator

Create AI-generated Animation without relying on complex CoLab notebooks. Input your prompts + keyframes and bring your ideas to life using the animation capabilities of Gooey & Stable Diffusion's Deforum. For more help on how to use the tool visit https://www.help.gooey.ai/learn-animation

Compare AI Image Generators

Create multiple AI photos from one prompt using Stable Diffusion (1.5 -> 2.1, Open/Midjourney), DallE, and other models. Find out which AI Image generator works best for your text prompt on comparing OpenAI, Stability.AI etc.

Wikipedia Article Sections

📝 Instructions

🧠 Language Model

📄 Knowledge

Capabilities

🗣️ Text to Speech & Lipsync

Text-to-Speech Provider

Voice name (Google TTS)

🫦 Add Lipsync Video

🔠 Translation & Speech Recognition

🩻 Photo & Document Intelligence

🗣️ Google Text-to-Speech Settings

Speaking rate

Pitch

📄 Knowledge Base

👩‍🏫 Search Instructions

Citation Style

Create Synthetic Data

Advanced Settings

👁‍🗨 Conversation Summarization

⌥ Variables

🔑 Keyword Extraction

Embeddings Model

Dense Embeddings Weightage

Max Citations

Max Snippet Words

Overlapping Snippet Lines

🔠 Language Model Settings

Answer Outputs

Max Output Tokens

Creativity (aka Sampling Temperature)

🛠️ Tools

👣 Steps

How to Use This Recipe

Related Workflows

Lip Sync with TTS

Search any document with GPT4

AI Animation Generator

Compare AI Image Generators

GET STARTED

LEARN

DEVELOPERS

SOCIAL

CONNECT

EXTRAS