Prompt Notebooks

Overview

A prompt is a set of messages and instructions sent to a language model to generate outputs. It typically includes system messages (defining the model's role), user messages (the actual request), and can contain variables (like user_query or context) that get filled in with values, either from the dataset, or by you. When you run an experiment, each prompt executes against each row of your dataset, with variables replaced by the corresponding column values. Prompts can be:

• Saved Prompts: Versioned prompts stored in Prompt Management. These are reusable across experiments and can be referenced by name and version.
• Unsaved Prompts: Ad-hoc prompts defined directly in the experiment. These are useful for quick testing without creating a saved prompt first.

You can test multiple prompts (both saved and unsaved) in a single experiment to compare their performance.

Example:

System message: "You are a helpful assistant that answers questions concisely."
User message: "Answer this question: {{question}}"

This prompt has one variable: question, which will be filled with values from your dataset

Create Notebook

1. Navigate to the Prompt Notebooks page.

   

2. Click "Create Notebook" button. It will open configuration form:

   

3. Enter a name and optional description.

4. Click on the Notebook name to take a look at Notebook details:

   

Prompt Playground

After creating new Notebook you will be redirected to Prompt Playground page where prompts can be created and saved:

Step 1: Select Provider

To add the providers go to Settings → Model Providers:

Enter API key to enable provider:

Select one of the providers on Prompt Playground page: openai, anthropic, google gemini.

Step 2: Select Model

Select one of the model version to test (e.g., chatgpt-4o-latest, gpt-3.5-turbo)

Step 3: Select Message Role

Select one of the Message Role: System, User, Assistant, Tool.

Step 4: Enter the Prompt Message

Use variables:

User message: "What is the capital of {{variable_name}}?"

To run the prompt fill in the values for variables below:

Step 5: Run Prompt

Step 6: Save Prompt

To save the prompt click on the second icon from the right:

Enter the name of the prompt:

Note: Saving a prompt with an existing name will create a new version of the prompt.

Saved prompts will appear in dropdown, as well as on Prompt Management page:

Run All Prompts

If there is more than one prompt you can click button "Run All Prompts" instead of clicking on every single one separately: