Importing Dataset Test-Cases

PromptHub now supports mass importing of Dataset Test-Cases from a CSV file. Once imported, your Dataset can be re-used with all Prompts that share the same variables.

Note: We currently only support import files up to 20MB

Supported Formats

We recommend using CSV or TSV format for imports because it's simple and well-documented.

Importing From CSV

In order for us to import your test-cases, your CSV file must adhere to a prescribed format.

Header Row

The first row must contain column headers for any of the test-case properties such as name, description, target, as well as the names of any variables you want to import.

Reserved Columns (Optional)

These are optional columns used to pre-fill each test-case with helpful context. If excluded, the name will be auto-generated and the description and target fields will be left empty. You can always edit these after you finish importing.

Column	Details
name (string)	The name of this test-case.
description (string)	The description of this test-case.
target (string)	The target output for this test-case.

Reserved Chat Columns (Optional)

Chat batches are not yet supported in PromptHub, but will be available soon.

These are optional columns reserved for Chat projects specifically. They allow you to provide either a new singular user message, or an array of prior message history for any valid role.

Column	Details
prompt (string)	The next user message to send. The `prompt` string will be appended as a `user` role message after all existing template messages for the project, and any ephemeral message history passed via the `messages` array explained below.
messages (array)	An array of objects each containing a `role` and `content` entry to construct a message history that will be passed to the LLM. Will be appended after any template messages already defined on the chat project. Valid message roles are `system`, `user`, and `assistant`

Variable Columns

Reserved column names above cannot be used as dynamic variable keys

Any other columns defined in your CSV file will be considered variable names that should be attached to each test-case.

Header Row Example

name,description,target,type_var,target_var,tone_var,...

Value Rows

All subsequent rows in the CSV each represent a new "test-case" and the data that should be injected into it.

Since CSV files use commas as a delimiter, any column values that also include commas must be wrapped in quotation marks: "Here's my value, with commas"

Value Row Example

Test 1,My first test,A song about the ocean,poem,ocean,from the perspective of a whale

Putting the Pieces Together

Here's a full CSV file example that defines the test-case reserved columns, and defines 3 dynamic variables for each test-case. You can include as many rows (i.e. test-cases) as you like, as long as the total file size is less than 20MB.

name,description,target,type_var,subject_var,tone_var
Test 1,"Should return a song about hotdogs, from a dog's perspective.",A short song about hotdogs,song,hotdogs,as a dog
Test 2,"Short poem about the ocean from whale's perspective",A poem about the ocean,poem,ocean,as a whale
Test 3,"Haiku about being a beat cop",Haiku about being a beat cop,haiku,daily life,as a beat cop
Test 4,"Poem about being a software engineer",Poem about being software engineer,poem,daily life,as a software engineer
Test 5,"Song about being a CEO",Song about being CEO,song,running a SaaS company,as CEO of that company

Importing this CSV file would produce a Dataset that looks like this:

As we can see, "Test 4" has our 3 dynamic variables attached with the appropriate values for that row:

Here's another template with 2 variables.

Here's a video demo: