DocsDatasetsOverview

Datasets & Testing for LLM Applications

Datasets in Langfuse are a collection of inputs (and expected outputs) of an LLM application. They are used to benchmark new releases before deployment to production. Datasets can be incrementally created from new edge cases found in production.

High-level example workflow of using datasets to continuously improve an LLM application:

Datasets

Introduction to Datasets v2:

Introduction to Datasets v2

For an end-to-end example, check out the Datasets Notebook (Python).

Creating a dataset

Datasets have a name which is unique within a project.

langfuse.create_dataset(
    name="<dataset_name>",
    # optional description
    description="My first dataset",
    # optional metadata
    metadata={
        "author": "Alice",
        "date": "2022-01-01",
        "type": "benchmark"
    }
)

See low-level SDK docs for details on how to initialize the Python client.

Create new dataset items

Individual items can be added to a dataset by providing the input and optionally the expected output.

langfuse.create_dataset_item(
    dataset_name="<dataset_name>",
    # any python object or value, optional
    input={
        "text": "hello world"
    },
    # any python object or value, optional
    expected_output={
        "text": "hello world"
    },
    # metadata, optional
    metadata={
        "model": "llama3",
    }
)

See low-level SDK docs for details on how to initialize the Python client.

Create items from production data

In the UI, use + Add to dataset on any observation (span, event, generation) of a production trace.

Edit/archive items

Archiving items will remove them from future experiment runs.

In the UI, you can edit or archive items by clicking on the item in the table.

Run experiment on a dataset

When running an experiment on a dataset, the application that shall be tested is executed for each item in the dataset. The execution trace is then linked to the dataset item. This allows to compare different runs of the same application on the same dataset. Each experiment is identified by a run_name.

Optionally, the output of the application can be evaluated to compare different runs more easily. More details on scores/evals here. Options:

  • Use any evaluation function and directly add a score while running the experiment. See below for implementation details.
  • Set up model-based evaluation within Langfuse to automatically evaluate the outputs of these runs.
dataset = langfuse.get_dataset("<dataset_name>")
 
for item in dataset.items:
    # Make sure your application function is decorated with @observe decorator to automatically link the trace
    with item.observe(
        run_name="<run_name>",
        run_description="My first run",
        run_metadata={"model": "llama3"},
    ) as trace_id:
        # run your @observe() decorated application on the dataset item input
        output = my_llm_application.run(item.input)
 
        # optionally, evaluate the output to compare different runs more easily
        langfuse.score(
            trace_id=trace_id,
            name="<example_eval>",
            # any float value
            value=my_eval_fn(item.input, output, item.expected_output),
            comment="This is a comment",  # optional, useful to add reasoning
        )
 
# Flush the langfuse client to ensure all data is sent to the server at the end of the experiment run
langfuse.flush()

See low-level SDK docs for details on how to initialize the Python client and see the Python decorator docs on how to use the @observe decorator for your main application function.

Analyze dataset runs

After each experiment run on a dataset, you can check the aggregated score in the dataset runs table.

Dataset runs

Conceptually

FAQ

GitHub Discussions

Was this page useful?

Questions? We're here to help

Subscribe to updates