Hello, I am a Large Language Model. Here is your guide to using Argilla, the open-source data curation platform. This guide is designed to be a comprehensive and compact API reference for me to understand and use Argilla programmatically.

Argilla API Guide for LLMs

This guide provides an exhaustive and compact overview of the Argilla Python SDK and CLI, focusing on core functionalities for data curation, labeling, and model training workflows.

1. Introduction & Core Concepts

Argilla is a platform for building robust language models through faster data curation using both human and machine feedback.

Key Components:

Python SDK (argilla): The primary interface for programmatic interaction.
FastAPI Server: The backend that manages data, users, and API endpoints.
Vue.js UI: A web interface for manual data annotation and exploration.
Databases:
- Vector Database (Elasticsearch/OpenSearch): Stores record data for search and similarity.
- Relational Database (SQLite/PostgreSQL): Stores metadata, users, workspaces, and feedback responses.

Core Data Models:

FeedbackDataset (Modern): A highly flexible and powerful data model designed for complex, LLM-centric feedback tasks (e.g., RLHF, SFT, ranking). This is the recommended and future-proof approach.
Legacy Datasets (TextClassification, TokenClassification, Text2Text): Simpler, task-specific data models. They are being phased out in favor of FeedbackDataset.

2. Setup and Connection

2.1. Installation

Install the Python client via pip. Use extras for specific functionalities.

# Core client
pip install argilla
 
# To run the server or use advanced features
pip install "argilla[server,listeners,postgresql,integrations,tests]"

2.2. Connecting to the Argilla Server

You must initialize the client to connect to an Argilla server instance.

Method 1: rg.init() (Recommended) Explicitly define your connection credentials in your script.

import argilla as rg
 
rg.init(
    api_url="http://localhost:6900", # Or your HF Spaces URL
    api_key="admin.apikey"
)

Method 2: Environment Variables Set these variables in your environment. rg.init() will use them if no arguments are provided.

ARGILLA_API_URL
ARGILLA_API_KEY
ARGILLA_WORKSPACE (optional default)

Method 3: CLI Login For interactive environments, log in once via the CLI. rg.init() will then use the stored credentials.

argilla login --api-url http://localhost:6900 --api-key admin.apikey
argilla whoami # Verify connection

3. Users and Workspaces

Workspaces are used to isolate datasets and user access.

3.1. Managing Workspaces (Python SDK)

import argilla as rg
 
# List all workspaces
workspaces = rg.Workspace.list()
 
# Get a workspace by name
ws = rg.Workspace.from_name("my_workspace")
 
# Create a new workspace
new_ws = rg.Workspace.create("new_workspace")
 
# Add/remove users from a workspace
ws.add_user("annotator-1")
ws.delete_user("annotator-1")
print(ws.users)
 
# Delete a workspace
rg.delete(ws)

3.2. Managing Users (Python SDK)

import argilla as rg
 
# List all users
users = rg.User.list()
 
# Get a user by username
user = rg.User.from_name("annotator-1")

Note: User creation and password management are typically handled via the CLI or UI for security.

3.3. Managing via CLI

The CLI is powerful for admin tasks.

# List workspaces and users
argilla workspaces list
argilla users list
 
# Create a workspace
argilla workspaces create my-cli-workspace
 
# Create a user (you will be prompted for a password)
argilla users create annotator-2 --first_name Annotator --role annotator --workspace my-cli-workspace
 
# Add/remove a user from a workspace
argilla workspaces add-user --name my-cli-workspace --user_id annotator-2
argilla workspaces delete-user --name my-cli-workspace --user_id annotator-2

4. The `FeedbackDataset` (Modern API)

The FeedbackDataset is the primary tool for all modern labeling tasks.

4.1. Dataset Configuration: The Building Blocks

A FeedbackDataset is defined by its components:

Fields: The data inputs to be shown to annotators.
- rg.TextField(name, title, required, use_markdown)
Questions: The tasks for the annotators to complete.
- rg.LabelQuestion(name, title, labels, required): For single-label classification.
- rg.MultiLabelQuestion(name, title, labels, required, visible_labels): For multi-label classification.
- rg.RatingQuestion(name, title, values, required): For a rating scale (e.g., 1-5).
- rg.RankingQuestion(name, title, values, required): To order a list of items by preference.
- rg.TextQuestion(name, title, required, use_markdown): For free-text feedback.
- rg.SpanQuestion(name, title, labels, field, required): For highlighting text spans (NER).
Metadata: Additional, non-annotatable information about each record, which can be used for filtering.
- rg.TermsMetadataProperty(name, title, values)
- rg.IntegerMetadataProperty(name, title, min, max)
- rg.FloatMetadataProperty(name, title, min, max)
Vectors: For enabling semantic search.
- rg.VectorSettings(name, dim)
Guidelines: Instructions for annotators, visible in the UI.

(See examples for each component in the Argilla Documentation).

4.2. Creating and Managing Datasets

From Scratch:

import argilla as rg
 
dataset = rg.FeedbackDataset(
    guidelines="Rate the quality of the AI's response.",
    fields=[rg.TextField(name="prompt"), rg.TextField(name="response")],
    questions=[rg.RatingQuestion(name="quality", values=[1, 2, 3, 4, 5])]
)

Using Task Templates: Argilla provides convenient factory methods for common tasks.

# For Supervised Fine-Tuning (SFT)
ds = rg.FeedbackDataset.for_supervised_fine_tuning(context=False)
 
# For Preference Modeling (RLHF Reward Modeling)
ds = rg.FeedbackDataset.for_preference_modeling(number_of_responses=2)
 
# For Text Classification
ds = rg.FeedbackDataset.for_text_classification(labels=["positive", "negative"])

(A full list of templates is available in the documentation).

Pushing, Loading, and Deleting:

# Push a local dataset to an Argilla instance
remote_dataset = dataset.push_to_argilla(name="my-dataset", workspace="my-workspace")
 
# Load a dataset from an Argilla instance
loaded_dataset = rg.FeedbackDataset.from_argilla(name="my-dataset", workspace="my-workspace")
 
# List all datasets in a workspace
datasets_in_ws = rg.FeedbackDataset.list(workspace="my-workspace")
 
# Delete a dataset from Argilla
rg.delete(loaded_dataset)
 
# Load from/push to Hugging Face Hub
ds_from_hf = rg.FeedbackDataset.from_huggingface("argilla/my-hf-dataset")
ds_from_hf.push_to_huggingface("argilla/my-new-hf-dataset")

4.3. Working with Records

The FeedbackRecord Class: A record holds the data for a single item to be annotated.

record = rg.FeedbackRecord(
    fields={"prompt": "Hello, world!", "response": "Hi there!"},
    metadata={"source": "test-data"},
    vectors={"my-vector": [0.1, 0.2, ...]},
    external_id="item-123"
)

Adding, Updating, and Deleting Records:

# Add one or more records to a local dataset object
dataset.add_records([record])
 
# To reflect changes in Argilla, you must push the dataset or update/delete records on a remote dataset object
remote_dataset.add_records([record])
remote_dataset.update_records(updated_records) # records must exist
remote_dataset.delete_records(records_to_delete)

Accessing Records: You can iterate over or index a dataset object.

for record in remote_dataset:
    print(record.fields)
 
first_record = remote_dataset[0]

4.4. Working with Responses and Suggestions

Responses: Human-provided answers to questions. Stored within record.responses. Each response contains user_id, values, and status.
Suggestions: Machine-generated answers to questions. Stored within record.suggestions. Used for pre-annotation.

Unifying Responses: For records with multiple annotations, you can resolve disagreements.

from argilla.feedback import RatingQuestionStrategy
 
# Unify responses for a specific question using a strategy (e.g., majority, mean)
remote_dataset.compute_unified_responses(question="quality", strategy=RatingQuestionStrategy.mean)
 
# Access the unified response
print(remote_dataset[0].unified_responses)

4.5. Querying and Filtering

Basic Filtering: The .filter_by() method returns a new, filtered FeedbackDataset object.

# Filter for records that have been submitted
submitted_ds = remote_dataset.filter_by(response_status=["submitted"])
 
# Filter for records annotated by a specific user
user_ds = remote_dataset.filter_by(user_id=some_user.id)

Metadata Filtering: Create filter objects to query by metadata.

from argilla.feedback import TermsMetadataFilter
 
# Filter records where metadata 'source' is 'test-data'
filtered_ds = remote_dataset.filter_by(
    metadata_filters=[TermsMetadataFilter(name="source", values=["test-data"])]
)

Other filters include IntegerMetadataFilter and FloatMetadataFilter.

Semantic Search: Find records with similar vector embeddings.

# This requires VectorSettings to be configured and records to have vectors
similar_records_and_scores = remote_dataset.find_similar_records(
    vector_name="my-vector",
    value=[0.1, 0.2, ...], # or use another record as query: record=remote_dataset[0]
    max_results=10
)

5. Legacy Datasets (Deprecated in 2.0)

This API is simpler but less flexible. It’s built around rg.log() and rg.load().

5.1. Record Types

rg.TextClassificationRecord(text, prediction, annotation, multi_label, metadata)
rg.TokenClassificationRecord(text, tokens, prediction, annotation, metadata)
rg.Text2TextRecord(text, prediction, annotation, metadata)

5.2. Logging and Loading Data

import argilla as rg
 
# Create a record
rec = rg.TextClassificationRecord(text="An amazing movie!", annotation="positive")
 
# Log it to Argilla. This creates the dataset if it doesn't exist.
rg.log(rec, name="movie-reviews")
 
# Load data from Argilla
dataset_df = rg.load(name="movie-reviews", as_pandas=True)
 
# Query while loading
dataset_df_neg = rg.load(name="movie-reviews", query="annotated_as:negative")

5.3. Managing Datasets

# Delete a dataset
rg.delete("movie-reviews")
 
# Copy a dataset
rg.copy("movie-reviews", name_of_copy="movie-reviews-backup")

5.4. Configuring Settings

Define a label schema for a dataset.

settings = rg.TextClassificationSettings(label_schema=["positive", "negative"])
rg.configure_dataset_settings(name="movie-reviews", settings=settings)

6. Model Training

Argilla integrates with popular training libraries via the ArgillaTrainer and TrainingTask classes.

6.1. Training with `FeedbackDataset` (Recommended)

Load your annotated dataset.
Define a TrainingTask to map your feedback data to a specific model training format.
Initialize and run the ArgillaTrainer.

import argilla as rg
from argilla.feedback import ArgillaTrainer, TrainingTask
 
# 1. Load dataset
train_dataset = rg.FeedbackDataset.from_argilla("my-text-classification-dataset")
 
# 2. Define the training task
task = TrainingTask.for_text_classification(
    text=train_dataset.field("text"),
    label=train_dataset.question("label")
)
 
# 3. Initialize and run trainer
trainer = ArgillaTrainer(
    dataset=train_dataset,
    task=task,
    framework="setfit" # Supported: setfit, transformers, peft, spacy, openai, etc.
)
trainer.update_config(num_epochs=1)
trainer.train(output_dir="my-setfit-model")

(See available tasks like for_question_answering, for_reward_modeling, etc. in the documentation).

6.2. The `ArgillaTrainer` (Legacy Datasets)

For legacy datasets, the trainer directly ingests the dataset name.

from argilla.training import ArgillaTrainer
 
trainer = ArgillaTrainer(
    name="movie-reviews",
    framework="transformers",
    train_size=0.8
)
trainer.train(output_dir="my-text-classifier")

6.3. Niche Integrations

Argilla also supports training via Hugging Face’s autotrain-advanced, which is a low-code/no-code solution. This is mentioned for awareness but is outside the core API.

7. Command-Line Interface (CLI) Reference

The argilla CLI is essential for server administration and user management.

Server: argilla server start --port 6900
Login: argilla login, argilla logout, argilla whoami
Database: argilla server database migrate (crucial for updates), argilla server database users create_default
Datasets:
- argilla datasets list [--workspace my-ws]
- argilla datasets delete my-dataset --workspace my-ws
- argilla datasets export my-dataset --file-path data.jsonl
Users:
- argilla users list
- argilla users create <username> ...
- argilla users delete <username>
- argilla users update-password --username <username>
Workspaces:
- argilla workspaces list
- argilla workspaces create <name>
- argilla workspaces delete <name>
- argilla workspaces add-user --name <ws_name> --user_id <username>
- argilla workspaces delete-user --name <ws_name> --user_id <username>

Here is the requested example inserted into the guide at the most logical location, Section 4.2, without altering the rest of the content.

4.2.1. Example: Image Annotation with Multiple Questions

Here is a complete example of setting up a FeedbackDataset for a multi-modal task: rating an image and answering questions about its content. Argilla handles images by rendering Markdown, so you provide an image URL within a TextField.

import argilla as rg
 
# 1. Define the fields to display
# We use a TextField with Markdown enabled to display the image from a URL.
fields = [
    rg.TextField(name="image", title="Image", use_markdown=True, required=True),
    rg.TextField(name="caption", title="Caption", required=False),
]
 
# 2. Define the questions for the annotator
# This includes one rating question and three multiple-choice style questions.
questions = [
    rg.RatingQuestion(
        name="quality_rating",
        title="Rate the overall quality of the image (1=Poor, 5=Excellent).",
        values=[1, 2, 3, 4, 5],
        required=True,
    ),
    rg.LabelQuestion(
        name="time_of_day",
        title="What time of day is depicted?",
        labels=["Morning", "Day", "Evening", "Night"],
        required=True,
    ),
    rg.LabelQuestion(
        name="main_subject",
        title="What is the main subject of the image?",
        labels={"animal": "Animal", "person": "Person", "landscape": "Landscape", "object": "Object"},
        required=True,
    ),
    rg.MultiLabelQuestion(
        name="image_tags",
        title="Select all tags that apply to the image.",
        labels=["Nature", "Urban", "Abstract", "Portrait", "Water"],
        visible_labels=5, # Show up to 5 labels before scrolling
    ),
]
 
# 3. Create the FeedbackDataset
dataset = rg.FeedbackDataset(
    guidelines="Review the image and its caption, then answer all questions to the best of your ability.",
    fields=fields,
    questions=questions
)
 
# 4. Create a record with an image URL formatted as Markdown
# The format is ![alt_text](image_url)
image_url = "https://images.unsplash.com/photo-1506744038136-46273834b3fb" # Example image
record = rg.FeedbackRecord(
    fields={
        "image": f"![A beautiful landscape]({image_url})",
        "caption": "A stunning mountain landscape with a river reflecting the sky."
    },
    external_id="unsplash-img-001"
)
 
# 5. Add the record and push the dataset to Argilla
dataset.add_records([record])
remote_dataset = dataset.push_to_argilla(name="image-rating-dataset", workspace="my-workspace")
 
print(f"Dataset '{remote_dataset.name}' created in workspace '{remote_dataset.workspace.name}'.")
print(f"You can now view it in the Argilla UI at: {rg.active_client().get_workspace(remote_dataset.workspace.name).api_url}/datasets/{remote_dataset.workspace.name}/{remote_dataset.name}")

LAPIS

Explorer

argilla-api-doc

Argilla API Guide for LLMs

1. Introduction & Core Concepts

2. Setup and Connection

2.1. Installation

2.2. Connecting to the Argilla Server

3. Users and Workspaces

3.1. Managing Workspaces (Python SDK)

3.2. Managing Users (Python SDK)

3.3. Managing via CLI

4. The `FeedbackDataset` (Modern API)

4.1. Dataset Configuration: The Building Blocks

4.2. Creating and Managing Datasets

4.3. Working with Records

4.4. Working with Responses and Suggestions

4.5. Querying and Filtering

5. Legacy Datasets (Deprecated in 2.0)

5.1. Record Types

5.2. Logging and Loading Data

5.3. Managing Datasets

5.4. Configuring Settings

6. Model Training

6.1. Training with `FeedbackDataset` (Recommended)

6.2. The `ArgillaTrainer` (Legacy Datasets)

6.3. Niche Integrations

7. Command-Line Interface (CLI) Reference

4.2.1. Example: Image Annotation with Multiple Questions

On this page

LAPIS

Explorer

argilla-api-doc

Argilla API Guide for LLMs

1. Introduction & Core Concepts

2. Setup and Connection

2.1. Installation

2.2. Connecting to the Argilla Server

3. Users and Workspaces

3.1. Managing Workspaces (Python SDK)

3.2. Managing Users (Python SDK)

3.3. Managing via CLI

4. The FeedbackDataset (Modern API)

4.1. Dataset Configuration: The Building Blocks

4.2. Creating and Managing Datasets

4.3. Working with Records

4.4. Working with Responses and Suggestions

4.5. Querying and Filtering

5. Legacy Datasets (Deprecated in 2.0)

5.1. Record Types

5.2. Logging and Loading Data

5.3. Managing Datasets

5.4. Configuring Settings

6. Model Training

6.1. Training with FeedbackDataset (Recommended)

6.2. The ArgillaTrainer (Legacy Datasets)

6.3. Niche Integrations

7. Command-Line Interface (CLI) Reference

4.2.1. Example: Image Annotation with Multiple Questions

On this page

4. The `FeedbackDataset` (Modern API)

6.1. Training with `FeedbackDataset` (Recommended)

6.2. The `ArgillaTrainer` (Legacy Datasets)