Data Forge User Guide

Overview

Data Forge v2 is a core feature of the Foundry platform — a standalone workspace that transforms raw, unrefined data into high-quality training data for LLM fine-tuning.

The unit of work is a Session (workflow), which combines reusable Connections (LLM connections) and Configs (generation settings).

Key Features

• Multiple source types: PDF, DOCX, PPTX, HTML, TXT, Markdown files and images (Multimodal), URLs, YouTube videos
• Automated data generation: QA pairs, Chain-of-Thought, Summaries, Multimodal QA
• Quality control: AI-powered automatic curation and manual review
• Multiple output formats: Alpaca, ChatML, OpenAI Fine-tuning (FT), JSONL
• Foundry Repo integration: Publish generated data directly to a Dataset Repo

Pipeline (6 Stages)

Getting Started

1. Access Data Forge

From the dashboard, click "Data Forge" in the left sidebar.

2. Set Up an LLM Connection

Before generating data you must configure an LLM connection.

1. Navigate to the Connections page
2. Click "Add Connection"
3. Choose a provider:
   • vLLM: Local vLLM server (no API key required)
   • API Endpoint: OpenAI-compatible API
     • Choose from OpenAI (GPT-4, etc.), Claude (Anthropic), OpenRouter, or Custom
4. Enter and save your API key (required for api-endpoint)
5. Click "Test Connection" to verify

Security: API keys are encrypted with AES-256-GCM before storage.

Creating a Session

1. Start a New Session

1. Click "New Session" on the dashboard
2. Enter a session name
3. Add sources, select a Connection and Config, then start processing

Up to 5 sessions can be active at the same time.

2. Choose a Content Type (Config)

Source Management

Supported File Formats

Maximum file size: 50 MB per file; maximum sources per session: 20.

Adding Sources

File Upload

1. Drag and drop or select files on the project page
2. Monitor upload progress
3. Text extraction starts automatically

Add a URL

1. Select the "Add URL" tab
2. Enter a web page URL
3. Content is crawled automatically

YouTube Video

1. Select the "YouTube" tab
2. Enter a YouTube URL
3. Subtitles / transcript are extracted automatically

Source Status

Data Generation

Generation Settings

Basic Settings

Chunking Settings

Prompt Templates

Using Default Templates

Use the optimised templates provided by the system.

Custom Templates

1. Create a new Config on the Config page
2. Write your prompt content (variables are supported)
3. Select the custom Config in your session

Running Generation

1. Navigate to the "Generate" page
2. Review settings
3. Click "Start Generation"
4. Monitor progress on the Jobs page

Curation

Automatic Curation

The AI evaluates the quality of generated data on a scale of 1–10.

Quality Criteria

• Accuracy: Alignment with the source content
• Completeness: Sufficiency of the answer
• Clarity: Clarity of the question and answer
• Usefulness: Practical value

Threshold Settings

Manual Review

1. Review generated items on the "Curate" page
2. For each item:
   • Approve: Pass curation
   • Reject: Exclude from curation
   • Edit: Modify content then approve
3. Batch operations allow processing multiple items at once

Export

Supported Formats

Export Options

Download

1. Select a format on the "Export" page
2. Click "Download"
3. The file downloads to your device

Publish to Foundry Repo

1. Select "Publish to Repo"
2. Choose an option:
   • Existing Repo: Append data to an existing repo
   • New Repo: Create a new Dataset Repo
3. Enter a filename and commit message
4. Click "Publish"

Job Management

Job Types

Job Status

Job Monitoring

1. View all jobs on the "Jobs" page
2. Real-time progress is displayed
3. Logs are available for each job
4. Running jobs can be cancelled

Limits

LLM call costs are billed directly through the Connection you register.

Reusing Configs (Generation Settings)

Generation parameters and prompts are saved as Configs and can be reused across multiple sessions.

Creating a Config

1. Create a new Config on the Configs page
2. Fill in the details:
   • Name / Description
   • Content type: QA, CoT, Summary, Multimodal QA
   • Prompt / Parameters: temperature, top_p, chunk size, num pairs, curate threshold, etc.
3. Save, then select it in a session (you can set a default Config)

Default Prompts

If left blank, the default prompt is used. Variables such as {text} are supported in prompts.

Troubleshooting

Common Issues

Source Processing Fails

• Check the file format
• Check the file size limit
• Make sure the file is not corrupted

Low Generation Quality

• Adjust Temperature (lower values produce more consistent output)
• Adjust chunk size
• Use a custom prompt

Job Timeout

• Reduce source size
• Reduce the number of pairs to generate

Support

If problems persist, please report them on GitHub Issues.