Reactors — GPU Computing Resource Management
Beta notice: Reactors is currently in beta. Some features may change.
1. Overview
1.1 What is Reactors?
Foundry Reactors is an integrated GPU computing resource management platform for AI/ML workloads. Through a web interface you can manage the entire lifecycle — from cluster creation and job execution to model serving.
Core value:
- Ready to use immediately: Create GPU clusters without building separate infrastructure
- Multi-cloud support: Integrate diverse environments including AWS, GCP, Azure, and Kubernetes
- Cost optimisation: Reduce costs with automatic Spot instance usage and Autostop
- Enterprise-grade security: Role-based access control, audit logs, and built-in security validation
1.2 Key Features
| Feature | Description |
|---|---|
| Cluster management | Create / start / stop / terminate GPU clusters, configure Autostop |
| Managed Jobs | Long-running jobs with automatic recovery and Spot instance support |
| SkyServe | AI model serving with autoscaling and load balancing |
| Distributed training | Support for PyTorch DDP, Ray, DeepSpeed, and Horovod |
| Infrastructure management | Integration with Kubernetes, SSH Node Pool, and Cloud Policy |
| Volume management | Persistent / ephemeral storage and cloud storage mounts |
2. System Architecture
2.1 Overall Composition
Foundry Reactors consists of a user interface, a job queue, workers, and connections to computing backends.
2.2 Job Processing Flow
- Request received: User submits a cluster creation request via the web interface
- Queue registration: Registered in the job queue according to priority
- Distributed lock acquired: Lock obtained to prevent concurrent job conflicts
- Job executed: Reactor Worker executes commands through the computing API
- State synchronised: Results are saved to the database and reflected in the UI in real time
2.3 Queue Architecture
| Queue | Concurrency | Purpose | Key Jobs |
|---|---|---|---|
| Control | 2 | Cluster control | launch, stop, start, down, autostop |
| Jobs | 5 | Job management | job.submit, job.cancel, cluster.exec |
| IO | 10 | Input / output | logs.collect, rsync.sync, output.collect |
3. Connection Types
Foundry Reactors provides three connection modes to support a variety of environments.
3.1 Embedded
A ready-to-use option suited for development and test environments
| Item | Detail |
|---|---|
| Infrastructure | Included in the Foundry package |
| Configuration | Automatic; no separate setup required |
| Use cases | Development environments, PoCs, small teams |
| Advantage | Fast start, simple management |
3.2 Managed
An auto-provisioned option for production environments
| Item | Detail |
|---|---|
| Infrastructure | Automatically deployed to your Kubernetes cluster |
| Configuration | Upload kubeconfig; provisioning is automatic |
| Use cases | Production environments, large teams, multi-cloud |
| Advantage | Scalability, high availability, automatic updates |
Key characteristics:
- Automatic namespace creation and isolation
- Service Account automatically issued (renewed every 30 days)
- Computing controller automatically deployed
3.3 External
A direct-connection option for leveraging existing infrastructure
| Item | Detail |
|---|---|
| Infrastructure | A computing server you already operate |
| Configuration | Manually enter an endpoint URL and token |
| Use cases | Reusing existing infrastructure, special network environments |
| Advantage | Flexible configuration, leverage existing investment |
4. Cluster Management
4.1 Cluster Lifecycle
| State | Description | Cost incurred |
|---|---|---|
| INIT | Cluster being created | Yes |
| UP | Running | Yes |
| STOPPED | Stopped (storage retained) | Partial |
| TERMINATED | Fully terminated | No |
4.2 Cluster Operations
| Operation | Description | Time required |
|---|---|---|
| Launch | Create and provision a new cluster | 5–15 min |
| Start | Restart a stopped cluster | 1–3 min |
| Stop | Stop cluster (storage retained) | 1–2 min |
| Down | Fully terminate cluster | 1–2 min |
4.3 Autostop
Automatically stops idle clusters to reduce costs:
| Preset | Idle time | Recommended use |
|---|---|---|
| Development | 30 min | Development / testing |
| Production | 2 hours | Production workloads |
| Custom | User-defined | Special requirements |
| Disabled | None | Always-on operation (not recommended) |
4.4 Spot Instances
Leverage Spot / Preemptible instances from cloud providers to reduce costs.
| Recovery strategy | Description |
|---|---|
| FAILOVER | Switch to On-Demand on interruption |
| ALWAYS_SPOT | Always maintain Spot instances |
| EAGER_NEXT_REGION | Retry Spot in another region |
5. Job Management
5.1 Managed Jobs
A job type with automatic recovery for long-running workloads.
Key characteristics:
- Automatic checkpointing and recovery
- Automatic restart on Spot instance interruption
- Real-time progress monitoring
- Detailed log streaming
| Status | Description |
|---|---|
| PENDING | Waiting to run |
| STARTING | Allocating resources |
| RUNNING | Currently executing |
| SUCCEEDED | Successfully completed |
| FAILED | Failed (check logs) |
| CANCELLED | Cancelled by user |
| RECOVERING | Recovering after failure |
5.2 Cluster Jobs
One-off jobs run directly on a cluster.
Use cases:
- Interactive development and debugging
- Running short scripts
- Environment setup and package installation
5.3 Job Pool (Batch Jobs)
A batch system for efficiently processing large numbers of similar jobs.
Characteristics:
- Job group management
- Automatic scaling
- Automatic retry for failed jobs
6. Service Deployment (SkyServe)
A managed platform for deploying AI models as services.
6.1 Key Features
| Feature | Description |
|---|---|
| Autoscaling | Automatic replica adjustment based on load |
| Load balancing | Automatic request distribution |
| Rolling updates | Zero-downtime service updates |
| Health checks | Automatic status monitoring and recovery |
6.2 Service Status
| Status | Description |
|---|---|
| INITIALIZING | Service initialising |
| READY | Ready to handle traffic |
| FAILED | Service error |
| SHUTTING_DOWN | Shutting down |
7. Distributed Training
Supports multi-node distributed training for large-scale AI model training.
7.1 Supported Frameworks
| Framework | Description | Recommended use |
|---|---|---|
| PyTorch DDP | Native distributed data parallelism | General distributed training |
| Ray Train | Auto-scaling distributed training | Large-scale experiments |
| Horovod | High-performance distributed training | Existing Horovod users |
| DeepSpeed | Microsoft optimisation library | Large language models |
7.2 GPU Allocation Strategies
| Strategy | Description | Recommended when |
|---|---|---|
| Pack | Maximise GPU utilisation per node | Cost optimisation |
| Spread | Distribute across nodes | Fault tolerance |
| Balanced | Balanced placement | General use |
7.3 Example Supported Cloud Instances
| Cloud | Instance | GPU | Recommended use |
|---|---|---|---|
| AWS | p4d.24xlarge | A100 x 8 | Large-scale training |
| AWS | g5.12xlarge | A10G x 4 | Cost-effective |
| GCP | a2-ultragpu-8g | A100-80GB x 8 | Maximum memory |
| Azure | Standard_ND96asr_v4 | A100 x 8 | InfiniBand support |
8. Infrastructure Management
8.1 Infrastructure Types
| Type | Description | Use cases |
|---|---|---|
| Kubernetes | Connect a Kubernetes cluster | Cloud / on-premises k8s |
| SSH Node Pool | SSH-based node connection | On-premises servers, VMs |
| Cloud Policy | Cloud provider policy | Direct connection to AWS, GCP, Azure |
8.2 Secret Management
Securely manage sensitive credentials.
| Secret type | Purpose |
|---|---|
| Kubeconfig | Access a Kubernetes cluster |
| SSH Private Key | SSH access to nodes |
| Cloud Credential | Access cloud APIs |
Security characteristics:
- Encrypted with AES-256-GCM
- Decrypted values are never exposed in API responses
- Version control and audit logs
8.3 Apply Workflow
Safely apply infrastructure changes.
- Validate: Check configuration validity
- Apply: Apply all changes atomically
- Verify: Confirm the result; rollback is supported
9. Security
9.1 YAML Security Validation
Security checks are performed on user-submitted YAML configurations.
Blocked items:
- Attempts to access the local file system
- Access to cloud metadata services (169.254.169.254)
- Reverse shell patterns
- References to sensitive environment variables
Validated areas:
- workdir: local paths are forbidden
- file_mounts: only allowed cloud storage
- docker: local build contexts are forbidden
- Commands: dangerous patterns are detected
9.2 Access Control
| Permission level | Allowed operations |
|---|---|
| Owner | All operations, including deleting the Connection |
| Admin | Manage clusters / jobs, change settings |
| Access | View, check logs |
9.3 Audit Logs
Detailed audit logs are recorded for all major operations.
Logged events:
- Cluster creation / start / stop / termination
- Job submission / cancellation
- Service deployment / update
- Infrastructure changes
- Secret creation / modification
9.4 Distributed Locks
A distributed lock mechanism prevents concurrent job conflicts:
- Redis-based distributed lock: Fast lock acquisition and release
- TTL management: Appropriate lock timeout per job type
- Automatic extension: Lock extension for long-running jobs
- Database fallback: Fallback locking when the cache is unavailable
10. Monitoring
10.1 Dashboard
An integrated dashboard for real-time status monitoring:
- GPU utilisation overview
- Cluster status summary
- Cost overview
- Recent job history
10.2 Queue Monitoring
Monitor job queue status and performance:
- Number of queued jobs
- Throughput trends
- Failure rate analysis
- Worker status
10.3 Notifications
Notifications are provided for key events.
| Event | In-app | |
|---|---|---|
| Cluster started | Yes | No |
| Cluster stopped | Yes | No |
| Cluster failed | Yes | Yes |
| Job completed | Yes | Yes |
| Job failed | Yes | Yes |
| Service deployed | Yes | Yes |
| Service failed | Yes | Yes |
11. Limits
| Item | Limit |
|---|---|
| Connections per organisation | Max 10 |
| Volumes per Connection | Max 100 |
| Infrastructures per Connection | Max 50 |
| Secrets per organisation | Max 100 |
| YAML file size | Max 1 MB |
| Cluster name length | Max 100 characters |
| Maximum Autostop duration | 24 hours |
12. Glossary
| Term | Description |
|---|---|
| Connection | Configuration linking Foundry to a computing backend |
| Cluster | A group of GPU instances |
| Managed Job | A long-running job with automatic recovery |
| SkyServe | The model serving service |
| Spot instance | A discounted cloud instance (subject to interruption) |
| Autostop | Automatic cluster shutdown after an idle period |
| Infrastructure | The backend infrastructure on which clusters run |
| Volume | Persistent or ephemeral storage |
| Job Queue | The job queue management system |