System Components

Laminar consists of several core components that work together to process streaming data in real-time.

Component Overview

API Server

The API Server is the control plane for Laminar, handling all management operations.

Responsibilities

Function	Description
Connection Profiles	Manage credentials and connection settings for sources and sinks
Table Definitions	Store schemas and table configurations
Pipeline Management	Create, update, start, stop, and delete pipelines
Schema Registry	Handle schema inference and validation
Health Monitoring	Track pipeline status and health metrics

API Endpoints

The API server exposes RESTful endpoints:

POST   /api/v1/connection-profiles    # Create connection profile
GET    /api/v1/connection-profiles    # List connection profiles
POST   /api/v1/tables                 # Create table
GET    /api/v1/tables                 # List tables
POST   /api/v1/pipelines              # Create pipeline
GET    /api/v1/pipelines              # List pipelines
POST   /api/v1/pipelines/:id/start    # Start pipeline
POST   /api/v1/pipelines/:id/stop     # Stop pipeline

Stream Engine

The Stream Engine is the data plane, responsible for executing SQL transformations on streaming data.

Components

Component	Description
Coordinator	Manages pipeline lifecycle, distributes work, handles checkpointing
Workers	Execute SQL operators, process data partitions in parallel
Operators	Individual SQL operations (filter, project, aggregate, join)

Execution Model

Pipeline Compilation: SQL is parsed and compiled into a directed acyclic graph (DAG) of operators
Partition Assignment: Input partitions are distributed across workers
Data Processing: Each worker processes its assigned partitions independently
Checkpointing: Periodic snapshots ensure exactly-once processing

State Store

The State Store manages intermediate state required for stateful operations like aggregations, joins, and windowing.

State Types

Type	Storage	Use Case
In-Memory State	Memory	Active windows, aggregations, and joins
Checkpointed State	Parquet on Object Storage	Durable state for recovery

State Tables

Table Type	Description
GlobalKeyedTable	Key-value store for aggregations
ExpiringTimeKeyTable	Time-indexed store with TTL for windowed operations

Metadata Store

The Metadata Store persists all configuration and runtime state using PostgreSQL.

Stored Entities

Entity	Description
Connection Profiles	Source and sink connection configurations
Tables	Schema definitions and table metadata
Pipelines	Pipeline configurations and SQL queries
Checkpoints	Checkpoint metadata and state references
Jobs	Pipeline execution history and status

Component Interaction

This diagram shows how components interact during pipeline execution:

Startup Sequence

Client sends start request to API Server
API Server loads pipeline configuration from Metadata Store
Stream Engine initializes workers and operators
Workers connect to sources and begin consuming
Transformed data flows to configured sinks
Checkpoints are periodically written to State Store

Deployment Topology

In production deployments:

API Server: Runs as a stateless deployment with multiple replicas
Stream Engine: Runs as pods with resource requests/limits
PostgreSQL: Deployed as a StatefulSet or managed service
Object Storage: S3 or compatible storage for checkpoints