Configuration

Overview

Laminar has a flexible and powerful configuration system that allows options to be set via files (in TOML or Yaml format) and environment variables.

The system will look for configuration in the following places, from highest to lowest priority:

1. LAMINAR__* environment variables
2. Config file specified via the --config option
3. Any *.toml or *.yaml files in the --config-dir directory
4. laminar.toml in the current directory
5. $(user conf dir)/laminar/config (toml or yaml) — (this is ~/.config/laminar on Linux and ~/Library/Application Support/laminar on MacOS)
6. Default configuration

Config files

In TOML or YAML, nested configurations are specified as tables under the given key name, for example:

checkpoint-url = 's3://my-bucket/checkpoints'
 
[controller]
scheduler = 'node'
 
[database]
type = "postgres"

Environment variables

All configuration options can be set as environment variables as well. To convert a config name into an environment variable, the following rule are applied:

1. Start with LAMINAR__
2. Replace all dots (i.e., layers of nesting) with __ (double underscore)
3. Replace all - with _ (single underscore)
4. Uppercase all letters

Some examples:

• checkpoint-url => LAMINAR__CHECKPOINT_URL
• pipeline.compaction.enabled => LAMINAR__PIPELINE__COMPACTION__ENABLED
• api.bind-address => LAMINAR__API__BIND_ADDRESS

Reasonable type conversions will be applied for values specified as environment variable, for example numbers and booleans will be parsed into the correct type.

Options

Here we list all of the available configuration options by the key they are nested under. So for example, the option in the Pipeline section listed as source-batch-size would be specified in the config file as pipeline.source-batch-size or as a table

[pipeline]
source-batch-size = 128

Top-level options:

Pipeline

Configuration that applies to individual pipelines.

Key: pipeline

Run (pipeline clusters)

Configuration for pipeline clusters

Key: run

API

Configuration for the API service

Key: api

Controller

Configuration for the controller service

Key: controller

Admin

Configuration for the Admin service

Key: admin

Node

Configuration for the Node service

Key: node

Worker

Configuration for pipeline workers

Key: worker

Schedulers

Configuration for the various schedulers

Process Scheduler

Key: process-scheduler

Kubernetes Scheduler

Key: kubernetes-scheduler

Some values for the kubernetes scheduler are complete Kubernetes object, for example, the worker.resources object can be specified as a Kubernetes resource object.

When specifying these via environment variables they should be encoded as Yaml.

See the Kubernetes deployment docs for more details.

There are two modes for allocating resources for Kubernetes, specified as the kubernetes-scheduler.resource-mode:

• In per-slot mode, tasks are packed onto workers up to the task-slots config, and for each slot the amount of resources specified in resources is provided. This can be much more efficient for diversely-sized pipelines
• In per-pod mode, every pod has exactly task-slots slots, and exactly the resources in resources, even if it is scheduled for fewer slots. This is the behavior from before 0.11.

Database

Key: database

Logging

Key: logging