Skip to main content

Features

The following list gives an overview of the features of Smart Data Lake Builder (SDLB) 3.x. SDLB is actively developed, see the release notes for what is new, and Upgrading to SDLB 3.x if you come from SDLB 2.x.

Declarative approach, file based metadata​

  • Easy to version with a VCS for DevOps
  • Flexible structure by splitting over multiple files and subdirectories
  • Easy to generate from third party metadata (e.g. source system table catalog) to automate transformation of large number of DataObjects
  • Support to handle multiple environments, see also Configure for different environments, Hocon Variables
  • Configuration Schema Viewer with descriptions and HOCON examples for every configuration type and attribute, see also Schema Viewer Usage

see also Hocon Configurations

Support for complex workflows & streaming​

  • Fork, join, parallel execution, multiple start- & end-nodes possible
  • Execution state tracking and Recovery of failed runs
  • Switch a workflow between batch or streaming execution by using just a command line switch, see also Command Line

see also DAG, Streaming

Multi-Engine​

Choose the engine per Action through an engine connection, and combine engines in the same data pipeline:

  • Spark (DataFrames), Spark session inside the SDLB process
  • Spark Connect (DataFrames) on a remote Spark Connect server, without a Spark session in the SDLB process, see also Spark Connect Engine
  • Snowflake Snowpark (DataFrames), see also Snowpark blog post
  • SQL (ELT): transformations are rendered as one SQL statement by SQLGlot and executed in the database (Postgres, Snowflake, SQL Server, Oracle, Databricks, DuckDB, ...), see also SQL Engine
  • Plain Scala: lightweight engine without Spark, e.g. for unit tests and small pipelines
  • File (Input&OutputStream)
  • Parameter: coordinate work outside SDLB, e.g. scripts and ML training runs

see also Execution Engines

Connectivity​

  • Spark: diverse connectors (HadoopFS, DeltaLake, Iceberg, JDBC, Kafka, Snowflake, BigQuery, Webservice) and formats (CSV, JSON, XML, Avro, Parquet, Excel, Access …), see also Delta Lake format, Custom Webservice, Databricks blog post
  • Change data capture from databases with Debezium, historized without further configuration
  • File: SFTP, Local, Webservice
  • Support for getting secrets from different secret providers, see also Hocon Secrets
  • Support for SQL update & merge (Jdbc, DeltaLake, Iceberg)
  • Database views and materialized views (JdbcViewDataObject), keeping grants and replaced only if their query changed, see also SQL Engine
  • Support for integration of Airbyte sources, see also Airbyte blog post
  • Easy to extend by implementing predefined Scala traits, see Extending SDLB

see also Data Objects

Generic Actions​

see also Actions

Customizable Transformations​

  • DataFrame Transformations:
    • Chain predefined standard transformations (e.g. filter, row level data validation and more) and custom transformations within the same action, see also Own transformer
    • Custom Transformation Languages: SQL, Scala (Class, compile from config, notebook), Python, see also Use in Notebooks
    • Flexible transform method signatures: typed Datasets, options and primitive parameters are mapped automatically
    • SQL transformations can be written in Spark SQL and executed on another database by the SQL engine
    • Many input DataFrames to many outputs DataFrames (but only one output recommended normally, in order to define dependencies as detailed as possible for the lineage)
    • Use runtime information of previous Actions (metrics, partition values, state) in options and expressions
    • Add metadata to each transformation to explain your data pipeline.
  • File Transformations:
    • Language: Scala (Class or compile from config)
    • One to one, or one input file to many output files, with mapping of partition values

see also Transformations

Early Validation​

Execution in 3 phases before execution

  • Load Config: validate configuration, see also Config validation
  • Prepare: validate connections
  • Init: validate DataFrame lineage of all engines (missing columns in transformations of later actions will stop the execution)

Errors in constraint and expectation definitions name the failing definition.

see also Execution Phases

Execution Modes​

Select data to process, e.g.

  • Process all data
  • Partition parameters: give partition values to process for start nodes as parameter, see also Partitions
  • Partition Diff: search missing partitions and use as parameter
  • Incremental: use stateful input DataObject, or compare sortable column between source and target and load the difference, see also Incremental Mode
  • File incremental move: process new files and archive or delete them afterwards
  • Column filters, optionally propagated to the following Actions
  • Spark Streaming: asynchronous incremental processing by using Spark Structured Streaming, see also Streaming
  • Spark Streaming Once: synchronous incremental processing by using Spark Structured Streaming with Trigger=Once mode
  • Implement your own execution mode, see also Own ExecutionMode

see also Execution Modes

Schema Management​

  • Automatic evolution of data schemas (new column, removed column, changed datatype widened without loss of data), see also Schema Evolution
  • Support for changes in complex datatypes (e.g. new column in array of struct)
  • Automatic adaption of DataObjects with fixed schema (Jdbc, DeltaLake, Iceberg, SQL engine)
  • Manage tables at deploy time with CatalogSchemaUpdater: create missing tables, evolve schemas, apply comments, primary and foreign keys, deploy views - with a plan mode reporting the changes first, see also Managing tables in the catalog at deploy time
  • Export schemas of all DataObjects with a dry-run, to develop without access to data, see also Dry run

see also Schema

Metrics​

  • Number of rows read/written per DataObject
  • Execution duration per Action
  • Arbitrary custom metrics defined by aggregation expressions, see also Expectations
  • Predefined metric for transfer rate, completness and ensuring unique constraints.
  • Result of every expectation (ok/warn/error) in the metrics log
  • Run summary including the Actions of previous attempts, see also Run State & Recovery
  • StateListener interface to get notified about progress & metrics

see also Metrics, metricsFailCondition

Data Catalog​

  • Report all DataObjects attributes (incl. primary and foreign keys if defined) for visualisation of data catalog in BI tool
  • Metadata support for categorizing Actions and DataObjects, see also Metadata
  • Custom metadata attributes
  • Column descriptions from Markdown files, schemaMin or the ScalaDoc of case classes, applied as column comments to the catalog, see also Column descriptions from ScalaDoc

see also Metadata and SDLB UI, Use in Notebooks

Lineage​

  • Browse lineage of DataObjects and Actions in the UI
  • Column level lineage for Spark, plain-Scala and the SQL engine, exported in the OpenLineage column lineage format with a dry-run
  • Trace a column through the data pipeline in the UI

see also Column Lineage

Data Quality​

  • Metadata support for primary & foreign keys
  • Check & report primary key violations by executing primary key checker action
  • Define and validate row-level Constraints before writing DataObject
  • Define and evaluate Expectations when writing DataObject, trigger warning or error, collect result as custom metric
  • Future: Report data quality (foreign key matching & expectations) by executing data quality reporter action

see also Data Quality

Testing​

see also Testing

Performance​

  • Execute multiple Spark jobs in parallel within the same Spark Session to save resources, see also DAG, Command Line (--parallelism)
  • Explicit caching of input and output DataFrames (cacheInput, cacheOutput), released automatically at the end of the run, see also CustomDataFrameAction parameters
  • Push down transformations into the database with the SQL engine, see also SQL Engine

Housekeeping​

  • Delete, or archive & compact partitions according to configurable expressions, for file, DeltaLake, Iceberg (deletion only) and Jdbc DataObjects, see also HousekeepingMode
  • Extend with custom housekeeping logic

see also Housekeeping, Housekeeping blog post

Remote Agents (experimental)​

  • Execute Actions on a remote SDLB instance, e.g. on-premise, controlled by a main instance in the cloud

see also Remote Agents

User Interface​

  • Configuration viewer with catalog, search and lineage view, grouped by feed, subject area and layer
  • Column lineage and entity relationship diagram of DataObjects and their keys, see also Column Lineage
  • Comprehensive workflow visualization: run graph with action states and metrics, timeline, partition values processed
  • Documentation from metadata and code approach - all configuration elements can be described in the metadata, and are enriched with documentation from code where possible, see also Metadata and SDLB UI
  • Runs locally (SQLite & filesystem) or as cloud service

see also UI blog post, and the UI Demo visualizing the Getting Started data pipeline.