Your data. One place to start.

Master data: definition and ingestion

Master data describes reusable business objects such as customers, products and locations. Transaction data records individual events, such as orders. Preserve stable keys when importing master data so later analyses can associate events with the right object. lavalake loads source tables into Iceberg; mappings and cleanup can be defined during transformation.

Master data quality starts with keys and values

Practical checks include unique customer numbers, consistent country values and valid data types. Compare row counts after import and check duplicate keys with SQL. Business rules such as allowed statuses can then be checked in lavaflow. Maintaining master data in the source remains part of the process.

Orchestration for recurring data integration

Orchestration defines when processing steps run and how they follow each other. For example, load a source first, transform its data, then refresh the analysis. Schedules and lavaflow connect these tasks in lavalake; run history helps you review results and failures.

Drop files into one place

Use “Upload files” to bring Excel and CSV files into the lake as tables. PDF, Word, text and Markdown become available to search and Pyro. Preview and column checks guide table imports.

Keep changes flowing

Scheduled delta loads pick up new data using a watermark column. An optional PostgreSQL CDC integration captures inserts, updates and deletes from the change log. Source tables require a primary key and configured logical replication.

Verified against real systems

PostgreSQL, MySQL, SQL Server and Snowflake imports have been tested against real systems. Real-system tests for SAP, CRM integrations and DATEV remain outstanding.

Databases as a source

A source is registered in the console: type, host, database, credentials. Credentials are stored AES-256-GCM encrypted, access is read-only, and TLS can be enforced per source rather than globally. You then pick the tables to bring over as Iceberg.

Type        postgres | mysql | mssql
Host        erp-db.dc.internal:5432
Database    sales
Login       reader / ********     (stored encrypted)
TLS         enforced

Bring over
  sales.public.orders     ->  lakehouse.raw.orders
  sales.public.line_items ->  lakehouse.raw.line_items

Outbound access is not unrestricted: an egress guard stops a registered address from pointing at internal metadata endpoints or into other network segments. A data source is user input, and user input gets checked.

• PostgreSQL, MySQL, MS SQL Server, SAP Business One and SAP S/4HANA
• Read-only access, credentials encrypted
• TLS enforceable per source
• PostgreSQL CDC, scheduled delta loads and automatic file imports

SAP Business One and S/4HANA

Business One connects through the Service Layer, S/4HANA through an OData service on the gateway. S/4HANA supports OData V2 and V4, detects the version from metadata and preserves the selected client across pages. Each source points to one service; the SAP system is read without modification.

Files with a modeling assistant

Uploading a CSV is the easy part. The hard part is everything after: which column is a date, which is an amount, what does an empty field mean, and which rows are duplicates. The import assistant takes exactly those steps — detect and correct types, rename columns, declare null tokens, drop duplicates — and only then loads.

Large files report loading progress in rows, and a running import stays visible while you work on another page. That sounds minor, but it is the difference between a tool people use and one they avoid after the second timeout.

Read DATEV files directly

File import automatically detects EXTF and DTVF files, reads column names from the second row and handles Windows-1252 encoding. Booking values remain unchanged. This is file import; DATEVconnect and the DATEV Cloud API are not connected. Testing with a real export file is still outstanding.

Documents become tables

This is the decision we care about most. A document is not placed in a second content store with its own access control. It is chunked, and the chunks become an ordinary Iceberg table: one row per chunk, with the text and character offsets back into the original.

Because a document becomes a table, permissions, audit and lineage apply to documents without a single extra rule.

The text column is then embedded — through the same pipeline as any other table. The search index stores embeddings, text chunks and source-row references; the lakehouse table remains authoritative. Search re-fetches the chunk from Trino later, under the searcher's token. Masked columns therefore stay masked in search results too.

Parsing runs in a separate worker with time and memory limits. A malformed or malicious PDF does not take down the service that manages permissions.

Fileshare crawler and MCP ingestion

Instead of uploading files one by one, the crawler points at a directory. And because MCP servers can be registered as sources, you can attach systems that have no JDBC driver but do have an MCP server — SAP through SAP-MCP, for instance.

You pick the target schema, and the convention stays yours — bronze/silver/gold if that is how you work, or whatever your house already uses.

Data engineering: build reliable data paths

Data engineering covers the technical steps from source to usable data: connecting, loading, transforming and validating it. A data engineer also defines keys, data types and update methods. In lavalake, source management, file imports and lavaflow connect these tasks. Business definitions are agreed with the people who use the results.

Parsing: turn files into processable content

Parsing means reading structured input according to its format rules. CSV parsing identifies rows and fields; document processing extracts text for further use. Check the preview and column types when importing a table. Successfully reading content still needs to be followed by business validation.