Tim van Leuverden,

A clean-sheet, multi-tenant ELTC (Extract, Load, Transform, Charge) platform for automated data ingestion, transformation, and billing. Together with one other backend engineer, we designed and built a Kubernetes-native microservice architecture with ~25 services in Rust and Go, all communicating over gRPC with shared Protocol Buffer contracts and generated Go and TypeScript SDKs.

We built a data extraction service (Rust) that could pull data from HTTP APIs (with OAuth2, JWT, and cloud IAM authentication), relational databases (MySQL, PostgreSQL, SQL Server, Oracle), and cloud storage (S3, Azure Blob, GCS). A data loading service (Go) then ingested the parsed data (JSON, XML, CSV, Parquet) into PostgreSQL with automatic schema management. A data transformation service (Go) applied SQL-based normalization to the loaded data. Finally, we wrote a billing engine (Rust) that executed configurable charging rules supporting tiered pricing, volume discounts, upcharges, and incremental bucketing (like tax brackets).

We made pipeline orchestration fully Kubernetes-native: a workflow orchestration engine (Rust) resolved job dependencies and scheduled ELTC stages as K8s Jobs via a job orchestration service (Go), while a dedicated watchdog service (Go) reported real-time job status back. Each tenant got its own Kubernetes namespace with isolated PostgreSQL and MongoDB instances, provisioned by a dedicated tenant lifecycle manager. Reporting ran on Cube.js with Keycloak authentication and Cerbos authorization. We deployed the entire stack via Helm charts with OpenTelemetry instrumentation for observability.

My primary contributions were the data extraction service, billing engine, a custom template engine (Rust, also compiled to WASM), a pluggable storage abstraction layer, and a CLI tool. I also contributed significantly to the workflow orchestration engine, data transformation service, tenant agent, data loading service, and database migrations.

When I wanted to play some games with my friends, often the question of “what game should we play” would come up. I found a site that would randomly give you a game (WhatShouldIPlayOnSteam.com) but only for one user. So we would start asking each other “do you have X” or “feel like playing some X” for half an hour, trying to find a good game to play. So I decided to make a site that would do just that, find a game we should play.

The site consists of two parts, a VueJS front-end and a NodeJS back-end which communicates to the Steam APIs. However, even during initial development I found out that the Steam API’s were quite slow, with response time upward of a second. So I had to add some form of caching, I did that using Redis.

After adding the first layer of caching, I also added another layer of caching for the finding of games everybody has.

So it now caches player ID lookup by “vanity URL”, player game list, and multiple player common game lists. Luckily Redis provides an easy solution that kills cache entries after a while, so I didn’t need to worry about the server using too much memory.