Senior Data Platform Engineer — Kafka & PostgreSQL

We are looking for an experienced Senior Data Platform Engineer to own the event-streaming, transactional database, and custom connectivity backbone driving the k0rdent-ai platform — our multi-tenant control plane for enterprise GPU infrastructure. Every cluster provisioned, every GPU-hour consumed, and every tenant action produces high-cardinality metadata that must be transactionally recorded and securely isolated.

You will architect the core pipeline connecting our streaming data plane (Apache Kafka via Strimzi) to our relational data tier (Open Source PostgreSQL via CloudNativePG), which runs across both elastic Kubernetes clusters and high-performance bare metal hardware. Beyond standard administration, a substantial focus of this role is custom ecosystem engineering: deep Go (Golang) systems programming to build custom open-source clients and connectors, design specialized integration libraries, and develop proprietary middleware that bridges our database multi-tenancy with enterprise identity planes like Active Directory (AD)/LDAP.

Because this system is the transactional record of every tenant action across the platform, this role also owns its disaster-recovery posture — designing how the platform survives a full region or cluster loss, not just a single-node failure.

Main Responsibilities

• Hybrid Database Architecture: Design, deploy, and operate high-availability Open Source PostgreSQL topologies (via CloudNativePG on Kubernetes and native Patroni-style topologies on bare metal) and distributed Apache Kafka clusters (via Strimzi), across both containerized Kubernetes environments and high-performance bare metal hardware.

• Custom Client & Connector Development: Write custom, enterprise-grade open-source Kafka clients, standalone connector services, and utility libraries from scratch using preferably Go (Golang) to extend data capabilities where off-the-shelf tooling falls short.

• Enterprise Identity & Data Integration: Architect and build system integrations connecting PostgreSQL authentication and row-level security (RLS) policy evaluation to enterprise Active Directory (AD), LDAP, and OIDC identity providers — via role-mapping and session-context layers (e.g., mapped Postgres roles, JWT claims consumed by RLS policies) rather than a direct connection.

• Transactional Architecture & CDC: Scale high-throughput Change Data Capture (CDC) pipelines via Debezium and Kafka Connect. Implement resilient architectural patterns to maintain absolute data integrity between databases and topics without dual-write risk.

• Cross-Region Resilience & Disaster Recovery: Design and operate cross-region failover and disaster-recovery orchestration for both PostgreSQL and Kafka — including replication topology (sync vs. async trade-offs), split-brain prevention via quorum/witness mechanisms, and explicit RPO/RTO targets for a full region or cluster loss, not just single-node HA.

• PostgreSQL & Infra Tuning: Optimize PostgreSQL instances for heavy ingestion and zero-downtime operations. Tune Write-Ahead Logs (WAL), logical replication streams, connection pooling (PgBouncer), and configure underlying Kubernetes infrastructure primitives (CSI storage volumes and CNI network paths) to eliminate replication lag and unnecessary cross-node latency.

• GitOps & Self-Service Platforming: Maintain a strictly declarative infrastructure-as-code (IaC) culture using Terraform and ArgoCD, creating self-service workflows so internal product teams can securely provision databases, topics, schemas, and ACLs through code.

• Security & Isolation: Implement strict multi-tenant isolation, combining database-level row-level security (RLS) with CNI network policies, mTLS, and Kafka topic-level RBAC.

We don't expect any one candidate to check every box below — if your experience is strong across most of these areas, we encourage you to apply.

Must Have

Platform Seniority: 10+ years in systems, software, data, or platform engineering, with a track record of owning large-scale, business-critical data infrastructure.

• Programming: Go (Golang) or similar engineering skills — comfortable writing clean, concurrent systems code, building custom client libraries, and interacting directly with database/network APIs.

• Open Source PostgreSQL Expertise: 5+ years managing native PostgreSQL. Deep knowledge of database internals (WAL streams, logical replication, publications/subscriptions, and connection architectures) across both Kubernetes stateful environments and bare metal physical nodes.

• Production PostgreSQL-on-Kubernetes: Hands-on experience running PostgreSQL on Kubernetes via a mature Postgres operator — CloudNativePG (CNPG) preferred, but equivalent experience with Zalando's postgres-operator or a comparable operator is acceptable. What matters is depth with declarative cluster lifecycle, failover, and backup/PITR management, not the specific product name.

• Production Kafka: Strong experience managing Apache Kafka at scale on Kubernetes via a declarative operator — Strimzi preferred, but equivalent experience with Confluent for Kubernetes, Koperator, or a comparable operator is acceptable. What matters is depth with operator-managed broker lifecycle and scaling, not the specific product name.

• Custom Data Movement & CDC: Proven experience tuning Kafka Connect and Debezium pipelines. Experience writing or contributing to open-source database connectors, Kafka client libraries, or integration frameworks.

• Enterprise Identity Planes: Hands-on experience building integrations connecting distributed systems to an enterprise IAM/SSO provider — Active Directory, LDAP, Okta, Azure AD, or a generic OIDC provider all count; the specific product matters less than real experience integrating application-level authorization with an enterprise identity system.

• Kubernetes & Cloud Infrastructure: Strong grasp of K8s primitives (Pod lifecycles, Operators, CSI storage layers, and CNI overlay routing), backed by production experience in AWS or GCP.

Nice to Have

• Experience designing and operating cross-region disaster recovery and failover orchestration for stateful systems (Postgres and/or Kafka), including split-brain prevention and RPO/RTO trade-off decisions.

• Active contributions to open-source Kafka connectors, PostgreSQL operators (e.g., CloudNativePG, Zalando), or Go-based database libraries.

• Experience with schema governance for event streams (e.g., Karapace or Confluent Schema Registry) and compatibility-mode management for evolving event contracts.

• Familiarity with high-cardinality telemetry ingestion using Apache Flink or Kafka Streams.

• Familiarity with US enterprise compliance benchmarks (SOC 2, ISO 27001).

• Hands-on knowledge of open-source Redis (caching, pub/sub, or as a lightweight data store) — a huge plus.

• Hands-on knowledge of open-source MongoDB (document modeling, replication, sharding) — a huge plus.

Education and Experience

Bachelor's degree in Computer Science & Engineering (or a related field), or 10+ years of equivalent related experience.

What does Mirantis offer you?

• Work with an established Silicon Valley leader in the cloud infrastructure industry;
• Work with exceptionally passionate, talented and engaging colleagues, helping Fortune 500 and Global 2000 customers implement next-generation cloud technologies;
• Be a part of cutting-edge, open-source innovation;
• Thrive in the high-energy environment of a young company where openness, collaboration, risk-taking, and continuous growth are valued;
• Professional development and training;
• Attend conferences and working groups;
• Company outings, happy hours, hackathons, and tech talks;
• Receive a competitive compensation package with a strong benefits plan.

We are a Leader for Container Management in G2 (#2 after AWS)!

Mirantis, an IREN company, is the Kubernetes-native AI infrastructure company, enabling organizations to build and operate scalable, secure, and sovereign infrastructure for modern AI, machine learning, and data-intensive applications. By combining open source innovation with deep expertise in Kubernetes orchestration, Mirantis empowers platform engineering teams to deliver composable, production-ready developer platforms across any environment—on-premises, in the cloud, at the edge, or in sovereign data centers. As enterprises navigate the growing complexity of AI-driven workloads, Mirantis delivers the automation, GPU orchestration, and policy-driven control needed to manage infrastructure with confidence and agility. Committed to open standards and freedom from lock-in, Mirantis ensures that customers retain full control of their infrastructure strategy.

Originally posted on Himalayas