Confluent Accelerates CI/CD by 70% After Migrating from Jenkins to Semaphore

Try out Semaphore

Sneha Wadhwa, Senior Software Engineer II at Confluent, shares how migrating from Jenkins to Semaphore transformed the company’s build and release workflows. By modularizing pipelines, maximizing parallelism, and leveraging Semaphore’s artifacts and promotion features, Confluent cut packaging build times by 70% and reduced release durations by 75%, while using 50% fewer resources.

In this video you’ll find the answers to these questions:

Why did Confluent migrate from Jenkins to Semaphore?

Confluent had been using Semaphore for several cloud workloads and decided to unify all CI/CD processes under a single platform. The goal was to reduce maintenance, streamline developer experience, and eliminate the overhead of managing multiple systems.

“We wanted one unified CI/CD system for the entire company—no more juggling tools across teams.”

What workflows were migrated?

The Confluent Platform (CP) packaging and release pipelines were the first to move. These pipelines handle more than 35–40 components, building and releasing complex combinations like tarballs, RPMs, Debian packages, JAR libraries, and Docker images for both all-in-one and modular deployments.

“Migrating such complex workflows wasn’t easy, but the effort paid off—our processes are now more efficient and scalable.”

What design principles guided the migration?

The team didn’t simply lift and shift Jenkins pipelines—they redesigned them using key principles:

Modularization: Decomposed monolithic Jenkins stages into smaller Semaphore blocks, enabling easier maintenance and better scalability.
Caching with Semaphore Artifacts: Used built-in artifact caching to transfer build outputs between pipeline stages, speeding up packaging and dependency handling.
Parallelism: Ran independent jobs simultaneously across agents, drastically reducing overall pipeline time.
Dynamic Workflows: Leveraged templating to dynamically generate and trigger workflows on the fly.
Promotions: Replaced single large pipelines with modular promotion workflows that can rerun selectively and execute in parallel for faster recovery and higher reliability.

“By modularizing and parallelizing, we transformed hours-long builds into sleek, efficient workflows.”

How did automation and monitoring improve reliability?

The Confluent team implemented automated Slack alerts, Jira ticket creation, and QuickSight dashboards for real-time visibility and faster resolution.

Failures instantly trigger Slack notifications and Jira issues with build details.
Metrics and pass/fail data are pushed to dashboards for trend tracking.

“The mean time to detect failures dropped from 30 minutes to zero—completely eliminating manual triage.”

What were the results?

The impact was dramatic:

Packaging builds: reduced from 7.5 hours → 2.5 hours (70% faster)
Resource usage: down by 50%
Release pipelines: cut from 30–35 hours → 8 hours (75% faster)

“The need for speed was our driving force—and Semaphore delivered.”

How was Semaphore’s support during migration?

Confluent’s engineers praised Semaphore’s responsiveness and technical collaboration.

“A virtual hi-five and a big thank-you to the Semaphore team for their incredible support throughout this journey.”

Confluent offers a streaming platform based on Apache Kafka that enables companies to easily access data as real-time streams.

Industry

Data Streaming / Software

Headquarters

Mountain View, California

Stack

Java

Docker

Kafka

Debian

Real Results. Real Impact. 🎯📈

View all case studies