Choosing a CI/CD tool in 2026 isn’t just a technical decision — it’s a cost and productivity decision. Slow pipelines mean slower shipping, and opaque pricing means budget surprises at the end of the month.
We benchmarked five of the most widely used CI/CD platforms against the same workload: a real Ruby on Rails application (Redmine), with identical OS, runtime version, and database backend across all providers. No cherry-picked runs, no parallelism tricks — just a realistic, repeatable pipeline measured 10 times per provider after cache warm-up.
Here’s what we found.
How We Ran the Benchmark
- Test application: Redmine — a mature, production-grade Ruby on Rails app used by thousands of teams.
- Workload per run: Repository checkout → cache restoration → dependency installation → PostgreSQL database setup → full test suite execution.
- Machine specs: 2 vCPU instances across all providers (the closest standardized tier available on each platform). Ruby 4.0, PostgreSQL, Linux.
- Measurement: 10 consecutive runs per provider, post cache warm-up, single-job execution (no parallelism). No outliers removed.
This methodology mirrors what most small-to-medium engineering teams actually run day-to-day.
At a Glance: Performance and Cost Rankings
1. Semaphore — Fastest and Cheapest
Avg build time: 5m 01s
Cost per run: $0.04
Machine used: f1-standard-2 (2 vCPU, 8 GB RAM)
Semaphore came in first on both speed and cost. Its pipelines finished in just over five minutes — nearly half the time of GitHub Actions and almost a third of CircleCI. At $0.04 per job, it’s also the cheapest option tested.
Semaphore uses ephemeral VMs spun up fresh for every job, which eliminates environment drift and makes test results reproducible. It supports both a fully managed cloud offering and self-hosted agents, and its YAML schema is strict enough to be reliably generated by AI tools.
Best for: Teams that want fast, predictable pipelines without managing infrastructure. Particularly strong for Ruby, Go, and Node.js workloads. The open source enterprise edition is available for free for companies under $5M ARR with 50 users.
Pricing: $0.0075/min → semaphore.io/pricing
2. Buildkite — Fast, But Pricier Than It Looks
Avg build time: 7m 15s
Cost per run: $0.09
Machine used: LINUX_AMD64_2X4 (2 vCPU, 4 GB RAM)
Buildkite is the second-fastest platform in this benchmark, finishing 44.86% slower than Semaphore but well ahead of GitHub Actions and GitLab. It follows a hybrid model: Buildkite orchestrates pipelines, but you supply the compute (your own agents on EC2, GKE, bare metal, etc.).
That architecture gives you full infrastructure control and can be cost-effective at scale if you already run your own servers — but it adds operational overhead. The $0.09 per run figure reflects Buildkite’s managed compute pricing ($0.013/min), which is the most expensive per-minute rate in this benchmark. However, compute costs are only part of the picture.
The hidden cost: per-seat fees. Buildkite’s Pro plan (required for SSO, priority support, and 1-year build retention) charges $30 per active user per month, on top of all compute. For a team of 10 engineers, that’s $300/month in platform fees before a single build runs. At 20 engineers, $600/month — before compute. This makes Buildkite significantly more expensive at team scale than the per-run figures alone suggest.
Best for: Larger engineering teams with existing infrastructure who want fine-grained control over their runners. Less suitable for smaller teams that don’t want to manage agents.
Pricing: $0.013/min (compute) + $30/active user/month (Pro plan)
3. GitHub Actions — Popular, But Slower Than You’d Think
Avg build time: 9m 44s
Cost per run: $0.06
Machine used: ubuntu-latest (2 vCPU, 7 GB RAM)
GitHub Actions is the most widely adopted CI/CD platform, largely because it ships with every GitHub repository. The integration is seamless and the marketplace of third-party actions is vast.
However, in benchmarks it’s 94% slower than Semaphore on the same workload. At scale — say, a million build minutes — that translates to over 15,670 extra engineering hours waiting for pipelines to finish. GitHub Actions also made headlines in early 2026 for a pricing restructure that frustrated many teams and accelerated migration to alternatives.
Best for: Teams already on GitHub who want zero-friction CI setup and aren’t yet optimizing for pipeline speed or cost. Less suitable once you’re running dozens of daily pipeline runs and build time is a bottleneck.
Pricing: $0.0060/min
4. GitLab CI — Integrated, But Expensive at Scale
Avg build time: 11m 15s
Cost per run: $0.11
Machine used: saas-linux-small-amd64 (2 vCPU, 8 GB RAM)
GitLab CI is deeply integrated with GitLab’s broader DevOps platform — from source control to security scanning to deployment. If your team already lives in GitLab, CI/CD is a natural add-on.
Performance-wise, it came in fourth in this benchmark at 11m 15s — 124% slower than Semaphore. The per-run cost is the highest of all five platforms at $0.11 per job ($0.01/min), making it the most expensive option at scale. At a million build minutes, you’d be paying meaningfully more than with any competitor.
Best for: Teams heavily invested in the GitLab ecosystem (merge requests, security, releases) who need tight DevOps integration in one product. The built-in security scanning features are a genuine differentiator.
Pricing: $0.0100/min
5. CircleCI — Slowest in This Benchmark
Avg build time: 13m 18s
Cost per run: $0.08
Machine used: Docker medium (2 vCPU, 4 GB RAM)
CircleCI has been a long-time favourite in the CI/CD space and has a mature feature set — parallelism, test splitting, orbs for reusable config. In this benchmark, however, it was the slowest platform, finishing 165% behind Semaphore. At a million build minutes, that’s 27,519 extra hours of wait time compared to running the same workload on Semaphore.
One caveat: CircleCI’s Docker medium machine provides 4 GB RAM vs Semaphore’s 8 GB — hardware constraints likely contributed to the timing gap. That said, the 4 GB tier is CircleCI’s standard entry-level machine, making it the realistic comparison point for most teams.
Best for: Teams that have invested heavily in CircleCI’s orb ecosystem and parallelism configuration. Worth revisiting pricing and speed benchmarks if you’re scaling build volume.
Pricing: $0.0060/min
The Scale Problem: What Slow Pipelines Actually Cost
Build time differences feel abstract until you multiply them across a real engineering team. Based on 1,000,000 build minutes run on Semaphore (our fastest baseline):
These aren’t just clock hours — they’re developer hours spent waiting for green checkmarks before merging, deploying, or moving to the next task. For a team of 20 engineers shipping 50+ builds a day, the difference between a 5-minute pipeline and a 13-minute pipeline is real, compounding time lost every single workday.
How to Choose
- Go with Semaphore if speed and cost efficiency are priorities and you want managed infrastructure without maintenance overhead. Especially strong for teams migrating from GitHub Actions or CircleCI.
- Go with Buildkite if you have existing server infrastructure and want to run your own agents with platform-level orchestration.
- Go with GitHub Actions if you’re a small team on GitHub and don’t yet have volume high enough for pipeline speed to matter.
- Go with GitLab CI if you’re fully embedded in the GitLab platform and value security scanning and DevOps integration over raw pipeline speed.
- Go with CircleCI if you have complex parallelism setups built around CircleCI’s orb ecosystem and a migration isn’t feasible in the short term.
Methodology Notes
Full benchmark details — including raw run logs, machine specs, and configuration files — are published in the original Semaphore CI/CD Benchmark report. All tests were run by the Semaphore team; we recommend running your own benchmark against your specific workload before making a final decision.
Want to try Semaphore on your own codebase? You can create a project for free and run the benchmark yourself — the configuration files are public.
Want to discuss this article? Join our Discord.
