CI optimization is easiest to reason about when the problem is concrete: one pipeline, one critical path, and one cost model.
Lawmatics reached out to us with that kind of problem. Their application pipeline was already parallelized and already using a sensible CI structure. The remaining question was whether the most expensive part of the pipeline was running on the right machine class.
The goal was not to make CI faster at any cost. It was to reduce compute spend without making developers wait longer for feedback.
That constraint shaped the work:
- Preserve or improve the pipeline pass rate.
- Preserve or improve active pipeline duration.
- Change one high-leverage variable first.
- Measure production runs before and after the rollout.
The result: Lawmatics reduced average application compute cost per pipeline by 39.3% while reducing active pipeline duration by 15.8%.
The Shape of the Pipeline
The application pipeline had a common CI profile: a parallel browser-test section dominated both runtime and compute cost.
Before the change, the main browser end-to-end test section ran 16 parallel jobs on f1-standard-4 machines. For this analysis, f1-standard-4 cost $0.015 per minute. The smaller f1-standard-2 machine cost $0.0075 per minute, or 50% less per minute.
The question was whether the browser workload justified the larger runner.
Browser tests do not always scale cleanly with more CPU. A job can still spend time in setup, browser execution, network calls, file I/O, and single-threaded work. When extra CPU is not on the critical path, a larger runner can increase cost without improving feedback time enough to justify it.
The First Recommendation
The first recommendation was intentionally narrow:
- Move the browser end-to-end test section from f1-standard-4 to f1-standard-2.
- Keep the existing parallelism at first.
- Add a 2 GB swap file if the smaller machine showed memory pressure.
- Measure production runs after the change.
We did not start by changing parallelism. On paper, reducing parallelism can look like an easy cost lever. In practice, it increases the amount of test work per job and can make the browser-test section slower.
The safer sequence was to change the machine class first, protect wall-clock time, and then re-evaluate parallelism with fresh data.
That sequencing matters. If machine type, parallelism, caching, database settings, and test distribution all change at once, the final result may still be good, but it becomes harder to understand which changes mattered.
What Lawmatics Implemented
The final implementation went beyond the first runner-sizing recommendation. That is where Semaphore’s Customer Success team can be the most useful.
Semaphore brought the external view: runner sizing, pipeline structure, cost modeling, and measurement. Lawmatics brought the application context: which services were safe to tune, which caches were valid, and which build outputs could be reused.
The implemented changes included:
- Moving the browser end-to-end test section from f1-standard-4 to f1-standard-2.
- Adding a 2 GB swap file as a memory safety backstop.
- Increasing browser-test parallelism from 16 jobs to 20 jobs.
- Using timing artifacts so the longest specs could run first.
- Tuning the CI database for a disposable test environment by disabling durability settings that did not matter after the job exited.
- Moving PostgreSQL data to memory-backed storage and reducing background database work.
- Tightening dependency and system-package caches for gems, node modules, and selected system packages.
- Running independent setup work in parallel with explicit waits.
- Moving the frontend build into a separate job and passing it forward as an artifact instead of rebuilding it where it was not needed.
Some of these changes came from the initial recommendation. Some came from the Lawmatics team after the optimization work created a reason to inspect the pipeline more closely.
That is the right outcome. A good optimization engagement should not create dependency on an outside reviewer for every pipeline edit. It should give the team enough structure and signal to find the next improvements themselves.
What Changed
The before sample contains 10 successful application pipelines. The after sample contains 10 successful application pipelines collected roughly two weeks after the production rollout.
Active pipeline duration excludes queue time. Queue time matters operationally, but it does not measure whether the pipeline itself became faster or slower.
| Metric | Before | After | Change |
|---|---|---|---|
| Application pipelines passed | 10/10 | 10/10 | |
| Browser-test jobs passed | 160/160 | 200/200 | |
| Browser-test parallelism | 16 jobs | 20 jobs | +25.0% jobs |
| Average browser job duration | 11:14.3 | 10:22.6 | -0:51.8 (-7.7%) |
| Average browser-test block duration | 12:50.4 | 10:47.1 | -2:03.3 (-16.0%) |
| Average active application pipeline duration | 13:05.0 | 11:01.1 | -2:03.9 (-15.8%) |
| Average browser-test compute cost per pipeline | $2.674 | $1.545 | -$1.130 (-42.2%) |
| Average application compute cost per pipeline | $2.860 | $1.737 | -$1.123 (-39.3%) |
| Average likely published passed tests per pipeline | 2,940.3 | 3,453.5 | +513.2 (+17.5%) |
The browser-test section became faster and cheaper at the same time. Average browser-test block duration dropped from 12:50.4 to 10:47.1, a 16.0% improvement. Average active application pipeline duration dropped from 13:05.0 to 11:01.1, a 15.8% improvement.
The cost reduction followed the same pattern. The browser-test section moved to a runner with a 50% lower per-minute rate. Even after increasing parallelism from 16 to 20 jobs, average browser-test compute cost fell from $2.674 to $1.545 per pipeline, a 42.2% reduction. Average total application compute cost fell from $2.860 to $1.737 per pipeline, a 39.3% reduction.
The passed-test count makes the comparison more conservative. The likely published passed-test count increased from 2,940.3 per pipeline before the rollout to 3,453.5 after the rollout. That is a 17.5% increase in observed passed tests. The pipeline still ran faster and cost less.
The Tail Improved Too
For parallel browser tests, average job duration is useful, but the slowest job often determines when the whole section finishes. The tail matters.
In the before sample, the slowest browser job took 14:55.0. In the after sample, the slowest job took 11:51.0. The p95 job duration moved from 13:44.0 to 11:17.0.
That matches the implementation. More jobs reduced the amount of test work per job. Longest-spec-first ordering reduced imbalance. The smaller runner did not cause the tail to regress.
Why This Worked
There was no single trick. The result came from matching several changes to the workload.
First, the most expensive section was right-sized. The previous machine type had a higher per-minute rate, but the browser workload did not appear to use the extra capacity in a way that justified the cost.
Second, feedback time was protected. Lawmatics did not simply move to cheaper machines and accept slower builds. Parallelism increased from 16 to 20 jobs, and test ordering reduced job imbalance. Wall-clock time improved while compute cost dropped.
Third, Lawmatics removed application-specific overhead. Database durability was unnecessary for a disposable CI database. Some setup work could run concurrently. Some build output could be produced once and reused. These are not generic YAML tips. They require application context.
Fourth, the team measured the result after rollout. The after data came from production pipeline runs collected roughly two weeks after the changes went live, not from a single proof-of-concept run.
The sample is still limited: 10 successful runs before and 10 successful runs after. But it is useful enough to show the direction and size of the improvement.
What This Says About CI Optimization
CI optimization works best when it is treated as shared engineering work, not a checklist of generic recommendations.
Semaphore brought runner sizing, pipeline mechanics, cost modeling, job timing, and a before/after measurement approach. Lawmatics brought the internal knowledge needed to make application-level changes safely.
That combination is why the outcome was larger than the first recommendation. The initial recommendation created a direction. The Lawmatics team used that direction to make deeper pipeline improvements.
A Practical Way to Approach Similar Pipelines
For teams with a similar CI shape, start with a few narrow questions:
- Which blocks control active pipeline duration?
- Which jobs account for most of the compute cost?
- Are the most expensive jobs using the machine size they run on?
- Is parallelism protecting wall-clock time, or hiding avoidable setup overhead?
- Are timing artifacts being used to reduce job imbalance?
- Which services are disposable in CI and can be tuned differently from production?
- Which setup steps are repeated because of pipeline structure rather than need?
The answer is rarely to apply every optimization at once. The better path is to isolate the first high-leverage change, roll it out with a rollback path, collect enough runs, and then decide what to tune next.
The Takeaway
The final numbers from this Lawmatics engagement are specific to one pipeline:
- 39.3% lower average application compute cost per pipeline.
- 15.8% faster active pipeline duration.
- 16.0% faster browser-test block duration.
- 17.5% higher observed passed-test count in the after sample.
The broader lesson is the process. Start from the logs. Find the expensive critical path. Check whether the current machine class matches the workload. Protect feedback time while changing the cost basis. Then use the team’s application knowledge to remove overhead that only they can fully see.
That is what made this optimization work: a short sprint, real production data on both sides of the change, and a team that used the first recommendation as a starting point rather than the end of the work.
Want to discuss this article? Join our Discord.
