Why Do Builds Pass Locally but Fail in CI?

“Works on my machine” is one of the most common and frustrating problems in test automation and CI/CD. A build that passes locally but fails in continuous integration can block deployments, slow teams down, and erode trust in automated tests.

This behavior is rarely random. In most cases, CI failures expose real differences between local development environments and CI environments—or hidden assumptions in tests themselves. Understanding why this happens is essential for building reliable CI pipelines and maintainable test automation.

The Core Problem: Local and CI Environments Are Different

Local development environments are rarely clean. Over time, they accumulate cached dependencies, globally installed tools, background services, and leftover state. CI environments are intentionally the opposite.

In CI/CD platforms such as Semaphore, each job runs in a clean, isolated environment (container or virtual machine) with:

• Explicitly defined operating systems and runtimes
• No hidden state unless configured
• Only declared dependencies
• Known machine types and resource limits

This difference is intentional. CI environments are designed to be reproducible. When something fails in CI but passes locally, CI is often revealing a real issue.

A Common Example: A Dependency That Works Locally but Fails in CI

One of the most frequent reasons builds pass locally but fail in CI is a dependency that exists on a developer’s machine but is not explicitly declared.

❌ Before: Passes Locally, Fails in CI

Locally, tests pass:

$ npm test
PASS user.test.js

In CI, the same build fails:

Error: Cannot find module 'uuid'

Here is the relevant code.

const { v4: uuidv4 } = require("uuid");

test("creates a user id", () => {
  expect(uuidv4()).toBeDefined();
});

{
  "name": "app",
  "scripts": {
    "test": "jest"
  },
  "dependencies": {
    "express": "^4.18.0"
  }
}

What’s actually happening

The uuid package exists locally because it was installed at some point in the past—perhaps indirectly or during earlier development. Node can resolve it locally, so the test passes.

In CI, dependencies are installed from scratch using only what’s declared in package.json and the lock file. Because uuid is not listed, the module cannot be found and the test fails.

Why CI fails but local passes

• Local environments accumulate state over time
• CI environments start clean on every run
• CI exposes undeclared dependencies that local machines hide

This is a classic “works on my machine” failure that CI is designed to catch.

✅ After: Explicit Dependency Declaration

The fix is simple and intentional: declare the dependency explicitly.

{
  "name": "app",
  "scripts": {
    "test": "jest"
  },
  "dependencies": {
    "express": "^4.18.0",
    "uuid": "^9.0.0"
  }
}

Result

• Tests pass locally
• Tests pass in CI
• The dependency graph is explicit and reproducible

Why Lock Files Should Be Committed

Even when all dependencies are declared, builds can still behave differently across environments if lock files are missing or ignored.

Lock files (such as package-lock.json, yarn.lock, Pipfile.lock, poetry.lock, go.sum, or Gemfile.lock) record the exact versions of every dependency and transitive dependency used in a project.

What Lock Files Do

Lock files ensure that:

• Every developer installs the same dependency versions
• CI environments install the exact same dependency tree
• Builds remain reproducible over time

Without a lock file, package managers resolve version ranges dynamically (for example, ^4.18.0). Over time, newer compatible versions may be installed, leading to dependency drift—where different machines run slightly different code.

Why This Matters in CI

CI pipelines always install dependencies from scratch. If no lock file is present, CI may resolve newer dependency versions than those installed locally, even if the code hasn’t changed. This often leads to:

• Unexpected test failures
• Subtle behavioral differences
• Non-deterministic builds

By committing lock files and using deterministic install commands (such as npm ci), teams ensure that local development, CI, and production environments all use the same dependency versions.

This consistency is foundational to reliable test automation and stable CI/CD pipelines.

Other Common Reasons Tests Fail in CI but Pass Locally

Missing Environment Variables

Tests often rely on configuration provided via environment variables.

Locally:

export DATABASE_URL=postgres://localhost/dev
pytest

In CI:

KeyError: DATABASE_URL

CI environments start empty unless variables are explicitly configured. Semaphore allows defining environment variables securely at the project or pipeline level.

OS and File System Differences

Local machines often run macOS or Windows, while CI environments typically run Linux. File system case sensitivity is a common source of failure.

Example:

Error: Cannot find module './Config/settings.json'

On macOS this may work; on Linux it fails. CI surfaces portability issues early.

Hidden Test State and Order Dependency

Tests that rely on leftover state from previous runs may pass locally but fail in CI, where every run starts clean.

Reliable test automation requires tests to be:

• Isolated
• Idempotent
• Order-independent

CI environments make violations of these rules visible.

Flaky Tests

Flaky tests fail intermittently due to timing assumptions, race conditions, or resource constraints. CI environments often expose flakiness more clearly due to parallel execution and constrained resources.

Semaphore provides test reports that aggregate results across runs, making flaky behavior easier to identify:

CI Failures Are Signals, Not Noise

CI environments don’t break builds. They reveal:

• Configuration drift
• Undeclared or drifting dependencies
• Brittle tests
• Platform assumptions

Semaphore’s use of clean environments and defined machine types helps surface these issues early, before they reach production.

How to Reproduce CI Failures Locally

To debug CI-only failures more effectively:

• Run tests using the same container image as CI
• Match runtime and dependency versions
• Clear local caches
• Run tests in isolation

Aligning local development environments with CI dramatically reduces “works on my machine” problems.

Summary

Builds pass locally but fail in CI because local environments hide issues that CI environments expose. Differences in dependencies, missing lock files, configuration, operating systems, test state, and resource constraints all contribute to this behavior.

By treating CI as the source of truth, committing lock files, writing isolated tests, and using CI insights such as test reports and flaky test detection, teams can build reliable, predictable test automation pipelines.

FAQs