AI agents can generate and modify code quickly. But when a test fails, speed is no longer enough. The agent needs enough context to understand what happened, why it happened, and whether the code or the test itself needs to change.
In our latest product update, Marko Gaฤeลกa is joined by Semaphore engineers Nick and Marcos to discuss how we are making testing more useful for both developers and their AI agents.
Watch the product update below.
More complete test context
Test reports often lose useful information while translating results from different test runners into a common format. This can leave developers and AI agents without the context they need to interpret a failure.
We are exploring a more native approach. Instead of relying on one generic format for every test runner, Semaphore can collect richer information directly from each testing framework.
For some ecosystems, this means using the most detailed output already available. For others, it means building thin integrations that preserve the information produced by the test runner. Initial work includes support for frameworks such as xUnit and RSpec.
The goal is simple: provide concise test reports without discarding the details needed to diagnose a failure.
Smarter flaky test detection
Semaphore already helps teams identify flaky tests. However, the current detection rules can miss tests that behave inconsistently across branches or over a longer period.
We are working toward a signal based model for detecting flakiness. Events such as test reruns and changing results for the same commit can contribute to a flakiness score, with different signals receiving different weights.
When a test crosses a defined threshold, Semaphore could notify the team through the product, notifications, or webhooks. The score would also decrease over time when the test stops producing inconsistent results.
This approach should help teams catch persistent flaky tests without permanently labelling tests because of an isolated incident.
Running the tests that matter
Another area under investigation is the relationship between code changes and test coverage.
By using code graphs, Semaphore could identify which parts of a system are affected by a change and determine which tests should run. This work is still at an early research stage, but initial results across some technology stacks are promising.
More targeted test selection could shorten feedback loops while maintaining confidence that the relevant code has been tested.
Faster processing of large test reports
Large test reports can also be difficult to store, transfer, and process.
We are changing how these reports are represented internally by moving toward JSON Lines. In this format, each line can be parsed and processed independently. This should improve processing performance and make it easier to work with large result sets.
Why code maps matter
The team also discussed how code maps are already improving AI assisted development inside Semaphore.
Semaphore has a large codebase composed of many services, technologies, and languages. Code maps give agents a shared view of how requests, functions, services, and dependencies connect.
Nick and Marcos have found that this helps agents:
- Locate the relevant part of the codebase faster
- Understand paths across different services and languages
- Work with shorter prompts
- Use fewer tokens
- Check existing context instead of guessing
- Produce fewer hallucinations
A code map does not have to begin as a complex system. Marcos started with Markdown files organised like a small internal wiki. He then added more specialised tools as his needs developed.
What comes next
Testing will be a major focus for Semaphore over the coming months.
We are working to make richer test context, better flaky test detection, faster report processing, and more intelligent test selection easier to use with less manual configuration.
The aim is to make test information useful out of the box, not only for developers, but also for the AI agents increasingly participating in the software development lifecycle.
Watch the full video to hear directly from the team and see where this work is heading.
Want to discuss this article? Join our Discord.
