AppSignal

Semaphore’s containers provide stability for AppSignal

  • ⛔️️ Flaky builds
  • ⛔️ Build agents hard to configure and debug
  • ⛔️ Ruby Gem build time 14min
  • ⛔️️ JRuby build time 7min
  • ⛔️ Confusing UI
  • ⛔️ ‘Mixed’ support experience
  • ✅ Reliable builds
  • ✅ Seamless container support
  • ✅ 6.5min Ruby gem build ⬇️ 2.2x
  • ✅ 1.5min JRuby build ⬇️ 4.5x
  • ✅ Intuitive UI
  • ✅ Prompt support

Discover what Semaphore can do for your team 🧑‍💻

The Challenge

The AppSignal team found themselves at the mercy of their old CI tools. Builds were unstable. The team decided it was time to change when they couldn’t get the help they needed from their old CI provider. “CircleCI made a change somewhere in their stack and that broke our build consistently,” explains Tom de Bruijn, an AppSignal developer. What’s more, it wasn’t just one CI provider. Tom continues, “We had a similar problem with TravisCI, builds were flaky and these issues only occurred on TravisCI.”

Because of the spurious build and test failures, the value of the team’s CI setup started to deteriorate. First, a lot of time was lost retrying builds to see if a failure was real. Second, Tom sank a lot of effort into debugging the brittle builds. Third, and most importantly, the random failures started to mask real issues. As Tom describes, “Failing tests snuck in. If the build always or randomly fails you’re not going to look at the build result anymore.”

To top off the problems with their previous CI setup, the team found the UI difficult to use. “People got lost in the CircleCI workflow UI”, relates Tom. This meant that the team avoided building the kind of advanced workflows that could improve build times. Moreover, the array of disparate jobs generated lots of notifications in Slack. Therefore, it was hard to tell when a full set of jobs for one pull request was complete. “The team thought builds were done before they actually were,” Tom recalls.

“Running the Ruby gem builds on Semaphore runs perfectly, no random failures, and it’s done a lot faster.”

Overall, AppSignal needed their new CI/CD solution to

  • ✔️ Give them full control of the execution environment
  • ✔️ Use low maintenance, SaaS build agents
  • ✔️ Be supported by prompt customer service
  • ✔️ Simply model the whole workflow

“The JRuby builds alone went down from 7 minutes to 1 minute 30 seconds a piece!”

The Solution

Tom’s number one piece of advice to anyone considering switching CI providers is to move the test environment to containers. Semaphore’s seamless support for Docker containers allowed the AppSignal workflow to be ported quickly. The team’s server test container, from AppSignal’s own, private Docker Hub collection, gave them total control of the runtime environment and insulated them from external changes.

All jobs in the workflow are collected in one model. “The workflow UI is better,” says Tom, “I no longer have to show people where they can see the entire workflow for example.” As a result of the simpler workflow modelling, Slack notifications arrive only for complete workflows – not for each individual job as before.

Through the improved user-experience of Semaphore, the team has been able to exploit the parallelism their workflow permits. “We’ve split up more builds so separate parts can run in parallel.” Semaphore’s high-quality documentation has been a boon, “I’m able to find most of the answers to my questions in the documentation,” Tom concludes.

Another important aspect of the solution is its low maintenance requirements. Semaphore provisions build-agents as and when needed. As a result, AppSignal engineers don’t have to expend time and people on setup and maintenance of the agents.

AppSignal Workflows
AppSignal workflow in Semaphore

The Results

Tom’s assessment of the results? “Running the Ruby gem builds on Semaphore runs perfectly, no random failures, and it’s done a lot faster.” The eradication of flaky builds directly produces a time-saving. Indirectly, the increased reliability boosts the team’s productivity even further.

The elapsed- and CPU-time of each build cycle has radically improved too. For example, AppSignal’s Ruby gem used to take about 14 wall-clock minutes (and 85 minutes of CPU time). Now it takes just 6.5 minutes (40 minutes of CPU). JRuby builds also sped up: 7 minutes on the old solution, and just 1 minute 30 seconds with Semaphore.

“The build is faster out of the box with the default machine type,” observes Tom. “We’ve switched all of our projects from CircleCI to Semaphore, and have also started migrating the TravisCI builds for our integrations to Semaphore.”

“I’m able to find most of the answers to my questions in the documentation. And if I’m not, support responds within a day with an answer.”

AppSignal is a leading European real-time APM provider founded in 2012.

Working remotely, the team develops an application monitoring tool that merges error catching, performance tracking, server monitoring, metric dashboards and anomaly detection into a combined UI. Bringing these features together allows AppSignal users to generate clear and focused insights.

AppSignal integrates with a diverse range of languages used to build the web: Ruby, Elixir, Node.js and Javascript and is currently used by over 1,000 teams.

Industry

Information Technology

Company Size

11-50 employees

Engineering team

6 people

Stack

Ruby on Rails

MongoDB

Javascript

React

More customer stories

“It’s way faster and we just pay Semaphore for the number of build seconds we actually use. As we’ve spent a bit of time optimizing our build speed, each build costs about 10 cents.”

CTO / Co-Founder

Read case study

“Semaphore allows us to build anywhere from two to twenty branches in parallel, each one in a threaded environment. We had thousands of tests that were running for a really long time and with Semaphore we got the build to run in under 20 minutes.”

QA Lead at 500px

Read case study

“We were using Jenkins to build the old system, and it was hard to maintain and quite painful. We looked at other hosted CI services, but they were very unreliable. When we found Semaphore, we got our application up and running pretty quickly. We now use it for all projects.”

CTO at Simply Business

Read case study

Star us on GitHub