It’s 2024 and decades-old memes about building software still hold up. Despite advances in the DevOps stack – Docker for containerization, CircleCI for CI/CD, and Terraform for infrastructure as code – many engineering organizations still struggle to deliver fast, consistent, and secure application builds.
The build problem is getting harder due to an increase in software project complexity. Today’s software teams are embracing monorepos and are pulling in record numbers of third party dependencies. At the same time, the number of builds in CI is growing as teams embrace continuous push. In 2024 CircleCI saw a 97% increase in daily workflow volume for top performing teams. The increase in build complexity is slowing down teams.
A new generation of advanced build systems are making builds faster and more reliable. This will change the way companies ship software.
The build bottleneck is growing
A software “build process” is broadly defined as the series of steps for building and testing a piece of software from source code. Software builds are kicked off by engineers locally and by CI systems remotely. Because building software is a core part of the software development lifecycle, slow and flaky builds can be a particularly potent bottleneck.
Engineers know the symptoms of a bad process: fresh builds in the morning take an hour due to cache misses, long builds in CI block PR merges, and onboarding takes days due to environment inconsistencies. All of these issues slow down developers and drive up infra bills. In StackOverflow’s 2024 developer survey, developers ranked quality developer environments and build environments as two of the most important factors in their overall job satisfaction.
The “long build” problem is unfortunately common. We’ve talked with many organizations that experience fresh build times in the 2+ hour range. Large teams are particularly impacted – Graphite reports that the P75 total CI time for teams with over 50 engineers is a whopping 130 minutes:
It’s also getting worse. In 2024, CircleCI workflow times on production branches grew by 11%. The obvious drivers of long build times – expanded CI investment, monorepo adoption, and increased third-party package adoption – show no signs of slowing down.
Flaky builds still continue to haunt teams. According to CircleCI, ~17% of builds on production branches fail. Even mature organizations report experiencing many unexpected breakages per week due to dependency issues.
Enter the advanced build system
While build tools have been around for over fifty years, the newest build systems deliver a major leap forward in power and capability.
Stuart Feldman introduced the staple Unix “make” utility back in 1979. Over the subsequent decades, build tools like CMake, Ant, and Maven incrementally improved the build process, helping engineers build cross-platform projects more efficiently. In the early 2010s, big tech companies took up the build systems torch.
Google, Meta, and X (formerly Twitter) all developed internal build systems while pioneering a monorepo approach to code organization where thousands of developers collaborate in multi-million line repositories. Google’s Bazel, Meta’s Buck, and X’s Pants all support speedy builds through effective caching and remote execution. Each of these build systems has now been open sourced, and a wave of startups have emerged to deliver complementary offerings.
Software teams plagued by build headaches are adopting these advanced build systems to improve developer velocity, reduce infrastructure spend, and improve build consistency. Some new build system offerings, like those from EngFlow, BuildBuddy, and Aspect, are building on top of existing open source projects. Other startups, like Nx, Dagger, and Earthly, offer ground up solutions based on similar principles. By providing solutions that increase build speed, reliability, and security, build system startups are answering the “long build” problem that plagues many software organizations.
Under the hood
Newer build systems are delivering 10x build time speedups and highly reproducible builds. These systems accomplish speedups by supporting features like the subdivision of builds into smaller targets, deterministic dependency management, and remote execution.
A main mechanism in newer build systems is the subdivision of large builds into smaller targets. With Bazel and Buck, developers define targets through strict BUILD files. During a build, the system scans the project for changes, re-building only the targets which have changed. Since most code changes only affect a handful of targets, this dramatically reduces the amount of time needed for a build.
Another feature of build systems is deterministic dependency management. Old systems lack precision: many do not have version pinning requirements and rely on package repositories that do not provide consistency guarantees. This causes inconsistent builds. New systems are more precise. Nix-based systems like Flox and Determinate Systems fully specify an applications’ dependency tree with details on the environment in which each package was built. They also cryptographically hash packages to guarantee incoming dependencies have not been altered since the most recent build.
The impact of these software features is amplified through remote execution. Newer build systems execute builds on remote machines, parallelizing the build of sub-targets and the installation of dependencies. They also cache the output of tasks to reduce overall compute required. Some, like Blacksmith, are running build tasks on high performance machines. Remote execution allows developers and CI systems to invoke builds consistently and quickly.
Many systems expose this advanced functionality through modern syntax, like Dagger’s programmatic build functions and Earthly’s Earthfiles, making them easy to use. They also offer cross language support. Different languages have different build processes and associated challenges – C++ and Rust are compiled whereas Python and Javascript are interpreted. But the common jobs of a build system – to subdivide build tasks, manage dependencies effectively, and execute builds quickly – hold constant across languages and project types.
Monorepos are complicating builds
One major driver of advanced build system usage is monorepo adoption. Monorepos were pioneered by big tech firms in the 2010s, and now, a growing number of teams are embracing them. While monorepos have many advantages – they make it easier to grep a codebase, synchronize cross-project changes, and standardize coding practices – they also have drawbacks. In particular, dependency management can be difficult. It can be challenging to unify package versions across a monorepo, and sheer package volume can lead to long build times. In the Javascript community, monorepo tools like Nx, Turborepo, and Rushstack, which help developers manage the complexity of monorepo builds, have been taking off. Turborepo was acquired by Vercel in 2021.
Chart showing adoption of JS-specific monorepo tools
Prioritizing consistency and security
For some companies, it is consistency, not speed, that drives the need for an advanced build system. In compliance-focused sectors like financial services and aerospace, software teams place a premium on reproducible builds and third party package auditability.
Nix, the open source ecosystem which includes NixOS and Nixpkgs, is gaining strong traction on this front. Nix’s purely functional package manager provides strong consistency guarantees for third party dependencies, and its NixOS linux distribution makes it easy to audit and manage OS configurations. Startups Flox and Determinate Systems, which build on top of Nix, are fueling Nix adoption with a suite of enterprise tools.
Other tools are also benefiting from security tailwinds. Bazel, for example, has positioned itself as an offering for compliance-oriented organizations and cites adoption by many fintech organizations. Bazel lead Tony Aiuto reports general traction with “organizations that worry a lot about compliance and recertifying what they are shipping.”
Scotty, do I need more power?
Not all teams need an advanced build system. Many offerings have a steep learning curve. Nix documentation is famously sparse, and Bazel can take months to integrate. Integrating an advanced build system to an existing codebase can require substantial refactors due to strict dependency management requirements. Small teams with modular codebases may decide the investment isn’t worth the cost.
For these teams, using free subtools can be a good alternative to a full-on build system. Many newer package managers like the pnpm JS package manager offer fast install times and easy onboarding. Tools from incumbent CI providers can also help teams speed up remote builds. CI features like dependency caching and parallelization are helping teams address the biggest bottlenecks in their build process quickly and cheaply.
That being said, the trends that necessitate advanced build systems – third party package adoption, monorepo growth, and CI/CD expansion – are here to stay. Even companies that don’t explicitly sell software, like RedBull, American Airlines, and Caterpillar, are starting to adopt these systems. As new build systems get easier to integrate and adopt, more and more companies stand to benefit from them.
The road ahead
Advanced build systems will make “it worked on my machine” a predicament of the past. By enforcing build hermeticity and offering advanced caching tools, the next generation of build tools will allow developers to quickly and confidently build projects from anywhere.
Usability and cost remain the biggest adoption hurdles. The winning solutions in this space will gain trust through high quality developer experience, cross-platform compatibility, and powerful integrations.
The build systems shift will change development workflows. CI volume will increase as build costs go down. Collaboration will increase as building new projects gets easier. The biggest effect of all, though? That the best justification for a coffee break – “my code’s compiling” – is on its last legs. Guess we’ll have to find a new excuse.
