GitLab monorepo CI done right: from a monolithic pipeline to tag-routed Runners

Introduction

We had a monorepo. Frontend, three backend services, a couple of shared libraries, all living in one repo. Great for developers who want to change an API and its caller in a single commit. Rough on CI.

The problem showed up slowly and then all at once. Every push, no matter how tiny, kicked off the whole pipeline. Fixed a typo in the frontend README? Cool, now go get a coffee, because all three backends are about to rebuild and rerun their full test suites for absolutely no reason. Twenty minutes later, you get a green check for work that never needed to happen.

That’s why our CI got that slow, how GitLab’s rules: changes and child pipelines pulled it back to sane, and the part I didn’t expect going in: how much of the fix was really about understanding my Runners, not my YAML.

Why do we even need a Runner?

Here’s the thing that isn’t obvious at the start: GitLab does not run your pipeline.

When you write jobs in .gitlab-ci.yml, GitLab acts as the orchestrator. It reads your config, works out which jobs exist, decides the order and what’s ready to run — and that’s where its job ends. It does not build your Docker image. It does not run your tests. It does not run terraform apply or docker build.

The thing that actually executes your commands is the GitLab Runner: a separate, lightweight agent that lives on some compute(AWS EC2) you point at GitLab. It polls GitLab asking “got any work for me?”, claims a job when one’s available, runs the commands on its machine, and reports the result back.

The cleanest way I’ve found to think about it is a restaurant kitchen. GitLab is the manager reading order tickets and deciding what goes out when. The Runner is the cook at the stove actually making the food. If there’s no cook on shift, orders pile up, and nothing gets served — and that pile of unserved tickets is exactly your pending job

One number matters a lot here, and almost nobody looks at it early on: a Runner’s concurrent setting. In config.toml, it decides how many jobs a Runner will run at the same time.

If your pipeline has ten jobs ready to go and your whole fleet can only run four at once, six of them sit in pending and wait their turn. That queue is invisible in the pipeline graph, so it feels like “CI is slow” when really it’s “CI is waiting for a free Runner.”

The actual problem: everything runs on every push

Our repo looked roughly like this:

And the pipeline built and tested all of it, top to bottom, on every commit. A one-line change in /frontend would rebuild auth, payments, and notifications, run every test they had, and package everything.

Here’s what a trivial frontend change actually triggered, and why the Runners were drowning:

One-line change in/frontendSingle monolithic pipelineBuild + test frontend(needed)Build + test auth(untouched)Build + test payments(untouched)Build + test notifications(untouched)All four compete for thesame Runner capacity

Three of those four builds are pure waste, and all four are fighting over the same limited pool of concurrent Runner slots. Multiply that by every push from every engineer, all day, and you get the worst of both worlds: Runners pinned at full utilisation, and a queue of pending jobs behind them, most of which never needed to run.

The fix came in two parts, and only the first is the one everyone talks about.

First, rules:changes, so a job only runs when files it actually depends on have changed:

Now the frontend job simply doesn’t run if nothing under frontend/ changed. Same pattern for each service. Fewer jobs created means fewer jobs competing for Runner slots.

Second, and this is the bigger lever, we split the one giant pipeline into child pipelines. Each service owns its own .gitlab-ci.yml, and a small parent decides which children to trigger:

So the same trivial frontend change now does this:

Only the frontend child runs. The other three never start, so they never even ask for a Runner. That README typo goes green in about two minutes instead of twenty.

The Runner half of the fix that nobody mentions

Here’s what surprised me. Cutting the wasted jobs was only half the win. The other half was making sure the jobs that did run landed on the right Runners, fast.

Two Runner-side things mattered.

The first is concurrency. Once you split into child pipelines, a change to /libs/shared can legitimately fan out into four child pipelines at once, each with its own build and test jobs. If your fleet’s total concurrent capacity is too low, all that parallelism just turns back into a queue, and you’ve gained nothing. So when we moved to child pipelines, we also tuned Runner concurrency and added a small autoscaling pool so the fan-out could actually run in parallel instead of politely lining up.

The second is routing, where the humble Runner tag earns its keep. Our services don’t all want the same machine. The frontend build wants a Node toolchain, and the payments service is a heavy Docker build that needs more memory. So we tagged each Runner pool by what it can offer, and let each child pipeline ask for what it needs.

In the diagram, every arrow represents a tag match and has two sides.

In the diagram, every arrow represents a tag match and has two sides. A Runner pool advertises its capabilities as tags (“I have Node”, “I have Docker and lots of memory”), and a child pipeline’s jobs request tags (“I need Node”). GitLab only sends a job to a Runner whose tag set contains every tag the job asked for. So the frontend child asks for Node and can only land on the Node pool. The payments child asks for docker and high-memory together, so it can only land on the heavy Docker pool. The auth child asks for just docker and goes to the standard pool, keeping the light job off the expensive box.

One subtlety worth knowing, because it bites people: since the heavy pool also advertises docker, the auth job asking for just [docker] could technically match it too, as its request is a subset of what the heavy pool offers. If you want to guarantee auth never lands on the expensive machine, you tag more specifically (say [docker, standard]) so the requirement only fits one pool. Tags describe capability on the Runner side, and requirement on the job side, and a job runs only where the Runner offers everything it asked for.

Conclusion

Slow CI usually isn’t a Runner problem; it’s a “you’re doing work you didn’t need to do” problem, so cut the wasted jobs first. But once you’ve cut them, the Runners come straight back into the picture, because whether the remaining work is fast depends entirely on having enough Runner concurrency and the right tags to place each job on the right machine.