Separating GHC's CI from the forge (or: woodpecker)

[hasufell]

Rendered: https://github.com/hasufell/tech-proposals/blob/woodpecker/proposals/0000-woodpecker.md

[Bodigrim]

As an occasional small-scale contributor to GHC, I can witness that the existent CI is counterproductive, especially if you are not looking to spend the next week babysitting it (which might be not too bad for the core team, who hang around working on other patches anyway, but terribly disheartening for others). And that's despite years of efforts from GHC team and Bryan. My view is that improving CI experience is vital for GHC's long-term sustainability and gaining new contributors.

[chreekat]

I agree with this direction.

One thing that may need more explicit mention is the unsuitability of GHC pipelines, themselves. I think everyone agrees that we do not want a 1-1 mapping of (current GitLab pipelines) to (future Woodpecker workflows). Certainly not in the fail-always state they're currently in¹.

To be clear, I believe this is already implied in the plan, and it would fall out naturally from an iterative move. But there's a lot of work and coordination hidden in there that might be worth describing.

As one small example, moving away from ci-images might not be necessary for a move to Woodpecker, but it sure would make the migration a lot less painful than it needs to be.

Footnotes

1. I would refer to my "CI Success Stats" dashboard, but it looks like it got broken by some infrastructure change on the GitLab server. ↩

[chreekat]

Suggested change

	- a full time devops paid by the Haskell Foundation, where part of the responsibilities were GitLab maintenance
	- a 1-day-a-week devops paid by the Haskell Foundation, where part of the responsibilities were GitLab maintenance

[hasufell]

I think everyone agrees that we do not want a 1-1 mapping of (current GitLab pipelines) to (future Woodpecker workflows). Certainly not in the fail-always state they're currently in.

Sure, but we have to be mindful about scope-creep here.

My idea would be to migrate a relatively small part, utilize both docker and VM backends and then see how things fall out at the end.

We can still consider this an experiment and say afterwards "oh well, maybe it won't work". So my idea would be to make this time and task bound. If we burn 1-2 person months on it and realize it's a failure... it's still not too bad and the insights could inform future efforts/decisions on possibly a different CI migration.

[hasufell]

Something I maybe have to make more clear in the document too is that I believe we have to avoid the following at all cost:

• manually provisioned VMs
• manually provisioned bare metal boxes (e.g. macOS)

A tangential discussion has come up on ghc-dev and some other issue trackers recently and I believe that we have to reconsider the role of CI as a whole: CI shouldn't be every developer's testsuite executor. It must be easy to run CI locally in any environment. And I believe the docker/libvirt approach could potentially deliver this.

Woodpecker supports local ad-hoc execution via e.g.:

woodpecker-cli exec --backend-engine docker .woodpecker/my-workflow.yaml

[MangoIV]

I really like this proposal. I think the two things I support strongest and like best are:

1. de-risking our deployments by splitting up our deployment in systems that can be maintained (and replaced) separately.
2. carefully evaluating a new system before committing to it.

The reason for (b) is that while having used woodpecker myself, I am not 100% sure it can replace our GitLab CI system although, especially with the newly added support for VMs and @hasufell's evaluation of the code base and contribution experience, I am quite confident that this could be a successful experiment and I'm certain that we should do it.

---

I also agree with @chreekat -- while gitlab CI is the reason for many CI failures (we have dozens of "runner system failures" that have no machine related reason), I think improving reliability of CI will only work if we're putting work into building a maintainable abstraction that is not hacks on top of hacks. For that reason, I also support to only port a small subset of the pipelines we are already running during the trial phase, to make sure those pipelines that do exist can be triaged for reliability.

---

What I think this proposal lacks is concrete success criteria. These ones I find particularly important:

1. "out of X pipelines, only Y failures can be attributed to the CI / test system" -- this requires appropriate monitoring to be in place.
2. there is buy in: part of why we arrived at the current situations is that there is next to no buy in -- maintenance for both the CI system and the infra that hosts it is done by very few people, mostly because code has to be closed source, the CI system is very complex and spans multiple repos, is done differently in many places, code is not shared, etc. -- if we want to make sure this does not happen again, devs have to at least be aware of how to modify pipelines and have to be willing and on-board for this, this needs communication and documentation
3. "we are able to propose a concrete architecture for building, testing and releasing GHC" -- this architecture does not need to be implemented, it just needs to be plausible that it can be implemented within the new system in reasonable time. Notably, this includes being able to do this on all supported platforms i.e. architectures / OSs.

---

In summary, I support this proposal. While there are many steps that can be done towards reducing the risk of whole system failure, I think the CI system is one of the biggest chunks. Gitlab's CI system is long-term unsustainable in that way because it's strictly coupled to Gitlab. Gitlab's internal strategy hints at changes that are going to affect us that will make Gitlab medium to long term unsustainable.

Therefore, carefully preparing a partial migration needs to be started as soon as possible.

[TeofilC]

I'm in favour of this! The key thing we need to ensure is that we don't get stuck maintaining too systems indefinitely.

I agree that local first CI is a worthy goal.

Your example in woodpecker-ci/woodpecker#6985 (comment) seems to suggest that settings about the VMs is configured as part of the in-repo CI config. Are there security concerns with this approach (and how can we alleviate them)? For instance, if I create a branch that tells the config to mount /var/secrets into the VM?

[AndreasPK]

And that's despite years of efforts from GHC team and Bryan. My view is that improving CI experience is vital for GHC's long-term sustainability and gaining new contributors.

@Bodigrim I think it's important what this would get us (forge independence) and what it wouldn't (magically "fix" CI).

There are benefits to this proposal, but also risks, most of all in terms of effort spent to get to parity, just to end up
in a similar situation.

We should be clear about the level of investment into the current setup. Bryan can say more but afaik he works very limited hours, and most of his time is spent on things unrelated to GHC Gitlab. Of the little time he spends helping with GHC Gitlab even less is spent on CI.

I can't say what the level of investment was when Ben or Matt were at the helm exactly, but I would estimate strictly CI related work was maybe oscillating around a week per month for WTs GHC team during that time? With it often being less than a week. Which did build up some debt over time that we only started to tackle more recently by increasing the effort there.

I say this because it's just not the case that there was a monumental effort with years of man hours wasted to make CI great
that went nowhere in the past. Just a long time of limited resources being spent on keeping it running and improving it here
and there.
Some of the complaints like long turnaround times will also not be solved by swapping the CI orchestrator.
They are a result of the design and architecture of GHC. And can't be solved in CI as such.

I say this because a fair evaluation has to measure up the cost of migration against other projects, including a similar effort
being spent on improvements on top of gitlab-based CI.

This should not be taken as a "we should" or "we should not" comment. But just a clarification on the state of affairs and how we got there.

---

I also want to give a shout out to cheng who stepped up and made a lot impactful improvements to CI recently. But I don't know how much time was spent on those.

[AndreasPK]

I think it would be useful if the proposal had a few more words on:

The Problem statement: While we know there are issues around Gitlab/CI it would be good to put concrete examples into it and how Woodpecker actually helps solving them. To me running woodpecker locally seems not much different than running a CI docker image locally. Which might be ignorant! But that is why it would be good for the proposal to shed light on that difference.

If there are other large projects using woodpecker, and which if any?

How Gitlab integration would look. Although I imagine much of that is in scope for the proposal itself to determine.

The proposal also mentions VMs a few times. As I understand it one of the motivations is explicitly to make it possible to run CI based on running inside virtual machines instead of docker for better isolation. The proposal should state that explicitly if this is a goal.

My recollection is that trying to enable such a workflow in Gitlab CI requires annoying workarounds. Might be worth contrasting that in more detail. As well as motivate why this is desirable to begin with.

As for separation of forge/CI orchestrator. I obviously see value there. But unless we actually move off Gitlab entirely (self-hosted or hosted) the payoff will remain hypothetical. And I imagine integration will naturally be worse when using a third party CI system, so there likely will be some ongoing friction.

Given this proposal was written by @hasufell I would also be interested in a contrast between woodpecker and github actions. Even if only as a data point for how a hypothetical post-gitlab world would look like.

[Bodigrim]

I can't say what the level of investment was when Ben or Matt were at the helm exactly, but I would estimate strictly CI related work was maybe oscillating around a week per month for WTs GHC team during that time? With it often being less than a week. Which did build up some debt over time that we only started to tackle more recently by increasing the effort there.

Perhaps we are on different scales here. I'd say that spending a week per month on strictly CI related work is already too bad, and if even this amount of effort still builds up technical debt, something is very wrong. At least in my professional experience CI work is maybe a few hours a month, if not zero.

[AndreasPK]

Perhaps we are on different scales here. I'd say that spending a week per month on strictly CI related work is already too bad, and if even this amount of effort still builds up technical debt, something is very wrong. At least in my professional experience CI work is maybe a few hours a month, if not zero.

I suppose to a large degree that depends on how you define strictly CI related. Maybe I'm not as specific as you imagine.
For example we regularly hit bugs where platform updates break CI and I would at least partially count the resulting fixes as CI related. Similarly we don't have the resources to test every GHC configuration on every platform we support pre-merge. Which means at times we have to track down bugs that break jobs that are not run pre-merge or simply don't fail reliably which I would also put into that bucket. Similarly setting up branches + CI for new major release branches, as those often add/remove platforms and toolchain versions. Which at times results in tracking down bugs in upstream toolchains or similar.

Not only do we test a large number of platforms. There is also just a lot of runs being done. At some point I looked and GHC had run over 1 Million build minutes of CI that month iirc. So I would say GHC CI is unusual just in sheer quantity alone, and then we have to consider the width of configurations we cover on top of that. I don't think it's surprising that it takes more effort to keep it running. Really CI for GHC has seen bugs I could have never imagined before working on GHC.

But if we are talking strictly about rewriting pipeline configs, updating docker images and similar it is lower than a week a month. It's not usually where the pain points are. But I don't have concrete numbers at that level of granularity.

[hasufell]

@AndreasPK indeed, there seems to be some major misunderstandings here.

1. This proposal is a bit pointless if we assume that self-hosting GitLab is a good long-term strategy
2. The rationale for separating CI from the forge is
   • better infrastructure handling: we can scale differently, have different spam/anti-bot configs for different subsystems
   • we can make faster and better decisions (migrating CI again or migrating the forge eventually are now smaller scale decisions, because they're decoupled)

Additionally, the rationale for picking woodpecker specifically is:

1. it is hackable! We need to be in control of our CI and... well... fix and change it at the core
2. it is flexible (works with all major forges)

If anyone was hoping for this proposal to fix all your CI problems, then I'm here to disappoint you. Yes, this proposal may as well be an opportunity to do CI differently.

However I do not claim that woodpecker itself will fix all our CI issues. I also do not claim that the majority of the CI issues are purely GitLab related.

The reason this proposal started is because I have effectively stopped all my GHC contributions (backporting my static linking patch, starting the rts split upstreaming process), because I believe it is a waste of time and money. I'm not sure how to make this more clear to you: as someone who was paid to work on GHC patches, I told my manager that I do not want to contribute upstream anymore. Do you understand what this means for a project like GHC? This is not some tongue-in-cheek speak or a trick to get attention. This conversation happened.

While it might certainly be interesting, I'm afraid I'm not going to entertain an archeology effort of going through all my previous contributions and trying to figure out why I had to restart CI 10 times, ping Magnus every other day and then rebase my patches for 3 months before they eventually landed in master. This type of analysis might be something GHC developers should do on a regular basis. But this proposal is about the holistic effort of eventually abandoning self-hosted GitLab and having more reliable infrastructure through separation and control.

That's why this is rather the first step in a bigger vision of unbreaking the GHC contribution experience. It is small-scoped for a reason, because otherwise we will not reach any decision anyway, as all previous discussions have revealed.

If you think GitLab is working fine (both as a forge and as a CI system), then I think we have no common basis for discussing this proposal further. This is not a "woodpecker is better than GitLab CI on all counts" proposal. If there was a self-contained fork of the CI-part of GitLab, then that would very well be an alternative choice as well. But as far as I can tell, there isn't.

[Ericson2314]

My view is that there is little point talking about GHC's CI without also talking about GHC's build system.

Hadrian needs to go, and it should either be Nix, or or Bazel, featuring per-module incremental builds, and supporting macOS Linux, and Windows. Per-module incremental builds don't help as much when bootstrapping, but I also think we should be able to run far more tests after a mere stage-1 build. Together, those things will massively improve GHC productivity, and also determine what CI runners strategy we go with.

For the Nix side:

• I am quite close to having a demonstration with Nix (Windows building just landed in beta form
• Per-module builds work with https://github.com/obsidiansystems/sandstone/
• I am about to open a PR in Nixpkgs showing that ditching Hadrian is easy.

For the Bazel side, someone that has worked on the Mercury/Tweag projects can comment bather than I.

Either of these options will get us "industrial-strength" incremental builds, but if we don't do this and just switch runners IMO we will be putting lipstick on a pig --- now matter how robust the runners are, the build latency will still be unacceptable.

[hasufell]

@Ericson2314 while I agree that hadrian needs to go, I don't agree with your conclusion. GHC should be built via cabal.

But I'm afraid this is out of scope for this proposal.

I can't even send patches for hadrian, because most of the time, GitLab is dysfunctional (infrastructure wise even).

So I want to focus on that part with the smallest step possible. And I believe that step is separating the CI from the forge.

[Ericson2314]

We don't need to move slowly here. We can redo the build system and the CI at once. Two step will confine us to "dumb builders" of arbitrary executables which are not good CIs.

It is only the very assumption of low productivity that this proposal seeks to fix that makes us think we cannot do everything at once.

When I say Nix/Bazel, the point is that neither of these contain GHC-specific logic: both work from arbitrary cabal plans. I do strongly agree that we should not have a GHC-specific build system.

The problem is just that "raw cabal" is also not right for an industrial-scale CI/build farm/caching. Cabal's own caching is too janky and unreliable for that purpose. The right division of labor is to use Cabal for planning, and use one of these other tools to build and execute the build graph.

We can do plain cabal from-scratch in a nightly job, but per-PR CI should not be blocked on such slowness.

[Bodigrim]

@Ericson2314 you can raise your own proposal if you want to redo everything at once. I don't think it's productive to extend the scope that far.

[Ericson2314]

Sounds good.

[AndreasPK]

There could always be more to say, and I think the text itself could do better in pointing out some of the benefits.

But it's clear that as an experiment it will have value if funded. We won't know if that means a migration would be a good idea until after the experiment is done. If the system is as good as advertised then that will be a good motivation to migrate and the proposal will give us a headstart on such an effort. If it turns out to have more issues than expected I assume many of the lessons learned can still be applied to upstream GHC CI without a migration. So I think this is a good use of HF funds.

[hasufell]

I guess it's a fair ask to be more clear about why woodpecker is worth our time at all.

1. it's a fork of the originally popular open source Drone CI system after that went proprietary
2. it is supported by codeberg as a hosted service and there's a plethora of pipeline examples... I guess one could scan through codeberg to find real-world examples of users
3. however the biggest user I could find is OpenCloud on github with 5.9k stars
   • the CI config is massive
   • also interesting is that they seem use starlark to define their pipelines and then (I suppose) use the configuration extension mechanism to dynamically turn it into woodpecker pipelines... this is yet another example of the lean, but powerful unix-like architecture that excited me about woodpecker in the first place

You can also look at their own marketing docs, which seem to somewhat undersell it though: https://woodpecker-ci.org/awesome

But I have to repeat myself: the way I arrived at woodpecker was not "let me find the best CI system out there that could replace GitLab CI with minimal effort". The way I arrived at woodpecker is to nail down dedicated open source CI systems and compare them with the list of requirements (see the proposal text). Then I went on looking at their ecosystem, architecture and code. Then the only two that remained were buildbot and woodpecker.

By all means, if anyone knows something that blows woodpecker out of the water... I'm not married to this experiment.

We have to agree that one of the main benefits is the separation from the forge. Is this a worthwhile goal to everyone? It was my impression that at least @MangoIV was on board with this premise.

The other benefits have all been mentioned already: hackability (I proved it), extensibility and unix architecture, clean core, active community.

I do want to excite GHC developers about woodpecker, but I'm afraid it's gonna be a bit of a chicken and egg problem without being able to really test the waters. If I do such experiments in ghcup or stable-haskell, I don't think it would carry much spice. The real test is tackling GHC.

[Ericson2314]

By all means, if anyone knows something that blows woodpecker out of the water... I'm not married to this experiment.

Appreciate it! :)

We have to agree that one of the main benefits is the separation from the forge.

To be clear, I am definitely on board with that goal. There is no "nixforge" or "bazelforge" nor should there be, so that squarely dovetails with my goals.