Upgrading from 11.0.0 to 11.1.1 causes a big increase in time for message creation

[rickpastoor]

👓 What did you see?

My tests consist of 1584 scenarios and 18147 steps. On 11.0.0 this takes roughly 3 mins. Since the bump, it takes at least 4.5mins.

AI did a bit of digging and profiling, and it pinned it down to the following change in the MessageBuilder. I've included it, it might be a helpful pointer but obvious you folks know best what the reason for that change was.

---

Cucumber 11.1.x rewrote Formatter::MessageBuilder#on_test_step_started / #on_test_step_finished. What was an O(1) hash lookup:

test_case = @test_case_by_step_id[event.test_step.id]

became a full scan of every test case accumulated so far, checking every step of each:

@repository.test_case_by_id.values
           .detect { |tc| tc.test_steps.any? { |s| s.id == event.test_step.id } }

That runs twice per step, and the repository grows as the run progresses, so the cost is quadratic in suite size.

✅ What did you expect to see?

Upgrading should not impact the test run in such a big way.

📦 Which tool/library version are you using?

No response

🔬 How could we reproduce it?

No response

📚 Any additional context?

No response

[luke-hill]

The 5second answer is the migration from 11 -> now is migrating away from the "fake" queries to the new queries. However if there is a noticeable degradation, we may need to rethink part of this.

I'm unlikely to have much of an answer for a few weeks, but thanks for bringing this to my attention. I will dive deep into this.

[rickpastoor]

@luke-hill thanks Luke. If there's anything I can do to help, let me know!

[luke-hill]

For now if you want to dive into anything and/or contribute - then any help is welcomed.

For now, the next plan is to cut a v12 which includes query, this will require a core v18.1/v19. I hope to get this done before the end of the year.

[luke-hill]

In terms of here. Around cucumber v7-v9 there was a bunch of "fake" query objects added. I say "fake" because inherently they were used in the "building" phase.

The "true" or "other" query, is the one that is used in the post-phase, i.e. "querying"

I'm trying to remove most of the fake queries and use the true query/repository to build up all messages. Simultaneously I'm also working towards migrating as many of the generating items OUT of MessageBuilder into the relevant appropriate areas.

There is a doc here: https://github.com/cucumber/cucumber-ruby/blob/main/docs/adrs/0001-slim-down-the-messagebuilder.md

EDIT: (In other words), this change was done because I'm trying to remove those iVars (Such as @test_case_by_step_id), and want to remove even more of them. But I also want to move the methods out of the MessageBuilder proper. So this might be a casualty of me doing refactoring in an incremental way rather than a "correct" way - i.e. I should just move the methods out

Examples of envelope generation migration (Slimming down MessageBuilder)

• GherkinSourceRead
• UndefinedParameterType
• OnTestRunHookStarted & OnTestRunHookFinished

[rickpastoor]

Ok cool, thanks for those pointers! Can't promise anything but guarding the performance of my tests is high on my list.

Have you considered adding a few tests that could help lock in performance and/or help easily benchmark so we can monitor regressions?

[luke-hill]

As mentioned, we have a plan to massively overhaul the internal structure of these things. If things were incidentally high or low performance wise this isn't something that can be overlooked in the favour of massively overhauling the way in which maintainability is.

Given our time is limited, this is something that just wouldn't ever really "get done". And actually unpicking lots of the things we're unpicking atm are things that should have been done years ago, but always got deprioritised. So we're making those choices now.

Once the MessageBuilder class is gone (Which alone is going to be potentially a year maybe), there is already the next 2 steps, to then unpick the event protocol in favour of a message protocol. Whilst this is going on, we're also rewriting all of our formatters in the message format instead of a bespoke format for each one (If you check the formatters directory, you'll see thousands of lines of code for bespoke formatters, that don't use the message protocol anywhere).

[luke-hill]

Hi @rickpastoor I've just realised something (Albeit incidentally).

Cucumber v12 (Which is the next release, probably in a month or two), will generate these 2 messages in a completely different way (Ironically, in a new location in a new repo).

As such I'd be intrigued to know if you have as much performance degradation with that.

Being open, v12 will also change a lot about the internals of cucumber, as well as entirely removing strict - which I hope you've already begun to remove (If you used it at all).

TL;DR - Try cucumber v12 in a month or so, let us know how you get on...