Upgrade Jint to 4.15.3 and adopt its cache-gate and lazy-value APIs

[lahma]

Upgrades Jint from 4.8.0 to 4.15.3. There are two independent parts:

1. The version bump — one line in a .csproj. No source change needed, everything builds and all scripting tests pass unmodified. Scripts get 2–39% faster on their own, depending on what they do.
2. Six small changes that tell the new Jint what our scripting integration actually needs. Jint has to be careful when we do not tell it, and we have been paying for that for years. This part adds another 35–50% on the scenarios it aims at.

Scripts behave exactly the same in both parts — this PR is only about speed.

Sorry for the wall of engine jargon in the first version of this description. This one tries again.

The terms I keep using

They come from how JavaScript engines are built (V8, SpiderMonkey, and by now Jint too):

Inline cache. When a script does ctx.data.title.iv, the engine remembers at that exact spot in the script where it found title last time, and reuses that on the next run instead of searching again. It can only do that when nothing is able to change the answer behind its back. If we register a hook and do not tell Jint what the hook is for, Jint has to assume the worst and switches these caches off — for the whole engine, not just for the objects we hooked. Two of our registrations did exactly that.

Shape (V8 calls it a hidden class). Objects that are created with the same keys can share one description of their layout, like a class, instead of every object carrying its own property dictionary. All content items of one schema have the same keys, so this applies to almost all JSON we push into scripts. A shared layout is cheaper to read from, and it is what makes the inline cache above possible in the first place. Objects built from a custom ObjectInstance subclass — which is how JsonMapper built them — can never take part in this.

Host. Jint's word for the application that embeds it, so: us. A host contract is a place where Jint calls into our code and then trusts the answer without verifying it, because verifying would cost as much as the shortcut saves. ContentDataObject and friends are such places.

What the six changes are

#	File	In one sentence
1	NullPropagation.cs	Tell Jint that this resolver only reacts to null/undefined, so it can keep its read caches for everything else.
2	JintObjectConverter.cs	Tell Jint which .NET types this converter handles, so it does not have to offer it every property of every object.
3	JsonMapper.cs	Build JSON objects so they share their layout, instead of giving every object its own property dictionary.
4	ContentFieldObject.cs	Answer 'iv' in field without converting the field value.
5	WritableContext.cs, JintExtensions.cs	Convert a script variable when it is first read, not when the engine is built.
6	tests only	Switch on Jint's own self-checks in the test suite, so a mistake in 1–5 fails a test instead of being silent.

Numbers

Measured on an idle machine with BenchmarkDotNet, strictly serial, three checkouts sharing an identical harness: A = master on 4.8.0, B = master with only the version line changed, C = this PR. Every scenario asserts its exact expected output before being measured, and all three arms produced the same outputs.

Scenario	A: today	B: bump only	C: this PR	overall
Plain arithmetic (control)	86.1 μs	52.6 μs	54.7 μs	−37%
Trigger script	7.6 μs	6.1 μs	6.1 μs	−19%
Content transform	16.5 μs	14.2 μs	14.8 μs	−10%
JSON.stringify(data)	11.3 μs	9.4 μs	9.2 μs	−19%
Walking a JSON object (change 3)	55.2 μs	52.4 μs	32.4 μs	−41%
Many variables, few read, ctx (change 5)	12.9 μs	12.6 μs	6.7 μs	−48%
Many variables, few read, globals (change 5)	12.6 μs	11.3 μs	5.6 μs	−55%
Many .NET property reads (changes 1+2)	108.9 μs	85.0 μs	55.0 μs	−50%

Two things worth being honest about:

• The control row moves a lot between A and B, so it is a result of the upgrade, not a measurement floor. The floor for judging part 2 is the B→C control pair (same engine, same harness): ±4%. The four rows above that clearly beat it are the ones I claim as wins; the small ones sit inside it and I do not claim they moved.
• Change 5 has a real cost on the other side: on a script with a handful of cheap variables it adds ~2–3 KB per evaluation for the deferred-conversion bookkeeping (the trigger row). It pays that back many times as soon as variables are numerous or expensive. That is the trade it is designed to make.

Allocations are down in every scenario compared to today.

The changes in detail

1. Tell Jint what NullPropagation does

SetReferencesResolver without a second argument means "this resolver might react to anything", and Jint answers by disabling its member-read caches engine-wide, for every property read in every script.

NullPropagation.TryPropertyReference returns false for everything that is not null or undefined, so for the other cases Jint asking us can never change the result. Declaring that re-enables the caches:

engineOptions.SetReferencesResolver(NullPropagation.Instance, NullPropagation.Interests);

Jint documents this list as a subscription filter and not a promise: for a case we do not subscribe to, it behaves as if no resolver was registered at all — which is what it would have done anyway.

Why not the built-in NullPropagatingReferenceResolver that 4.15 ships?

Because it is not the same behavior. It does not answer unresolvable names or non-callable members, where our resolver does, so swapping it in would turn unknownName and ctx.value.notAFunction() into errors for existing scripts. Two tests pin that difference.

One oddity found on the way and left alone: our TryUnresolvableReference passes the reference base straight through, and for an unknown name that base is an internal Jint marker string reading [[Unresolvable]], not undefined. So String(unknownName) has always been "[[Unresolvable]]" here. Changing it to undefined would be tidier but is a behavior change for existing scripts, so it is only documented and pinned by a test.

2. Tell Jint which types JintObjectConverter handles

Same shape of problem: a converter that has not declared its types might be handed anything, so Jint has to offer it every .NET property read and cannot use its compiled property reader for any of them. The converter handles a closed set of nine types, now declared as JintObjectConverter.HandledTypes and kept directly above the switch it must match.

Matching is by assignability, so typeof(IUser) still covers every implementation, and the filter errs towards claiming — the converter is still offered every value it was offered before.

The Enum case is dropped in favor of Interop.EnumConversion = EnumConversionMode.String, which Jint documents as the member name as produced by ToString() — including "ContentScript, Transform" for a [Flags] combination and the number as a string for a value with no name. Same output as the branch it replaces, tests for all three cases.

The obvious risk is that someone adds a case here and forgets the list. That is not silent: change 6 makes it fail a test by name (verified by deliberately breaking it).

3. Let JsonMapper build shared-layout objects

JsonMapper built every JSON object as a private ObjectInstance subclass that existed only to be instantiable — and such an object can never share a layout, so all of ctx.data's leaf objects and every JSON variable were permanently on the slow path.

JsObject.CreateFromEntries builds the same object the way Jint builds an object literal — same keys, same order, same flags, indistinguishable from a script — but through the layout machinery, so repeated calls with the same key sequence share one layout. That is the JSON walk row above (−38% of it from this change alone).

Jint falls back to the per-object dictionary silently when it cannot build the shared form, so JsonMapperTests asserts it actually happened (HasSharedShape) rather than assuming it.

Three unrelated small fixes in the same file: the reverse direction allocated a string per array element (a.Get(i.ToString())) where the indexer reads the array storage directly, and JsNumber.Create/JsString.Create reuse cached instances where new always allocated.

4. ContentFieldObject answers existence questions without converting

ContentFieldProperty converts the stored JsonValue on first read. Questions like 'iv' in field, hasOwnProperty, Object.keys, spread and JSON.stringify do not need the value at all, but used to pay for it, because they all went through GetOwnProperty. The new ProbeOwnProperty answers them from the property flags instead.

It is deliberately a copy of GetOwnProperty minus the value, because Jint trusts it without checking — a wrong answer would silently drop keys from every enumeration above. Change 6 is what checks it.

ContentDataObject deliberately does not get the same treatment; see below.

5. Convert script variables on first read

Every variable was converted when the engine was set up, whether the script mentioned it or not — and some are not cheap (the user variable walks and groups all claims). A typical script uses two or three of them.

Both paths now defer that, with the two APIs 4.15 added for it:

// ctx.* — WritableContext
SetOwnProperty(key, PropertyDescriptor.CreateLazy(...));

// globals — JintExtensions
engine.Advanced.AddLazyGlobal(key, e => MapVariable(e, item));

The property itself is added immediately in both cases, so key order, Object.keys, in, getOwnPropertyNames, delete and the write-through to ScriptVars are unchanged — only the conversion waits. MapVariable reproduces what Engine.SetValue does, including its special case for a Type, so a deferred variable cannot look different from an eager one. Six tests use a ClaimsPrincipal that counts how often its claims are read to prove all three properties: not converted when unread, still visible in enumeration, converted on first read.

One edge case worth recording

Engine.SetValue assigns through the normal write path while AddLazyGlobal installs the property directly, so a variable named exactly like a non-writable built-in (undefined, NaN, Infinity) would now shadow it where it was silently ignored before. ScriptVars keys are domain names, so this cannot happen in practice.

6. Jint's self-checks run in the test suite

Changes 1, 2 and 4 all have the same failure mode: we tell Jint something, Jint believes us without checking, and if we are wrong nothing throws — a key just disappears from Object.keys, or a type stops being converted.

4.15 can turn those checks on in the shipped release package through an AppContext switch (before, this needed a source build of Jint in Debug). A module initializer in the test assembly sets it, so every CI run verifies our extension points against the same NuGet package production uses, and a violation is an ordinary test failure. It stays off in production, where the checks would only cost performance.

Both checks were confirmed by deliberately breaking them first:

• a wrong ProbeOwnProperty fails with "ContentFieldObject.ProbeOwnProperty answered 'iv' with Missing but its GetOwnProperty reports Enumerable";
• an undeclared converter type fails with "JintObjectConverter converted a value of type System.Version, which is not among the types it was registered as handling".

With both correct, all 1279 tests in Squidex.Domain.Apps.Core.Tests pass with the checks on.

What I deliberately did not do

• Engine pooling. This is the biggest remaining win — an engine is built per evaluation, and Jint's per-script caches only pay off from the second evaluation on the same engine. It is not safe today, and the reason is on our side, not Jint's: ScriptExecutionContext's scheduler re-enters the engine from task continuations under lock (Engine), guarded only by IsCompleted, and on timeout WaitForCompletionAsync returns while the TaskCompletionSource is neither completed nor faulted — so a late scheduler.Run can re-enter the engine afterwards. Today that engine is garbage and nobody notices. Pooled, it would write one evaluation's data into the next one's variables. That needs a kill flag in the scheduler first, and is its own PR.
• ContentDataObject.GetOwnProperty creating a field for any name it is asked about. Pre-existing and visible to scripts (data.hasOwnProperty reads as a field rather than the function, and asking about a name makes it show up in Object.keys). Not this PR's business — but it is also the reason ContentDataObject gets no ProbeOwnProperty in change 4: to stay consistent with that GetOwnProperty, the probe would have to create phantom fields too.
• Options.AddImmutableCrossing (new in 4.15, lets Jint cache reads of .NET objects a host promises will not change while scripts can see them). Does not fit: ScriptVars is written through by design and AssetMetadata is mutated by scripts on purpose, there is a test for it. It is also an options-time setting, and the types that would be candidates live in Squidex.Domain.Apps.Entities, which the engine setup cannot reference. A wrong promise here means stale reads, so this is not a place to guess.
• Removing SetTypeConverter. It is needed (extension delegates take JsonValue parameters). Worth recording that it costs something: with a custom type converter installed, Jint excludes indexer accessors from its shared cross-engine cache. If that ever shows up in a profile, the fix is a shared TypeResolver for all Squidex engines, not removing the converter.

Tests

backend/tests/Squidex.Domain.Apps.Core.Tests/Operations/Scripting/ goes from 128 to 164 tests.

Everything behavior-adjacent was pinned before the change and run green on the old code first: the first commit adds 28 tests and passes on Jint 4.8.0, which is bisectable. They cover undeclared names, null-propagating chains, calls on non-callable members, ordinary reads and calls, enum conversion including [Flags], JSON object key order / stringify / for..in / mutation / round trip, content fields under in / hasOwnProperty / propertyIsEnumerable / Object.keys / spread / stringify (also after a delete and an add), and the context object's enumeration, typeof, write-through and delete.

The remaining 8 arrive with the features they describe and are red before the change, green after.

Verification runs

Run with the same commands CI uses (Dockerfile: dotnet test --filter "Category!=Dependencies & Category!=TestContainer" --configuration Release), so Meziantou.Analyzer and StyleCop ran in-build. One StyleCop diagnostic was introduced and fixed properly (SA1203); no suppressions added.

• Full backend solution build in Release: succeeds. The 35 warnings are all pre-existing and in untouched projects.
• Squidex.Domain.Apps.Core.Tests 1279/1279, including all 164 scripting tests, with Jint's self-checks on.
• Squidex.Infrastructure.Tests 1073/1073 · Squidex.Domain.Users.Tests 57/57 · Squidex.Web.Tests 167/167.
• Squidex.Domain.Apps.Entities.Tests 1524/1526 — the two failures are FFMpegAssetMetadataSourceTests, which need the ffmpeg binary that the CI image has and my machine does not. They touch no scripting code.

The first eight commits are logical units and each builds and passes on its own. The ninth rewrites the code comments in plainer language after review feedback and changes nothing else — it is deliberately a separate commit rather than folded into the others, so that a review already in progress stays valid.

Compatibility notes

Everything Squidex touches was verified present and unchanged across 4.8 → 4.15: ObjectInstance.Extensible is still public virtual; GetOwnProperty, Set, DefineOwnProperty, RemoveOwnProperty, GetOwnProperties, GetOwnPropertyKeys signatures unchanged; PropertyDescriptor(PropertyFlag) and protected virtual CustomValue unchanged, so CustomProperty and ContentFieldProperty behave as before; Engine.Constraints.Reset(), Options.Constraints.PromiseTimeout, AllowClrWrite, EvaluateAsync(in Prepared<Script>, CancellationToken), CancellationToken(ct), ObjectWrapper.Create, JsDate(engine, DateTime), JsValue.FromObject all present; the JsArray/JsString pattern matches in JintExtensions.ToIds and ScriptOperations.Reject are unaffected.

Transitive dependency: Acornima moves to 1.6.2. ParseErrorException, ScriptPreparationException and JintException, which JintScriptEngine.MapException switches on, all still exist.

Constraints need no change. TimeoutInterval and CancellationToken are the kind that keeps Jint's interpreter fast path armed.

Package provenance. After restore, .nupkg.metadata reports https://api.nuget.org/v3/index.json and the nuspec repository commit is a304aa5dacd340e2a5ff51e1ea0c465e38e50aa8, the v4.15.3 tag.

Also unchanged and already right: Prepared<Script> is cached via CacheParser and shared across engines (Jint documents that as safe), and Strict() is on.

The branch is still named 4.15.2 because renaming it would close this PR; the pinned version is 4.15.3.

[SebastianStehle]

No idea what this means.

[lahma]

It's just a simpler way to index array adirectly without creating a temporary string to create numeric index which Jint then has to transform to number again.

[SebastianStehle]

Thanks a lot, I have no idea what most of the comments actually mean, so I have to go over the PR manually:

I think the whole PR is really digging a lot into the irrelevant details of Jint and is hard to understand.

// A shared shape is what keeps a script reading a batch of content items monomorphic. It is a
        // performance property and never a correctness one, but it is silent when it regresses: building
        // these objects as a host ObjectInstance subclass again would put them back in the per-object

OR

    /// <summary>
    /// The situations this resolver actually answers, declared so the engine keeps the fast paths for
    /// everything else.
    /// </summary>
    /// <remarks>
    /// Deliberately omitted are <see cref="ReferenceResolverInterests.ObjectPropertyBase"/> and
    /// <see cref="ReferenceResolverInterests.PrimitivePropertyBase"/>, the pair that disables the
    /// non-computed member-read inline caches, the dense-array indexed-read lane and the member-call callee
    /// lane engine-wide. <see cref="TryPropertyReference"/> declines every base that is not null or
    /// undefined, so those are situations where the engine consulting this resolver could never change the
    /// result. Interests are a subscription filter and not a promise: a situation not subscribed to behaves
    /// exactly as if no resolver were registered.
    /// </remarks>

[lahma]

The shapes feature is the "hidden shapes", so basically when we know the form of objects we don't need to create expensive and slower dictionaries with property descriptors. If you just want the basic speed improvement, updating the library will do. It's the B→C part that is the interop optimization with Jint's new features.

[SebastianStehle]

It is really awesome, what you have achieved with AI. The number of PRs in jint is great. I just wish that the agent would create "normal" comments when creating PRs outside of Jint. For me it seems it uses the internal domain language and comments in a way that nobody really understands. Could you instruct the agent to write the comments in a more condensed and easy to understand form?

For example

                // Deferred instead of Engine.SetValue, which maps every variable now. The global itself is
                // installed eagerly, so existence checks and enumeration see the name without materializing
                // anything; only the mapping waits for the first read of the value.
                engine.Advanced.AddLazyGlobal(key, e => MapVariable(e, item));
                ```
                
                This is basically a one liner
                

// Sets a value, but runs the conversion only when the value is used for the first time

[lahma]

Fair point, and thanks for saying it directly — I rewrote both.

Comments are one or two lines each now (54871db3f, comments only). Your example was the target:

// Sets the value, but runs the conversion only when the script reads it for the first time.
// The name is added right away, so enumeration and "in" checks work as before.
engine.Advanced.AddLazyGlobal(key, e => MapVariable(e, item));

The rule I applied: say what it does and why it is worth it, and name a Jint concept only where you would have to look it up anyway — then explain it right there.

I kept it as a separate commit instead of folding it into the other eight, so that whatever you already reviewed manually stays valid.

Description now leads with the two independent parts (the version bump, which needs no code change, and the six adoption changes), then a table of what each change does in one sentence. There is a short section up front for the three terms that are hard to avoid — inline cache, shape, host — with an example from our code for each; they come from how JS engines are built (V8, SpiderMonkey), not from anything Squidex-specific. All the deep justification, edge cases and the compatibility checklist are in collapsed sections now.

No code changed, and the numbers are the same.