fix[next]: stop KeyError from masking compiled-program failures

[havogt]

Description

Draft — opening early to discuss the trade-offs below.

When a compiled program cannot be obtained, the traceback leads with a KeyError on the
dispatch dict even though the actual cause is something else. Real incident (ICON/icon4py,
10 MPI ranks, shared OTF cache on Lustre scratch):

ERROR - A Python error occurred: [Errno 116] Stale file handle
Traceback (most recent call last):
  File ".../gt4py/next/otf/compiled_program.py", line 419, in __call__
    compiled_program = self.compiled_programs[key]
KeyError: ((np.int32(557), np.int32(72619), np.int32(0), np.int32(80)), -5810085652480803948, None)
During handling of the above exception, another exception occurred:
...

The genuine cause was OSError: [Errno 116] Stale file handle while loading the artifact, but
the KeyError with its opaque key tuple is what everyone reads first — it looks like a
cache-key/dispatch bug, and it is not. That cost hours of misdirected debugging.

CompiledProgramsPool.__call__ did all cache-miss handling inside the except KeyError
handler, so everything raised there — future.result() re-raising a worker exception,
artifact.load() doing I/O — was chained to the KeyError.

Changes

• The dispatch lookup is now self.compiled_programs.get(key) plus an if ... is None. No
  exception is raised on a miss, so there is no context for the real failure to be chained to.
  The miss branch itself is unchanged.
• _load_artifact wraps artifact.load() failures as
  RuntimeError("Failed to load the compiled program '<name>'.") from <cause>, so the OSError
  is what the reader sees. Worker exceptions from future.result() are deliberately not
  wrapped — they propagate as themselves.
• The genuine-miss error now names the program and the static arguments
  (... of 'diffusion_run': scalar_int=3) and notes that a variant is also selected by the
  identity of the offset_provider entries.

Hot path

__call__ runs per program invocation (thousands per second in ICON), so the cost of the
successful lookup matters. Isolated dict lookup, key present:

	3.10	3.12	3.13
try: d[key] except KeyError:	30.0 ns	15.7 ns	16.1 ns
d.get(key)	30.6 ns	21.7 ns	20.1 ns
key in d + d[key]	43.1 ns	—	—

So .get() costs ~5 ns on ≥3.11, where the non-raising try became free, and nothing on 3.10.
That is 0.2 % of the ~2800 ns a full dispatch takes; a whole-dispatch microbenchmark (program
call with the executable replaced by a no-op, under -O) cannot resolve it above run-to-run
noise. Taken deliberately in exchange for a much smaller, non-convoluted diff
(review thread).

An earlier revision kept the try/except and restructured __call__ so the handler was left
before any miss handling (identical hit-path bytecode, zero cost). That version is in the
history of this branch if we ever want the ns back.

Note that just adding raise ... from e at the raise sites does not fix this: from e sets
__suppress_context__ on the outer exception, but the intermediate OSError still carries
__context__ = KeyError and is printed as the cause, so the KeyError still leads the
traceback. Hiding it that way would require setting inner.__context__ = None at every site.

Open for discussion

• Naming the static arguments in the miss error is ~12 of the ~30 net-new lines and is a
  message improvement, not part of the fix. Drop it?
• A dedicated exception type (CompiledProgramNotAvailableError) carrying the key and the cause
  kind was considered and skipped — nothing catches these today.
• _finish_compilation_job still has assert key not in self.compiled_programs, which never
  runs under icon4py's PYTHONOPTIMIZE=2. Left as an assert: it is only reachable right after a
  confirmed dict miss, so it cannot be violated by user input.

Out of scope: the underlying Lustre ESTALE, which belongs with the OTF cache write/replace
logic (#2691). This is only about which error the user is shown.

Requirements

• All fixes and/or new features come with corresponding tests.
  Three unit tests in tests/next_tests/unit_tests/otf_tests/test_compiled_program.py
  (load OSError, worker exception, genuine miss); all three fail on main and pass here.
• Important design decisions have been documented in the appropriate ADR.

[egparedes]

I would go further in the simplification of the code here

[egparedes]

It would be even better to get rid completely of the try-except. I don't think the minimal saving in python overhead is worth the convoluted code anymore:

Suggested change

	try:
	compiled_program = self.compiled_programs[key]
	except KeyError:
	# The handler is empty on purpose: everything that could raise while it is active
	# would be chained to this `KeyError`, whose opaque key then dominates the traceback
	# and hides the actual failure. Leaving the handler clears the exception context.
	pass
	else:
	with compiled_program_call_context(self, key, args, kwargs, offset_provider):
	compiled_program(*args, **kwargs, offset_provider=offset_provider)
	return
	compiled_program = self.compiled_programs.get(key)
	if compiled_program is None:
	self._dispatch_miss(
	key, args, kwargs, canonical_args, canonical_kwargs, offset_provider, enable_jit
	)
	with compiled_program_call_context(self, key, args, kwargs, offset_provider):
	compiled_program(*args, **kwargs, offset_provider=offset_provider)

[havogt]

Done in 95b66d3 — agreed, the ns aren't worth the convoluted control flow.

One correction to the snippet: as written it falls through to compiled_program(...) with compiled_program still None after _dispatch_miss, because that helper re-dispatched through self(...) and returned. But that's moot now — without a KeyError handler there is no exception context to escape, so _dispatch_miss is gone entirely and the miss branch went straight back inline where it was. The diff against main is now just the .get() + if, the error-message change, and _load_artifact.

Measured cost of the switch (isolated dict lookup, key present):

	3.10	3.12	3.13
try: d[key] except KeyError:	30.0 ns	15.7 ns	16.1 ns
d.get(key)	30.6 ns	21.7 ns	20.1 ns

~5 ns on >=3.11 (free on 3.10), against ~2800 ns for a full dispatch.

[havogt]

Superseded by stack #2735: #2733 (the fix) and #2734 (the error-message improvement), split per @egparedes' review and so the message part can be dropped independently. The .get() simplification from this branch is in #2733.