interface_tester discover does NOT verify charmlibs-layout interfaces — doc is incorrect - #67
Open
github-actions[bot] wants to merge 1 commit into
Open
interface_tester discover does NOT verify charmlibs-layout interfaces — doc is incorrect#67github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The documentation claim
canonical/operator PR #2710 (the "How to manage
interfaces" page) tells charm developers that after creating an interface specification in the
charmlibsmonorepo they can verify the setup with:run from the
charmlibsroot, and that this will discover the interface's schema and tests(displaying
schema OKand listing test cases, or<no tests>only when tests have not beenwritten yet).
My understanding (independent of the doc)
I read
pytest-interface-tester3.4.1 (the latest, confirmed from itspyproject.toml).The
interface_tester discovercommand callsinterface_tester.cli.discover._pprint_tests, whichcalls
interface_tester.collector.collect_tests. The discovery contract is:collect_testsiterates(root / "interfaces").glob(include)— findsinterfaces/<name>/._gather_tests_for_interfacethen doesfor version_dir in interface_dir.glob("v*"):— i.e. itlooks for version directories directly under
interfaces/<name>/._gather_test_cases_for_versionlooks for tests underversion_dir / tests_dirwhere_DEFAULT_TESTS_DIR = "interface_tests"(hardcoded default; thediscoverCLI exposes no flagto override it).
pytest-interface-testerwas built for the oldcharm-relation-interfaceslayout,interfaces/<name>/v<N>/interface_tests/. Thecharmlibslayout is different: it nests versiondirectories under an extra
interface/segment and calls the tests directorytests/:So with the charmlibs layout:
interface_dir.glob("v*")matches nothing — the only child ofinterfaces/<name>/isinterface/, which does not start withv. The versions loop never executes, socollect_testsreturns{"<name>": {}}(an empty versions dict).tests_dir="interface_tests"would not matchtests/._pprint_testsprints"<name>: <no tests>"whenever the versions dict is empty. Thereforeinterface_tester discover --include <name>prints<no tests>for the charmlibs layout evenwhen real schema and tests exist at
interfaces/<name>/interface/v<N>/tests/. The documentedverification step does not work.
I expect this to hold, so the test asserts it (not the doc). If CI passes, the doc is incorrect.
Why the real package is not installed
pytest-interface-tester3.4.1 depends onops-scenario>=7.0.1, and every releasedops-scenario(7.0.1–7.0.5) pinsops~=2.15. This charm requiresops~=3.7. No singleopsversion satisfies both
~=2.15and~=3.7, sopytest-interface-testercannot be installed nextto this charm (
uv lockfails on theopsconflict). The package'sinterface_tester/__init__.pyalso imports
scenariotransitively, so even importinginterface_tester.collectoris unreachableunder
ops 3.x.Instead, the test (
kepler/tests/unit/test_interface_tester_discover.py) reproduces thediscovery contract from
interface_tester/collector.pyat tag 3.4.1 — the twopathlib.Path.globcalls and the
tests_dirdefault are copied verbatim, and thetest_provider/test_requirermodule-import logic is reproduced. Only the schema/charm loading that runs after a version
directory is found is omitted (it never runs for the charmlibs case, since no version directory is
found). The version the claim is about (3.4.1) is fixed by reproducing that tag's source.
The test
test_charmlibs_layout_yields_no_discovered_testsbuilds a charmlibs-layout tree(
interfaces/my_fancy_database/interface/v1/tests/test_provider.pyandtest_requirer.py,i.e. real tests at the charmlibs paths) and asserts
collect_tests(root, "my_fancy_database")returns
{"my_fancy_database": {}}— no version directories discovered, sointerface_tester discoverwould printmy_fancy_database: <no tests>.test_legacy_layout_is_discoveredbuilds the legacycharm-relation-interfaces-layout tree (interfaces/my_fancy_database/v1/interface_tests/...)and asserts the
v1provider tests are discovered. This is a control proving the discoverylogic is not simply broken — it works for the layout the tool was designed for and fails only for
the charmlibs layout, isolating the cause to the directory nesting.
Both are plain unit tests (no Juju controller, no subprocess);
tox -e unitruns them.What the CI result means
run_tox -e format,lint,unitforkeplerpasses (3 unit tests green, 0 pyright/codespell/rufferrors).
interface_tester discover --include <name>does not verify charmlibs-layout interfaces — itsilently reports
<no tests>regardless of whether the schema and tests are correct, because thecollector looks for
interfaces/<name>/v*andinterface_tests/while charmlibs usesinterfaces/<name>/interface/v*andtests/.