Skip to content

interface_tester discover does NOT verify charmlibs-layout interfaces — doc is incorrect - #67

Open
github-actions[bot] wants to merge 1 commit into
mainfrom
probe/issue-66
Open

interface_tester discover does NOT verify charmlibs-layout interfaces — doc is incorrect#67
github-actions[bot] wants to merge 1 commit into
mainfrom
probe/issue-66

Conversation

@github-actions

Copy link
Copy Markdown

The documentation claim

canonical/operator PR #2710 (the "How to manage
interfaces" page) tells charm developers that after creating an interface specification in the
charmlibs monorepo they can verify the setup with:

pip install pytest-interface-tester
interface_tester discover --include my_fancy_database

run from the charmlibs root, and that this will discover the interface's schema and tests
(displaying schema OK and listing test cases, or <no tests> only when tests have not been
written yet).

My understanding (independent of the doc)

I read pytest-interface-tester 3.4.1 (the latest, confirmed from its pyproject.toml).
The interface_tester discover command calls interface_tester.cli.discover._pprint_tests, which
calls interface_tester.collector.collect_tests. The discovery contract is:

  1. collect_tests iterates (root / "interfaces").glob(include) — finds interfaces/<name>/.
  2. _gather_tests_for_interface then does for version_dir in interface_dir.glob("v*"): — i.e. it
    looks for version directories directly under interfaces/<name>/.
  3. _gather_test_cases_for_version looks for tests under version_dir / tests_dir where
    _DEFAULT_TESTS_DIR = "interface_tests" (hardcoded default; the discover CLI exposes no flag
    to override it).

pytest-interface-tester was built for the old charm-relation-interfaces layout,
interfaces/<name>/v<N>/interface_tests/. The charmlibs layout is different: it nests version
directories under an extra interface/ segment and calls the tests directory tests/:

interfaces/<name>/interface/v<N>/tests/

So with the charmlibs layout:

  • interface_dir.glob("v*") matches nothing — the only child of interfaces/<name>/ is
    interface/, which does not start with v. The versions loop never executes, so
    collect_tests returns {"<name>": {}} (an empty versions dict).
  • Even if a version dir were found, tests_dir="interface_tests" would not match tests/.

_pprint_tests prints "<name>: <no tests>" whenever the versions dict is empty. Therefore
interface_tester discover --include <name> prints <no tests> for the charmlibs layout even
when real schema and tests exist
at interfaces/<name>/interface/v<N>/tests/. The documented
verification step does not work.

I expect this to hold, so the test asserts it (not the doc). If CI passes, the doc is incorrect.

Why the real package is not installed

pytest-interface-tester 3.4.1 depends on ops-scenario>=7.0.1, and every released
ops-scenario (7.0.1–7.0.5) pins ops~=2.15. This charm requires ops~=3.7. No single ops
version satisfies both ~=2.15 and ~=3.7, so pytest-interface-tester cannot be installed next
to this charm (uv lock fails on the ops conflict). The package's interface_tester/__init__.py
also imports scenario transitively, so even importing interface_tester.collector is unreachable
under ops 3.x.

Instead, the test (kepler/tests/unit/test_interface_tester_discover.py) reproduces the
discovery contract from interface_tester/collector.py at tag 3.4.1 — the two pathlib.Path.glob
calls and the tests_dir default are copied verbatim, and the test_provider/test_requirer
module-import logic is reproduced. Only the schema/charm loading that runs after a version
directory is found is omitted (it never runs for the charmlibs case, since no version directory is
found). The version the claim is about (3.4.1) is fixed by reproducing that tag's source.

The test

  • test_charmlibs_layout_yields_no_discovered_tests builds a charmlibs-layout tree
    (interfaces/my_fancy_database/interface/v1/tests/test_provider.py and test_requirer.py,
    i.e. real tests at the charmlibs paths) and asserts collect_tests(root, "my_fancy_database")
    returns {"my_fancy_database": {}} — no version directories discovered, so
    interface_tester discover would print my_fancy_database: <no tests>.
  • test_legacy_layout_is_discovered builds the legacy
    charm-relation-interfaces-layout tree (interfaces/my_fancy_database/v1/interface_tests/...)
    and asserts the v1 provider tests are discovered. This is a control proving the discovery
    logic is not simply broken — it works for the layout the tool was designed for and fails only for
    the charmlibs layout, isolating the cause to the directory nesting.

Both are plain unit tests (no Juju controller, no subprocess); tox -e unit runs them.

What the CI result means

  • run_tox -e format,lint,unit for kepler passes (3 unit tests green, 0 pyright/codespell/ruff
    errors).
  • CI runs the same unit tests. If they pass, the documentation claim is incorrect:
    interface_tester discover --include <name> does not verify charmlibs-layout interfaces — it
    silently reports <no tests> regardless of whether the schema and tests are correct, because the
    collector looks for interfaces/<name>/v* and interface_tests/ while charmlibs uses
    interfaces/<name>/interface/v* and tests/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants