Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions architecture/supported-subset.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ script runs somewhere else, later, where that precondition may hold.

`tests/conformance/` enforces the hard direction mechanically rather than by
a hand-maintained table: it runs both `docker compose config` and the real
`_read_compose → resolve_extends → validate → emit_script` pipeline over a
`read → resolve_extends → validate → emit_script` pipeline over a
matrix generated from every key in `SERVICE_KEYS | STRUCTURAL_KEYS |
IGNORED_SERVICE_KEYS` crossed with a set of hostile shapes, plus a
hand-authored corpus (`tests/conformance/corpus/`) for what a single-key
Expand Down Expand Up @@ -119,8 +119,8 @@ Rejecting a non-string key **matches Docker**, which refuses one too
What Docker does *not* see is a bare `on:` / `off:` / `yes:` / `no:` as a
non-string key, because it parses **YAML 1.2**, where each of those is an
ordinary string. PyYAML implements YAML **1.1**, where each is a *boolean* —
so the CLI loads YAML with a 1.2-style boolean resolver (`_build_yaml_loader`,
`compose2pod/cli.py`), and only `true`/`false` resolve as booleans. Without it,
so `read()` loads YAML with a 1.2-style boolean resolver (`_build_yaml_loader`,
`compose2pod/read.py`), and only `true`/`false` resolve as booleans. Without it,
`environment: {on: 1}` would arrive as the key `True` and be refused — a file
Docker runs — and the *value* `SSL: on` would reach the container as `SSL=true`
rather than `SSL=on`. The spelling cannot be recovered downstream: once the
Expand Down
91 changes: 3 additions & 88 deletions compose2pod/cli.py
Original file line number Diff line number Diff line change
@@ -1,99 +1,14 @@
"""Command-line interface: read a compose document and emit the pod script."""

import argparse
import contextlib
import json
import re
import sys
from pathlib import Path
from types import ModuleType
from typing import Any

from compose2pod.emit import POD_NAME_PATTERN, EmitOptions, emit_script, referenced_variables
from compose2pod.exceptions import UnsupportedComposeError
from compose2pod.extends import resolve_extends
from compose2pod.parsing import validate


_yaml: ModuleType | None = None
with contextlib.suppress(ImportError): # the optional [yaml] extra is not installed
import yaml as _yaml


# YAML 1.2's boolean set: `true`/`false` only. PyYAML implements YAML *1.1*,
# where a bare `on`/`off`/`yes`/`no` is also a boolean -- but Docker's parser is
# YAML 1.2, so to `docker compose` each of those is an ordinary string. The
# difference is not cosmetic: `SSL: on` reaches the container as `SSL=true`
# instead of `SSL=on`, and `on:` used as a *key* resolves to the bool `True` and
# is then refused by the string-key rule -- rejecting a file Docker runs.
#
# It can only be fixed here. Once PyYAML has resolved `on` to `True`, the
# spelling is gone and no downstream pass can recover it.
_YAML_12_BOOL = r"^(?:true|True|TRUE|false|False|FALSE)$"

# YAML 1.2's core-schema float: a dotted mantissa with an optional exponent, OR
# an undotted mantissa with a *mandatory* exponent, OR one of the two named
# constants. PyYAML implements YAML *1.1*, whose float grammar requires a dot
# unconditionally -- a bare `1e3` has none, so PyYAML leaves it the string
# "1e3". Docker's parser is YAML 1.2, where the exponent alone is enough, so
# to Docker `1e3` is the float 1000.0. That is not cosmetic: `cpuset: 1e3` is
# a *string* to compose2pod today, so it slides past the "must be a string"
# rule Docker enforces on the float it sees -- accepting a document Docker
# refuses. The bare-mantissa branch is deliberately narrower than PyYAML's own
# (no digit-grouping underscore, no sexagesimal `:`) because YAML 1.2's core
# schema has neither; it must also never swallow a plain integer (`123`), so
# an exponent or a dot is required in every branch, never both optional at once.
_YAML_12_FLOAT = (
r"^[-+]?(?:[0-9]+\.[0-9]*|\.[0-9]+)(?:[eE][-+]?[0-9]+)?$"
r"|^[-+]?[0-9]+[eE][-+]?[0-9]+$"
r"|^[-+]?\.(?:inf|Inf|INF)$"
r"|^\.(?:nan|NaN|NAN)$"
)


def _build_yaml_loader(yaml_module: ModuleType) -> type:
"""Build a SafeLoader that resolves booleans and floats the way YAML 1.2 (and so Docker) does."""

class Loader(yaml_module.SafeLoader):
pass

# Drop PyYAML's YAML 1.1 bool and float resolvers, then install the 1.2 ones.
# Rebuilding the table is what removes `on`/`off`/`yes`/`no` from the boolean
# set and a bare `1e3` from the string set; they fall through to (respectively)
# the plain-string and float resolvers.
Loader.yaml_implicit_resolvers = {
first_char: [
(tag, regexp)
for tag, regexp in resolvers
if tag not in {"tag:yaml.org,2002:bool", "tag:yaml.org,2002:float"}
]
for first_char, resolvers in yaml_module.SafeLoader.yaml_implicit_resolvers.items()
}
Loader.add_implicit_resolver("tag:yaml.org,2002:bool", re.compile(_YAML_12_BOOL), list("tTfF"))
Loader.add_implicit_resolver("tag:yaml.org,2002:float", re.compile(_YAML_12_FLOAT), list("-+0123456789."))
return Loader


def _load_yaml(text: str) -> Any: # noqa: ANN401 - returns arbitrary parsed compose data
if _yaml is None:
msg = "YAML input requires the 'yaml' extra: pip install compose2pod[yaml] (or pipe JSON via yq)"
raise UnsupportedComposeError(msg)
try:
return _yaml.load(text, Loader=_build_yaml_loader(_yaml)) # noqa: S506 - SafeLoader subclass, not full load
except _yaml.YAMLError as error:
msg = f"invalid YAML: {error}"
raise UnsupportedComposeError(msg) from error


def _read_compose(text: str, fmt: str) -> Any: # noqa: ANN401 - returns arbitrary parsed compose data
if fmt == "json":
return json.loads(text)
if fmt == "yaml":
return _load_yaml(text)
try:
return json.loads(text)
except json.JSONDecodeError:
return _load_yaml(text)
from compose2pod.read import read


def main(argv: list[str] | None = None) -> int:
Expand Down Expand Up @@ -135,8 +50,8 @@ def main(argv: list[str] | None = None) -> int:
else:
text = sys.stdin.read()
try:
compose = _read_compose(text, args.format)
except (json.JSONDecodeError, UnsupportedComposeError) as error:
compose = read(text, args.format)
except UnsupportedComposeError as error:
sys.stderr.write(f"compose2pod: error: could not parse compose input: {error}\n")
return 2
options = EmitOptions(
Expand Down
2 changes: 1 addition & 1 deletion compose2pod/parsing.py
Original file line number Diff line number Diff line change
Expand Up @@ -607,7 +607,7 @@ def _require_string_keys_deep(where: str, node: Any) -> None: # noqa: ANN401 -
that would otherwise disagree: PyYAML's default YAML 1.1 resolver turns
it into a bool, which Docker's YAML 1.2 parser reads as an ordinary
string -- refusing that bool here would reject a file Docker runs. The
CLI's YAML loader (`_build_yaml_loader`, `compose2pod/cli.py`) closes
CLI's YAML loader (`_build_yaml_loader`, `compose2pod/read.py`) closes
that gap ahead of this function, not inside it: it resolves only
`true`/`false` as booleans, matching YAML 1.2, so a bare `on`/`off` never
arrives here as a non-string key at all.
Expand Down
99 changes: 99 additions & 0 deletions compose2pod/read.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
"""Read a compose document from text (JSON or YAML) the way Docker reads it."""

import contextlib
import json
import re
from types import ModuleType
from typing import Any

from compose2pod.exceptions import UnsupportedComposeError


_yaml: ModuleType | None = None
with contextlib.suppress(ImportError): # the optional [yaml] extra is not installed
import yaml as _yaml


# YAML 1.2's boolean set: `true`/`false` only. PyYAML implements YAML *1.1*,
# where a bare `on`/`off`/`yes`/`no` is also a boolean -- but Docker's parser is
# YAML 1.2, so to `docker compose` each of those is an ordinary string. The
# difference is not cosmetic: `SSL: on` reaches the container as `SSL=true`
# instead of `SSL=on`, and `on:` used as a *key* resolves to the bool `True` and
# is then refused by the string-key rule -- rejecting a file Docker runs.
#
# It can only be fixed here. Once PyYAML has resolved `on` to `True`, the
# spelling is gone and no downstream pass can recover it.
_YAML_12_BOOL = r"^(?:true|True|TRUE|false|False|FALSE)$"

# YAML 1.2's core-schema float: a dotted mantissa with an optional exponent, OR
# an undotted mantissa with a *mandatory* exponent, OR one of the two named
# constants. PyYAML implements YAML *1.1*, whose float grammar requires a dot
# unconditionally -- a bare `1e3` has none, so PyYAML leaves it the string
# "1e3". Docker's parser is YAML 1.2, where the exponent alone is enough, so
# to Docker `1e3` is the float 1000.0. That is not cosmetic: `cpuset: 1e3` is
# a *string* to compose2pod today, so it slides past the "must be a string"
# rule Docker enforces on the float it sees -- accepting a document Docker
# refuses. The bare-mantissa branch is deliberately narrower than PyYAML's own
# (no digit-grouping underscore, no sexagesimal `:`) because YAML 1.2's core
# schema has neither; it must also never swallow a plain integer (`123`), so
# an exponent or a dot is required in every branch, never both optional at once.
_YAML_12_FLOAT = (
r"^[-+]?(?:[0-9]+\.[0-9]*|\.[0-9]+)(?:[eE][-+]?[0-9]+)?$"
r"|^[-+]?[0-9]+[eE][-+]?[0-9]+$"
r"|^[-+]?\.(?:inf|Inf|INF)$"
r"|^\.(?:nan|NaN|NAN)$"
)


def _build_yaml_loader(yaml_module: ModuleType) -> type:
"""Build a SafeLoader that resolves booleans and floats the way YAML 1.2 (and so Docker) does."""

class Loader(yaml_module.SafeLoader):
pass

# Drop PyYAML's YAML 1.1 bool and float resolvers, then install the 1.2 ones.
# Rebuilding the table is what removes `on`/`off`/`yes`/`no` from the boolean
# set and a bare `1e3` from the string set; they fall through to (respectively)
# the plain-string and float resolvers.
Loader.yaml_implicit_resolvers = {
first_char: [
(tag, regexp)
for tag, regexp in resolvers
if tag not in {"tag:yaml.org,2002:bool", "tag:yaml.org,2002:float"}
]
for first_char, resolvers in yaml_module.SafeLoader.yaml_implicit_resolvers.items()
}
Loader.add_implicit_resolver("tag:yaml.org,2002:bool", re.compile(_YAML_12_BOOL), list("tTfF"))
Loader.add_implicit_resolver("tag:yaml.org,2002:float", re.compile(_YAML_12_FLOAT), list("-+0123456789."))
return Loader


def _load_yaml(text: str) -> Any: # noqa: ANN401 - returns arbitrary parsed compose data
if _yaml is None:
msg = "YAML input requires the 'yaml' extra: pip install compose2pod[yaml] (or pipe JSON via yq)"
raise UnsupportedComposeError(msg)
try:
return _yaml.load(text, Loader=_build_yaml_loader(_yaml)) # noqa: S506 - SafeLoader subclass, not full load
except _yaml.YAMLError as error:
msg = f"invalid YAML: {error}"
raise UnsupportedComposeError(msg) from error


def _load_json(text: str) -> Any: # noqa: ANN401 - returns arbitrary parsed compose data
try:
return json.loads(text)
except json.JSONDecodeError as error:
msg = f"invalid JSON: {error}"
raise UnsupportedComposeError(msg) from error


def read(text: str, fmt: str) -> Any: # noqa: ANN401 - returns arbitrary parsed compose data
"""Read a compose document from `text`. `fmt` is 'json', 'yaml', or 'auto' (JSON, then YAML)."""
if fmt == "json":
return _load_json(text)
if fmt == "yaml":
return _load_yaml(text)
try:
return _load_json(text)
except UnsupportedComposeError:
return _load_yaml(text)
96 changes: 96 additions & 0 deletions planning/changes/2026-07-15.18-extract-compose-reader.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
summary: Extract the JSON/YAML compose reader (the Docker-YAML-1.2 SafeLoader, the optional PyYAML import, and the format dispatch) out of cli.py into a new `read` module with a one-function interface `read(text, fmt) -> document`, so the repo's subtlest parsing logic gets a direct test surface; normalize its errors to a single `UnsupportedComposeError`.
---

# Design: extract the compose reader into `read.py`

## Summary

`cli.py` is otherwise argparse glue, yet it owns the subtlest domain logic in
the repo: reading compose text as JSON-or-YAML, with a custom `SafeLoader`
subclass that re-resolves booleans and floats the YAML-1.2 (Docker) way so a
bare `on` stays the string `"on"` and a bare `1e3` becomes the float `1000.0`.
Today that logic is reachable only through `cli.main()`. This change moves it
into a new `compose2pod/read.py` deep module whose sole public interface is
`read(text: str, fmt: str) -> Any`, and normalizes its failures to one error
type. `cli.py` shrinks to glue.

## Motivation

The reader is the one place `on`/`off`/`1e3` parity with Docker "can only be
fixed" (`cli.py`'s own comment). It has no direct test surface — every check of
it round-trips through `main()` with stdin monkeypatched, and the missing-extra
path is tested by patching `cli._yaml`. That is a shallow placement: the argparse
module owns Docker-YAML-compatibility semantics, and the interface (the test
surface) is `main()`, not the reader. Giving it its own module makes the
1.2-dialect behavior assertable as `read("k: on", "yaml")["k"] == "on"` instead
of a CLI round-trip.

## Design

**New `compose2pod/read.py`** — moves verbatim from `cli.py`: the optional
`_yaml` import (keeping the module-level `contextlib.suppress(ImportError)`
pattern), `_YAML_12_BOOL`, `_YAML_12_FLOAT`, `_build_yaml_loader`, `_load_yaml`,
and the format dispatch (was `_read_compose`). Public interface:

```python
def read(text: str, fmt: str) -> Any: # fmt in {"auto", "json", "yaml"}
# json: json.loads, wrapping JSONDecodeError -> UnsupportedComposeError("invalid JSON: ...")
# yaml: _load_yaml (missing extra -> "requires the 'yaml' extra ..."; YAMLError -> "invalid YAML: ...")
# auto: try json.loads, fall back to yaml on JSONDecodeError (unchanged)
```

`read` raises **only** `UnsupportedComposeError`. The JSON path — which today
lets `json.JSONDecodeError` propagate — now wraps it as `UnsupportedComposeError`
with an `invalid JSON:` prefix, symmetric with the existing `invalid YAML:`. This
is the one intended behavior change (see Non-goals for why it is not "pure move").

**`cli.py`** — `from compose2pod.read import read`; the call becomes
`read(text, args.format)`; the `except (json.JSONDecodeError,
UnsupportedComposeError)` narrows to `except UnsupportedComposeError`; and the
imports that go dark (`json`, `re`, `contextlib`, `ModuleType`, `Any`) are
removed so `ruff` stays clean. Everything else in `main()` is untouched.

## Non-goals

- Changing what compose2pod accepts/rejects — the parsed document is identical
for every input; only the *error type/message* for a malformed `--format json`
changes.
- A strictly byte-identical move — the maintainer chose the deeper single-error
interface over a pure relocation; the `invalid JSON:` prefix is that choice.
- Touching the YAML-1.2 grammar itself (`2026-07-15.01-yaml-12-floats`) — the
resolvers move unchanged.

## Testing

`just test-ci` at 100% coverage, `just lint-ci` clean, `just check-planning`.

New `tests/test_read.py` gives the reader its first direct surface:
- JSON: a dict round-trips; `read("not json", "json")` raises
`UnsupportedComposeError` matching `invalid JSON`.
- YAML 1.2 dialect (the payoff): `read("k: on", "yaml")["k"] == "on"`;
`read("k: true", "yaml")["k"] is True`; `read("k: 1e3", "yaml")["k"] == 1000.0`;
`read("k: 123", "yaml")["k"] == 123` (int, not swallowed by the float resolver);
a bare `on:` key stays the string `"on"`.
- `auto`: JSON input parses as JSON; YAML input falls back; both-invalid raises.
- Missing extra: `monkeypatch.setattr(read, "_yaml", None)` then
`read("x", "yaml")` raises matching `requires the 'yaml' extra`.

`tests/test_cli.py`: the YAML/JSON tests keep passing through `main()` unchanged;
`test_yaml_without_pyyaml_errors` updates its monkeypatch target from `cli._yaml`
to the `read` module; `test_malformed_json_returns_2` still passes (it asserts the
`could not parse` wrapper, which the CLI still prints around the wrapped error).

`architecture/supported-subset.md` is checked for any reader claim that the
`invalid JSON:` message change makes stale, and promoted in this PR if so.

## Risk

- **A `cli.py` import left dark or one still needed after the move** (low × low):
`just lint-ci` (ruff F401) catches the former; `just test-ci` the latter.
- **Coverage gap on a `read.py` branch** (low × med): the `_build_yaml_loader`
resolver rebuild is covered by the `on`→string and `1e3`→float assertions
(they prove the bool/float resolvers were replaced); every dispatch and error
branch has a direct test above.
- **`test_read.py` YAML cases assume PyYAML present** (low × low): the existing
`test_cli.py` YAML tests already rely on the same, so the test env provides it.
4 changes: 2 additions & 2 deletions tests/conformance/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@
import pytest
import yaml

from compose2pod.cli import _read_compose
from compose2pod.emit import EmitOptions, emit_script
from compose2pod.exceptions import UnsupportedComposeError
from compose2pod.extends import resolve_extends
from compose2pod.parsing import validate
from compose2pod.read import read


_DOCKER = shutil.which("docker")
Expand Down Expand Up @@ -100,7 +100,7 @@ def _compose2pod_accepts(text: str, workdir: Path) -> bool:
would understate compose2pod's strictness and report false violations.
"""
try:
compose = resolve_extends(_read_compose(text, "yaml"))
compose = resolve_extends(read(text, "yaml"))
validate(compose)
emit_script(
compose=compose,
Expand Down
Loading