feat(artifacts): narrow binary lookups to declared dependencies

[punchagan]

Description

Binary lookups made from a directory owned by a package (via the package's
(dir ...) field) are narrowed to the packages that package depends on,
transitively. This covers %{bin:...}, %{bin-available:...}, the PATH
given to actions, and the binaries staged by (deps %{bin:...}).

Directories with no owning package are unaffected: nothing is narrowed and
every package is searched, as before. Binaries found on the ambient PATH,
and those bound by (env (binaries ...)), are also unaffected.

Related Issue and Motivation

To know which binaries a lock directory package provides, dune has to build
it -- the metadata currently available does not say. So resolving a single
%{bin:X} meant building every package in the lock directory to find out
which one provides X. Narrowing the search to the owning package's
dependency closure removes that: only the closure is built.

This also removes an inconsistency. dune build -p masks away sibling
packages' install stanzas, so a stanza using an undeclared sibling's binary
fails at release time -- during opam install, or in the opam-repository CI
-- while a plain dune build resolves it happily. The narrowing surfaces
that failure at dev time, with a hint naming the missing dependency.

Shrinking the built set is also a prerequisite for the in-and-out work
(#8652), where forcing every package's install cookie closes a dependency
cycle.

Decisions worth review

• Transitive, not direct. Installing a package into an opam switch also
  installs its transitive dependencies, so their binaries are on PATH
  there. Narrowing to direct dependencies only would be stricter than what
  opam gives you, and would fail builds that opam-repository CI accepts. The
  counter-argument is that (deps (package ...)) (feat(pkg): install layouts for package sets #14373) and
  strict_package_deps are both immediate-only, so there is a case for
  consistency with those instead. This is decidable independently of the rest
  of the PR and does not affect the build-graph change.
• No lang version gate. The narrowing applies wherever (dir ...) is
  used. (dir ...) is itself opt-in and available since (lang dune 3.21).
• Not gated on a lock directory. Gating it would mean that adopting
  dune pkg lock silently turns on dependency checking for sibling binaries,
  which has nothing to do with lock files.

Known limitations

• This reduces the search space; it does not enforce that dependencies are
  declared. A binary that is narrowed out can still be found on the ambient
  PATH.
• Dependency filters such as {with-test} are not interpreted, so a
  dependency is visible whatever its filter says.

[Alizter]

The precedence in a few places is a bit confusing. Here is an example of something that is unintentionally changed in this PR:

  $ make_lockdir

  $ make_lockpkg provider <<'EOF'
  > (version 0.0.1)
  > (build
  >  (progn
  >   (system "echo '#!/bin/sh' > managed-tool")
  >   (system "echo 'exit 0' >> managed-tool")
  >   (system "chmod +x managed-tool")
  >   (system "echo 'bin: [ \"managed-tool\" ]' > provider.install")))
  > EOF

  $ make_dune_project 3.25
  $ cat >>dune-project <<'EOF'
  > (package
  >  (name consumer)
  >  (allow_empty)
  >  (dir .)
  >  (depends provider))
  > EOF
  $ mkdir empty-bin
  $ cat >dune <<EOF
  > (env
  >  (_
  >   (env_vars
  >    (PATH $PWD/empty-bin))))
  > 
  > (rule
  >  (alias prepare)
  >  (action (run %{bin:managed-tool})))
  > 
  > (rule
  >  (action
  >   (with-stdout-to output
  >    (bash
  >     "command -v managed-tool >/dev/null && echo visible || echo hidden"))))
  > EOF

  $ dune build @prepare
  $ dune build output
  $ cat _build/default/output
  hidden                                                                                                                       

I think we should try to preserve the previous semantics as much as possible. I'm not opposed to changing these semantics at a later point if we think it will make more sense however, but that should be a bit more intentional.

I've given some other comments about some subtle PATH changes.

I think the order should be something like:

1. Action-local additions: (deps (package ...)) and staged %{bin:...} dependencies.
2. Directory (env (binaries ...)) bindings
3. Directory (env_vars (PATH ...))
4. Automatically inferred package-closure paths
5. Context/ambient PATH

[Alizter]

I suppose this claim only holds for stanzas owned by a package, maybe worth mentioning that unowned stanzas still build every package.