Skip to content

nosman: system-wide artifact cache to avoid re-downloading and re-extracting dependencies #55

Description

@shamilatesoglu

Problem

Every workspace resolves and installs its own copy of every dependency, independently. Fetched packages land under <workspace>/Module/Downloaded/<name>/<version> (or Package/Downloaded for non-plugin types) — see install.rs's output_dir_inferred — and there is no cache outside a workspace at all.

On a machine with several workspaces (or CI runners rebuilding often), this means:

  1. The same package version is downloaded from the store repeatedly, once per workspace.
  2. The same package version is extracted to disk repeatedly, once per workspace — duplicate copies of identical binaries/plugin folders sitting side by side.

Wanted outcome

A package version that's already been fetched and extracted once on the machine should not be downloaded or extracted again for another workspace. nosman should own a per-machine (shared across users) cache of artifacts, keyed at least by package name + version (+ platform/build config, since these are compiled plugins), and Nodos should be able to find the resulting folder without each workspace holding its own private copy — i.e. it should ask nosman for a dependency's real location (or otherwise be made aware of it), rather than assuming everything lives under its own Downloaded/ folder.

Open questions

  • How Nodos gets the real path — a few options, each with a trade-off:
    • Nodos queries nosman at plugin-scan time (spawned process or linked call) to resolve name+version to a path. Explicit and testable, but adds a runtime dependency from engine startup onto nosman.
    • nosman writes a shared index/manifest that Nodos reads directly during its own scan. No process call needed, but two systems have to agree on and keep a format in sync.
    • nosman places a symlink/junction at the workspace's usual Downloaded/<pkg> path pointing into the shared store, so Nodos's existing directory scan needs no changes. Simple for Nodos, but junctions have platform quirks (permissions, tooling that doesn't follow links) and hide the indirection.
  • Cache key granularity — does a plugin build need to be keyed only by name+version, or also by compiler/ABI/SDK version, since two workspaces could build the same plugin version against incompatible toolchains?
  • Eviction — does the cache need a size cap / LRU policy, or is it left to grow (with a manual clear command) for now?
  • Permissions — a per-machine, shared-across-users cache means multiple users writing into it; does that need any access-control thought, or is it fine as an all-users-writable location?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions