diff --git a/Makefile b/Makefile index 7a47f558a..c4183913b 100644 --- a/Makefile +++ b/Makefile @@ -58,7 +58,7 @@ check-with-klee: $(MAKE) check RESOLVE_BUILD_KLEE=ON test: configure - cmake --build $(RESOLVE_CMAKE_BUILD_DIR) --target test-CVEAssert test-libresolve test-reach-rs + cmake --build $(RESOLVE_CMAKE_BUILD_DIR) --target test-CVEAssert test-libresolve test-resolve-reach test-with-klee: $(MAKE) test RESOLVE_BUILD_KLEE=ON diff --git a/docs/components/facts.md b/docs/components/facts.md index cea5a5ba8..212f6fbb3 100644 --- a/docs/components/facts.md +++ b/docs/components/facts.md @@ -1,8 +1,46 @@ # Facts -Fact generation is a static program analysis technique that extracts structured information about a program from its source code or intermediate representation (i.e. [LLVM-IR](https://llvm.org/docs/LangRef.html)). A *fact* is a piece of information that describes some property of a program. Facts can be used to describe relationships between code and data. The `EnhancedFacts` pass plugin constructs program facts based on the program's control- and data-flow, and embeds these facts into custom ELF sections for downstream analysis. +Facts are information about a program, extracted from [LLVM IR](https://llvm.org/docs/LangRef.html) at compile-time. Each fact describes a program node, property, or relationship. The collection of facts can form a Control Flow Graph, though they carry additional metadata beyond just that. -These facts are compressed with zstd and stored inside a custom ELF section in the compiled binary called `.facts`. Reachability analysis can be performed by the [reach](reach.md) tool, which consumes these facts in its analysis. The [reachability example](../examples/reachability.md) walks through generating and querying facts end-to-end. +[resolvecc](resolve-cc.md) will produce and embed facts into a `.facts` section inside the compiled ELF in a compact binary format, typically compressed with zstd. -!!! note - Developed for easy parsing and to encourage compatibility with third party tools, the facts format can consume quite a bit of storage and memory, particularly when uncompressed, due to being text-based. +The [reach](reach.md) command consumes these facts from an ELF file, shared objcet, or an extracted `.facts` file. The [reachability example](../examples/reachability.md) shows a complete end-to-end example of this. + +## Binary Format Specification + +The binary format is typically compressed with zstd when it is attached to compiled objects. A zstd frame can be identified by the leading bytes: `28 B5 2F FD`. + +For definitive structure, consult the [Rust schema](https://github.com/riversideresearch/resolve/tree/main/resolve-facts/rs/src/schema.rs). + +The uncompressed facts stream has no top-level header. It contains one or more modules in sequence: + +```text +Facts stream +├── ModuleHeader (16 bytes) +│ ├── version: u32 +│ ├── node_count: u32 +│ ├── edge_count: u32 +│ └── string_pool_len: u32 +├── Node[node_count] (32 bytes each) +│ ├── meta: u32 (multiple bitmasks) +│ ├── idx: u32 +│ ├── name: u32 (string offset) +│ ├── opcode: u32 (string offset) +│ ├── source_line: u32 +│ ├── source_col: u32 +│ ├── source_file: u32 (string offset) +│ └── function_type: u32 (string offset) +├── Edge[edge_count] (12 bytes each) +│ ├── src: u32 (NodeID) +│ ├── dst: u32 (NodeID) +│ └── kinds: u32 (bitmask) +├── String pool (string_pool_len bytes) +│ └── repeated [byte length: u32][UTF-8 bytes ...] +└── Next module, if present +``` + +A node ID is its index in the node array of its module. The `meta` field stores the node type, property flags, linkage, and call type. + +The `kinds` field is a bit set. One source and destination pair can have multiple relationships, such as `Calls`, `Contains`, or `ControlFlowTo`. + +The string pool has four-byte alignment. The writer adds zero padding after the final string when the pool requires it. diff --git a/docs/components/reach.md b/docs/components/reach.md index 56218b260..705ec40df 100644 --- a/docs/components/reach.md +++ b/docs/components/reach.md @@ -1,114 +1,96 @@ # Reach -`reach` performs static reachability queries on `resolve` program metadata. It consumes the [fact files](facts.md) extracted from binaries by `linker` and determines whether a path exists from the program entry point to a specified vulnerability. When a path is found, `reach` packages the results into a `.json` object and writes them either to a user-specified path or to `stdout` by default. +`resolve reach` determines whether a program entry point can reach a vulnerable function. It uses static control-flow data from RESOLVE facts. -A Python wrapper, `reach.py`, provides a convenient command-line interface to interact with `reach`. For more information about `reach`, see the [`reach`](https://github.com/riversideresearch/resolve/tree/main/reach) documentation. +The command accepts facts from these sources: -!!! tip - For a hands-on, end-to-end walkthrough of a reachability query, see the [reachability example](../examples/reachability.md). +- An ELF executable or shared library that contains a `.facts` section. +- An extracted `.facts` file. +- A directory that contains one or more `.facts` files. +- Multiple inputs through repeated `-f` arguments. -## Developer Information +The command writes one JSON result for each entry in `vulnerabilities.json`. -### Run +## Use the command +```bash +resolve reach \ + --input vulnerabilities.json \ + --facts program.facts \ + --output reach.json ``` -cmake -B build && cmake --build build/ -./build/reach --help -``` -### Description +The command uses `main` as the default entry point. Use `--entry` to select a different function. + +```bash +resolve reach -i vulnerabilities.json -f program.facts -e service_main +``` -This development is factored into a library part (under `lib/`) and an -executable tool (under `src/`) that uses the library. +If you omit `--output`, the command derives the path from the input name. For example, `vulnerabilities.json` produces `vulnerabilities.reach.json`. -See the `--help` output for command line arguments/options. +Use `--src` to read a package version from a Vcpkg manifest. The result becomes unreachable when the installed version is outside the vulnerable range. -The minimum required arguments for performing a reachability query are -`--facts_dir` (path to directory containing facts files extracted from -the program binary), and the `--src` and `--dst` node IDs. The tool -will construct a control-flow graph from the facts in `facts_dir`, and -attempt to find the shortest path from `src` to `dst` in it. +## Dynamic-link analysis -The `src` and `dst` node IDs should match how they appear in the facts -files, which is determined by the [**RESOLVE** LLVM -pass](https://github.com/riversideresearch/resolve/blob/main/resolve-cc/src/ResolveFactsPluginPass.cpp) -that generates the facts. +Use `--dynlink` to include compatible external-linkage functions as indirect-call targets. -For example, if `nodeprops.facts` contains the following line: +```bash +resolve reach -i vulnerabilities.json -f program.facts --dynlink ``` -/src/guestbook/src/main.cpp:f_GLOBAL__sub_I_main.cpp,Function + +Use `--dlsym-log` with `--dynlink` to restrict those targets to observed symbols. + +```bash +resolve reach \ + -i vulnerabilities.json \ + -f program.facts \ + --dynlink \ + --dlsym-log dlsym.json ``` -there is a node of type `Function` with ID -`/src/guestbook/src/main.cpp:f_GLOBAL__sub_I_main.cpp`. -Arguments can also be specified in an input JSON file instead of as -command line arguments. See the `--input` argument. If an argument is -provided in both the input file and at the command line, the command -line argument takes precedence. The input file format is specified by -the struct `config` in `src/config.hpp` (the JSON deserializer is -auto-generated from this definition). +The log has this structure: + +```json +{ + "loaded_symbols": [ + { + "symbol": "plugin_entry", + "library": "libplugin.so" + } + ] +} +``` -The input file format supports multiple queries (see struct `query` -and the `queries` field of struct `config` in `src/config.hpp`). +The graph matches the `symbol` value. The `library` value remains available for future matching changes. -### Architecture +## Architecture -The implementation is organized roughly as follows: +The Rust command owns input parsing, function lookup, version comparison, and report generation. It calls `libreach` through a small C interface. ```mermaid -graph LR; - A[/input.json
cmd args/]-.->B; - B[main.cpp]-->C; - C[[facts.hpp]]-->|facts database|D; - D[[graph.hpp]]-->|constructed graph|E; - E[[search.hpp]]-->|discovered paths|B; +graph LR + A[vulnerabilities.json] --> B[resolve-reach] + F[ELF or binary facts] --> B + B --> C[libreach] + C --> B + B --> O[reachability report] +``` + +The command loads all facts once. Then it builds one graph and uses that graph for all unresolved sinks. + +## Developer commands + +Build the command: + +```bash +cmake -B build +cmake --build build --target resolve-reach +``` - F[(nodes.facts
nodeprops.facts
edges.facts)]-.->C; +Run its existing tests: - B-.->O[/output.json/] +```bash +cmake --build build --target test-resolve-reach ``` -The main reads the input config (plus command line arguments), and -then uses the functionality declared in `lib/facts.hpp` to load the -facts files from the disk into an in-memory database. This database is -used by `lib/graph.hpp` to build a graph, which is passed to -`lib/search.hpp` for finding paths. Finally, the paths are packaged -into a JSON object and written to the provided output path or to -stdout if no path was given. - -### Code - -Under `lib/`: - -- facts.hpp, facts.cpp - - in-memory representation of fact databases, and loading from .facts files - - defns related to dlsym loaded symbol logs from dynamic analysis -- graph.hpp, graph.cpp - - weighted directed graphs with integer node labels, and functions - for constructing them from facts databases - - `handle_map`s for mapping between string node IDs and their - integer labels (handles) -- search.hpp, search.cpp - - pathfinding algorithms on graphs. Currently: - - BFS - - Dijkstra's shortest path - - Yen's K-shortest paths - - also computing distance maps for KLEE (min distance of each node - in the graph to a specified destination node) -- util.hpp - - misc helper functions - - `at` function for vector and unordered_map with slightly better - error reporting - - `time` function for measuring time to execute a given function -- distmap.hpp, distmap.cpp - - compute distance maps and blacklists for directed KLEE - -Under `src/`: - -- config.hpp - - specifications of the tool's input and output formats as structs - - JSON serializers and deserializers are auto-generated from these - specifications via the Lohmann JSON library -- main.cpp - - parse arguments, load facts, build graph, perform queries, output - results +See the [reachability example](../examples/reachability.md) for a complete workflow. diff --git a/docs/examples/reachability.md b/docs/examples/reachability.md index c9781da14..1b7ffce08 100644 --- a/docs/examples/reachability.md +++ b/docs/examples/reachability.md @@ -23,18 +23,15 @@ We want to ask **RESOLVE**: starting from `main`, can execution actually reach ` ## A Vulnerability Specification -First, describe the vulnerability we want to analyze in a JSON file (let's call it [`vulnerabilities.json`](../concepts/vulnerabilities-json.md) on disk). Each entry in the array is a *sink* (a function we would like to try to reach). All of the following fields are required, and will be fed-through into our final report: +First, describe the vulnerability in a [`vulnerabilities.json`](../concepts/vulnerabilities-json.md) file on disk. Each entry identifies one affected function, which is called a sink. ```json { "vulnerabilities": [ { "cve-id": "CVE-0000-00000", - "cve-description": "Null pointer dereference reachable from the program entry point.", "package-name": "reachability-example", "package-version": "vers:generic/*", - "cwe-id": "476", - "cwe-name": "NULL Pointer Dereference", "affected-function": "do_npd", "affected-file": "main.c" } @@ -53,34 +50,26 @@ Reachability analysis runs on program *facts* (see: [RESOLVE facts](../component resolvecc main.c -o main ``` -## Extracting the Facts - -Next, pull the embedded facts back out of the binary into a `main.facts` file with `resolve get-facts`: - -```bash -resolve get-facts -i main -``` - -This writes `main.facts` (alongside a compressed `main.facts.zst`) into the current directory. - ## Running the Reachability Query -Now we have everything [`resolve reach`](../components/reach.md) needs: the vulnerability specification and the facts. Point it at both and choose an output path for the report: +[`resolve reach`](../components/reach.md) reads embedded facts directly from the compiled ELF. Pass the path of the program and select an output file: ```bash -resolve reach -i vulnerabilities.json -f main.facts -o out.json +resolve reach -i vulnerabilities.json -f main -o out.json ``` !!! tip If your entry point is not `main`, pass `-e ` to `resolve reach`. For projects with a vcpkg source tree, pass `-s ` so the report can additionally check whether the pinned package version falls in the vulnerable range. -`resolve reach` locates the entry point (`main` by default), locates each sink in the facts, and searches the control-flow graph for a path between them. Along the way it prints what it found: +!!! note + If you need a separate facts file, use `resolve get-facts -i main`. This command writes `main.facts` and `main.facts.zst`. You can pass `main.facts` to `resolve reach`. + +`resolve reach` locates the entry point and each sink. Then it searches the control-flow graph for a path. ```txt -Found function 'main' in module 'src/main.c' -Found function 'do_npd' in module 'src/main.c' -[RW]: Invoking reach 'reach -f main.facts -i reach_wrap_input.json -o reach_wrap_output.json' -[RW]: Wrote out.json. +[REACH] Loaded 1 facts modules from 1 input files. +[REACH] Built a libreach graph with 3 edges. +[REACH] Wrote 'out.json'. ``` ## Interpreting the Report @@ -97,13 +86,13 @@ The report in `out.json` classifies each sink and, when it is reachable, spells "conclusion": "Statically Reachable", "reason": "Control Flow Graph analysis found the following candidate path...", "call_path": [ - "Function(main) ((1556769911, 9))", - "DirectCall -> Function(do_npd) ((1556769911, 1))" + "Function(main) ((0, 9))", + "DirectCall -> Function(do_npd) ((0, 1))" ], "control_flow_path": [ - "Function(main) ((1556769911, 9))", - "Contains -> BasicBlock() ((1556769911, 10))", - "DirectCall -> Function(do_npd) ((1556769911, 1))" + "Function(main) ((0, 9))", + "Contains -> BasicBlock(0) ((0, 10))", + "DirectCall -> Function(do_npd) ((0, 1))" ] } } @@ -111,10 +100,10 @@ The report in `out.json` classifies each sink and, when it is reachable, spells } ``` -The `call_path` is the human-readable answer: `main` makes a `DirectCall` to `do_npd`, so the vulnerability is reachable. The `control_flow_path` is the same route at basic-block granularity. +`call_path` gives an exact answer here: `main` makes a `DirectCall` to `do_npd`, so the vulnerability is statically reachable! The `control_flow_path` is the same route at basic-block granularity. !!! note - The classification is **potentially reachable (statically reachable)**, not **explicitely exploitable**. Reachability analysis only proves that a path exists in the control-flow graph; it does not prove a concrete input can drive execution down that path. Producing such an input is the job of [input synthesis](input-synthesis.md). + The classification is **potentially reachable (statically reachable)**, not **explicitly exploitable**. Reachability analysis proves that a control-flow path exists. It does not prove that a concrete input can use that path. [Input synthesis](input-synthesis.md) produces such an input. ### Other Classifications @@ -126,17 +115,14 @@ Depending on what `resolve reach` finds, a sink can come back as: | `unreachable` | Not Reachable | The function exists in the program, but no path reaches it from the entry point. | | `unreachable` | Not Found | The affected function was not found in the compiled program metadata (e.g. it was inlined, dead-code eliminated, or never linked in). | -## TDLR (Quick Reference) +## TLDR (Quick Reference) Given source code, you can run a reachability query with: ```bash resolvecc main.c -o main -resolve get-facts -i main -resolve reach -i vulnerabilities.json -f main.facts -o out.json +resolve reach -i vulnerabilities.json -f main -o out.json ``` !!! tip Once a path is confirmed, synthesize a concrete triggering input with input synthesis (above), or instrument a fix at compile time with [remediation](remediation.md). - - diff --git a/examples/misc/eboss_eval/src/analyze-images/compose-analyze-image.yml b/examples/misc/eboss_eval/src/analyze-images/compose-analyze-image.yml index a678072ec..c95c4e67d 100644 --- a/examples/misc/eboss_eval/src/analyze-images/compose-analyze-image.yml +++ b/examples/misc/eboss_eval/src/analyze-images/compose-analyze-image.yml @@ -92,7 +92,6 @@ services: /opt/resolve/bin/resolve-reach \ -i /challenge/vulnerabilities.json \ -o /facts-dir/reach_out.json \ - -f /facts-dir/build/ \ - -r /opt/resolve/bin/reach + -f /facts-dir/build/analyze-image.facts " depends_on: [server-remediated] diff --git a/examples/misc/openssl.sh b/examples/misc/openssl.sh index b0d504b69..edf5f3865 100755 --- a/examples/misc/openssl.sh +++ b/examples/misc/openssl.sh @@ -11,7 +11,7 @@ set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" EXTRACT_FACTS_SCRIPT="/opt/resolve/bin/extract_facts.py" -REACH_WRAPPER="/opt/resolve/bin/resolve-reach" +REACH_COMMAND="/opt/resolve/bin/resolve-reach" export CC="/usr/bin/clang" export CXX="/usr/bin/clang++" @@ -50,11 +50,10 @@ mkdir openssl_facts # Run reach analysis # ------------------- echo "[+] Running reachability analysis." -"$REACH_WRAPPER" \ +"$REACH_COMMAND" \ -i openssl_vulnerabilities.json \ -o openssl_reach_out.json \ -f openssl_facts/libcrypto.facts \ - -e "CMS_RecipientInfo_decrypt" \ - -r /opt/resolve/bin/reach + -e "CMS_RecipientInfo_decrypt" # TODO: Add remediation portion check for exit code 3 for successful remediation diff --git a/examples/reachability/main b/examples/reachability/main deleted file mode 100755 index ef01b5e86..000000000 Binary files a/examples/reachability/main and /dev/null differ diff --git a/examples/reachability/run.sh b/examples/reachability/run.sh index 389c16ad9..94671645d 100755 --- a/examples/reachability/run.sh +++ b/examples/reachability/run.sh @@ -3,8 +3,6 @@ set -euo pipefail resolvecc src/main.c -o main -resolve get-facts -i main - -resolve reach -i vulnerabilities.json -f main.facts -o out.json +resolve reach -i vulnerabilities.json -f main -o out.json echo "Reachability report written to out.json" diff --git a/resolve-cc/src/ResolveFactsPluginPass.cpp b/resolve-cc/src/ResolveFactsPluginPass.cpp index f0ddfea60..773c12a7f 100644 --- a/resolve-cc/src/ResolveFactsPluginPass.cpp +++ b/resolve-cc/src/ResolveFactsPluginPass.cpp @@ -3,7 +3,7 @@ * LGPL-3; See LICENSE.txt in the repo root for details. */ -#include "resolve_facts_llvm/resolve_facts_llvm.hpp" +#include "resolve_facts_llvm/binary_facts_llvm.hpp" #include "llvm/IR/Module.h" #include "llvm/IR/PassManager.h" @@ -12,8 +12,10 @@ struct ResolveFactsPluginPass : public PassInfoMixin { PreservedAnalyses run(Module &M, ModuleAnalysisManager &) { - resolve::getModuleFacts(M); - resolve::embedFacts(M); + resolve::BinaryLLVMFacts facts; + resolve::getBinaryModuleFacts(facts, M); + const auto serialized = facts.serialize(); + resolve::embedBinaryFacts(M, serialized.bytes()); return PreservedAnalyses::all(); } }; diff --git a/resolve-cli/CMakeLists.txt b/resolve-cli/CMakeLists.txt index b0bb545eb..4a7653a9a 100644 --- a/resolve-cli/CMakeLists.txt +++ b/resolve-cli/CMakeLists.txt @@ -4,53 +4,48 @@ set(RESOLVE_PYTHON_VERSION "3.12" CACHE STRING "Python version used for the resolve CLI environment") option(RESOLVE_BUNDLE_PYTHON "Install a uv-managed Python into the resolve install prefix" OFF) -# Build the Rust replacement for the Python reachability wrapper +# Build the reachability command. find_program(CARGO_EXECUTABLE cargo REQUIRED) if(CMAKE_BUILD_TYPE STREQUAL "Debug" OR CMAKE_BUILD_TYPE STREQUAL "") - set(REACH_RS_CARGO_PROFILE debug) - set(REACH_RS_CARGO_FLAGS) + set(REACH_CARGO_PROFILE debug) + set(REACH_CARGO_FLAGS) else() - set(REACH_RS_CARGO_PROFILE release) - set(REACH_RS_CARGO_FLAGS --release) + set(REACH_CARGO_PROFILE release) + set(REACH_CARGO_FLAGS --release) endif() -set(REACH_RS_CRATE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/src/resolve/reach") -set(REACH_RS_TARGET_DIR "${CMAKE_CURRENT_BINARY_DIR}/reach-rs-target") -set(REACH_RS_BINARY "${REACH_RS_TARGET_DIR}/${REACH_RS_CARGO_PROFILE}/reach") -set(REACH_RS_CARGO_ENV - "CARGO_TARGET_DIR=${REACH_RS_TARGET_DIR}" +set(REACH_CRATE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/src/resolve/reach") +set(REACH_TARGET_DIR "${CMAKE_CURRENT_BINARY_DIR}/reach-target") +set(REACH_BINARY "${REACH_TARGET_DIR}/${REACH_CARGO_PROFILE}/resolve-reach") +set(REACH_CARGO_ENV + "CARGO_TARGET_DIR=${REACH_TARGET_DIR}" "RESOLVE_LIBREACH_DIR=$" ) -add_custom_target(reach-rs ALL +add_custom_target(resolve-reach ALL COMMAND ${CMAKE_COMMAND} -E env - ${REACH_RS_CARGO_ENV} - ${CARGO_EXECUTABLE} build --locked ${REACH_RS_CARGO_FLAGS} - WORKING_DIRECTORY "${REACH_RS_CRATE_DIR}" - BYPRODUCTS "${REACH_RS_BINARY}" + ${REACH_CARGO_ENV} + ${CARGO_EXECUTABLE} build --locked ${REACH_CARGO_FLAGS} + WORKING_DIRECTORY "${REACH_CRATE_DIR}" + BYPRODUCTS "${REACH_BINARY}" DEPENDS libreach - COMMENT "Building the Rust reach binary" + COMMENT "Building resolve-reach" USES_TERMINAL VERBATIM ) -add_custom_target(test-reach-rs +add_custom_target(test-resolve-reach COMMAND ${CMAKE_COMMAND} -E env - ${REACH_RS_CARGO_ENV} + ${REACH_CARGO_ENV} ${CARGO_EXECUTABLE} test --locked - WORKING_DIRECTORY "${REACH_RS_CRATE_DIR}" + WORKING_DIRECTORY "${REACH_CRATE_DIR}" DEPENDS libreach - COMMENT "Running the Rust reach tests" + COMMENT "Running the resolve-reach tests" USES_TERMINAL VERBATIM ) -install(PROGRAMS "${REACH_RS_BINARY}" - DESTINATION "${CMAKE_INSTALL_BINDIR}" - RENAME reach-rs -) - # Make the install prefix a Python environment for the resolve CLI tools. install(CODE " set(_resolve_python_version \"${RESOLVE_PYTHON_VERSION}\") @@ -140,3 +135,22 @@ install(CODE " message(FATAL_ERROR \"uv pip install failed with exit code \${_uv_result}\") endif() ") + +# Install the Rust command after the Python package. This replaces the legacy +# Python entry point during an in-place upgrade. +install(CODE " + set(_install_prefix \"\${CMAKE_INSTALL_PREFIX}\") + if(DEFINED ENV{DESTDIR} AND NOT \"\$ENV{DESTDIR}\" STREQUAL \"\") + if(IS_ABSOLUTE \"\${_install_prefix}\") + set(_install_prefix \"\$ENV{DESTDIR}\${_install_prefix}\") + else() + set(_install_prefix \"\$ENV{DESTDIR}/\${_install_prefix}\") + endif() + endif() + + file(REMOVE \"\${_install_prefix}/${CMAKE_INSTALL_BINDIR}/resolve-reach\") +") + +install(PROGRAMS "${REACH_BINARY}" + DESTINATION "${CMAKE_INSTALL_BINDIR}" +) diff --git a/resolve-cli/pyproject.toml b/resolve-cli/pyproject.toml index b725314c7..934cec1d0 100644 --- a/resolve-cli/pyproject.toml +++ b/resolve-cli/pyproject.toml @@ -11,12 +11,10 @@ dependencies = [ "ollama>=0.6.1", "pydantic>=2.12.5", "pyelftools>=0.32", - "univers>=31.1.0", ] [project.scripts] resolve = "resolve.cli:main" -resolve-reach = "resolve.reach:main" resolve-remediate = "resolve.remediate:main" resolve-get-facts = "resolve.get_facts:main" resolve-crash-analysis = "resolve.crash_analyzer.smith:main" diff --git a/resolve-cli/src/resolve/cli.py b/resolve-cli/src/resolve/cli.py index 908d606c5..a7a299c99 100644 --- a/resolve-cli/src/resolve/cli.py +++ b/resolve-cli/src/resolve/cli.py @@ -31,7 +31,9 @@ def files_in_path_dirs(path_dirs: list[Path]): path_dirs.insert(0, argv0_path.resolve().parent) for file in files_in_path_dirs(path_dirs): if (sub := is_subcommand(file)) and os.access(file, os.X_OK): - subcommands[sub] = file + # Use the command from the first matching directory. This keeps + # the installed command ahead of stale commands later in PATH. + subcommands.setdefault(sub, file) return subcommands def subcommand_cli(program: str): diff --git a/resolve-cli/src/resolve/reach.py b/resolve-cli/src/resolve/reach.py deleted file mode 100644 index 90499e3dd..000000000 --- a/resolve-cli/src/resolve/reach.py +++ /dev/null @@ -1,735 +0,0 @@ -#!/usr/bin/env python3 -# -# Copyright (c) 2025 Riverside Research. -# LGPL-3; See LICENSE.txt in the repo root for details. - -from dataclasses import dataclass, field - -from operator import attrgetter -import os -import gc -import json -import argparse -import subprocess -from pathlib import Path -from enum import Enum, auto -from typing import Any, Callable, Iterable, TypeVar - -from univers.version_range import GenericVersionRange -from univers.versions import SemverVersion - -class Reachability(Enum): - UNKNOWN = auto() - - UNREACHABLE_NOT_FOUND = auto() - UNREACHABLE_NO_PATH = auto() - UNREACHABLE_NOT_VULNERABLE = auto() - - REACHABLE = auto() - -@dataclass -class Sink: - """ - Represent a target fn we would like to reach in a reachability - query - - Preserves the info about a sink from vulnerabilities.json - so we don't have to blindly assume we map 1-to-1 with an output - reach query, since not every sink is reachable - """ - - # supplied by vulnerabilities.json - cve_id: str - cve_description: str - package_name: str - vulnerable_package_version: str - package_version: str | None - cwe_id: str - cwe_name: str - affected_function: str - affected_file: str - - @classmethod - def from_vuln_dict(cls, vuln: dict[str, str]) -> "Sink": - """ - Load metadata from TA2 supplied vulnerabilities.json - """ - - def get(key: str): - val = vuln.get(key, None) - if val is not None: - return val - return vuln[key.replace("-", "_")] - - return cls( - # The only required fields for our analysis - cve_id=get("cve-id"), - affected_function=get("affected-function"), - # Misc - cve_description=get("cve-description"), - package_name=get("package-name"), - vulnerable_package_version=get("package-version"), - package_version=None, # populate later if we get the src dir - cwe_id=get("cwe-id"), - cwe_name=get("cwe-name"), - affected_file=get("affected-file"), - ) - -T = TypeVar("T") -K = TypeVar("K") -def group_by(items: Iterable[T], key_func: Callable[[T], K]): - result: dict[K, list[T]] = {} - for item in items: - key = key_func(item) - result.setdefault(key, []).append(item) - - return result - -NodeID = tuple[int, int] -EdgeID = tuple[NodeID, NodeID] -NodeKind = str -EdgeKind = str - -@dataclass -class Node: - id: NodeID - kind: NodeKind - props: dict[str, Any] = field(default_factory=dict[str, Any]) - - T = TypeVar('T') - def get(self, key: str, default: T = None) -> str | T: - return self.props.get(key, default) - - def __getitem__(self, key: str): - return self.props[key] - - def __setitem__(self, key: str, value: Any): - self.props[key] = value - - def get_name(self): - if demangled_name := self.get("demangled_name", None): - return demangled_name - - if name := self.get("name", None): - return name - - if idx := self.get("idx", None): - return idx - - return "" - - def __str__(self): - return f"{self.kind}({self.get_name()}) ({self.id})" - -@dataclass -class Nodes: - def __init__(self, nodes: Iterable[Node] = []): - self.ids = {node.id: node for node in nodes} - self.kinds = group_by(nodes, attrgetter("kind")) - - ids: dict[NodeID, Node] - kinds: dict[NodeKind, list[Node]] - - def __getitem__(self, id: NodeID): - return self.ids[id] - - def __iter__(self): - return self.ids.values().__iter__() - -@dataclass -class Edge: - id: EdgeID - src: NodeID - dst: NodeID - kinds: list[EdgeKind] - props: dict[str, Any] = field(default_factory=dict[str, Any]) - - T = TypeVar('T') - def get(self, key: str, default: T = None) -> str | T: - return self.props.get(key, default) - - def __getitem__(self, key: str): - return self.props[key] - - def __setitem__(self, key: str, value: Any): - self.props[key] = value - -@dataclass -class Edges: - def __init__(self, edges: Iterable[Edge] = []): - self.ids = {edge.id: edge for edge in edges} - self.kinds = {} - for e in edges: - for k in e.kinds: - self.kinds.setdefault(k, []).append(e) - - self.srcs = group_by(edges, attrgetter("src")) - self.dsts = group_by(edges, attrgetter("dst")) - - ids: dict[EdgeID, Edge] - kinds: dict[EdgeKind, list[Edge]] - srcs: dict[NodeID, list[Edge]] - dsts: dict[NodeID, list[Edge]] - - def __getitem__(self, id: EdgeID): - return self.ids[id] - - def __iter__(self): - return self.ids.values().__iter__() - -def demangle(names: Iterable[str]): - name_input = "\n".join(names) - res = subprocess.run(["c++filt"], input=name_input, stdout=subprocess.PIPE, text=True) - if res.returncode: - print(f"[RW]: ERROR: c++filt tool exited with code {res.returncode}") - print("[RW]: c++filt STDOUT:", res.stdout) - print("[RW]: c++filt STDERR:", res.stderr) - return names - - return res.stdout.split("\n") - -class FactParser: - def __init__(self, facts_file: Path): - self.facts_file = facts_file - - # def deserialize_edge(k: str, e: dict[str, Any]): - # # glaze serializes pairs as {first:second} - # d = json.loads(k) - # src, dst = d - - # sid = (mid, int(src)) - # did = (mid, int(dst)) - # return Edge((sid, did), sid, did, e["kinds"], {}) - - # Load Nodes - with (facts_file).open() as f: - all_nodes: list[Node] = [] - # all_edges: list[Edge] = [] - # When multiple modules are combined they will be separated by newlines. - for line in f: - facts = json.loads(line.strip('\n')) - - for mid, m in facts["modules"].items(): - mid = int(mid) - all_nodes += [ - Node((mid, int(id)), n["type"], n) - for id, n in m["nodes"].items() - ] - - # all_edges += [deserialize_edge(k, e) for k, e in m["edges"].items()] - - self.nodes = Nodes(all_nodes) - # self.edges = Edges(all_edges) - - def demangle_names(self): - with_names = [n for n in self.nodes if "name" in n.props] - demangled = demangle([n["name"] for n in with_names]) - for n, demangled_name in zip(with_names, demangled): - if n["name"] != demangled_name: - n["demangled_name"] = demangled_name - - def get_node_module_name(self, id: NodeID): - module_id = (id[0], id[0]) - name = self.nodes[module_id].props["source_file"] - return name - - def get_func_id(self, func_name: str, file_name: str = ""): - matches: list[NodeID] = [] - # First try true symbol names - for f in self.nodes.kinds["Function"]: - # try to get the function that we are precisely looking for - if func_name != f.get("name", ""): - continue - # Function.source_file is based on debug info, which may or may not be populated, but is more specific in the case of i.e. header files - if file_name in f.get( - "source_file", "" - ) or file_name in self.get_node_module_name(f.id): - return f.id - - # Next try demangled C++ symbol names - for f in self.nodes.kinds["Function"]: - # If we fail to get an exact match, try a substring match on demangled names - if func_name in f.props.get( - "demangled_name", "" - ) and file_name in self.get_node_module_name(f.id): - matches.append(f.id) - - match len(matches): - case 0: - return None - case 1: - pass - case _: - print( - f"[RW]: WARNING: multiple metadata matches for {file_name}:{func_name}" - ) - - return matches[0] - -@dataclass -class ReachToolResult: - nodes: list[Node] = field(default_factory=list[Node]) - edges: list[str] = field(default_factory=list[str]) - - def _as_cfg_path(self): - nodes = iter(self.nodes) - edges = iter(self.edges) - try: - yield str(next(nodes)) - while True: - edge = next(edges) - node = next(nodes) - yield f"{edge} -> {node}" - except StopIteration: - return - - def as_cfg_path(self): - return list(self._as_cfg_path()) - - def _as_call_path(self): - nodes = iter(self.nodes) - edges = iter(self.edges) - try: - yield str(next(nodes)) - while True: - edge = next(edges) - node = next(nodes) - if edge in ["Succ", "Contains"]: - continue - yield f"{edge} -> {node}" - except StopIteration: - return - - def as_call_path(self): - return list(self._as_call_path()) - - def _as_edges(self): - nodes = iter(self.nodes) - edges = iter(self.edges) - try: - source = next(nodes) - while True: - edge = next(edges) - destination = next(nodes) - yield (source, edge, destination) - source = destination - except StopIteration: - return - - def as_edges(self): - return list(self._as_edges()) - -# A mapping from dst_id to path list -ReachToolResults = dict[NodeID, list[ReachToolResult]] - -@dataclass -class ReachabilityResult: - sink: Sink - reachability: Reachability = Reachability.UNKNOWN - - # found in facts.facts - func_id: NodeID | None = None - - # from reach tool - paths: list[ReachToolResult] | None = None - - def update_from_fact_parser(self, fact_parser: 'FactParser') -> None: - """ - Looks for the function signature of a sink in nodeprops.facts - and updates it with its func_id - """ - func_id = fact_parser.get_func_id( - self.sink.affected_function, self.sink.affected_file - ) - if func_id is None: - self.reachability = Reachability.UNREACHABLE_NOT_FOUND - return - - file_name = fact_parser.get_node_module_name(func_id) - print(f"Found function '{self.sink.affected_function}' in module '{file_name}'") - - self.func_id = func_id - - def update_from_tool_results(self, tool_results: ReachToolResults): - assert self.func_id is not None - try: - self.paths = tool_results[self.func_id] - except KeyError: - return - - self.reachability = Reachability.REACHABLE if len(self.paths) else Reachability.UNREACHABLE_NO_PATH - - def get_dict(self) -> dict[str, Any]: - """ - Gets a dict of what we expect each sink to appear as - in our output json to TA2 - """ - match self.reachability: - case Reachability.UNREACHABLE_NOT_FOUND: - classification = "unreachable" - justification = { - "conclusion": "Not Found", - "reason": f"The affected function {self.sink.affected_file}:{self.sink.affected_function} was not found in compiled program metadata.", - } - case Reachability.UNREACHABLE_NO_PATH: - classification = "unreachable" - justification = { - "conclusion": "Not Reachable", - "reason": f"Control Flow Graph analysis found no paths to target function {self.sink.affected_file}:{self.sink.affected_function}.", - } - case Reachability.REACHABLE: - assert self.paths - classification = "potentially reachable" - justification: dict[str, str | list[str]] = { - "conclusion": "Statically Reachable", - "reason": "Control Flow Graph analysis found the following candidate path...", - "call_path": self.paths[0].as_call_path(), - "control_flow_path": self.paths[0].as_cfg_path() - } - case Reachability.UNREACHABLE_NOT_VULNERABLE: - classification = "unreachable" - justification = { - "conclusion": "Not Vulnerable", - "reason": "The package version is not considered vulnerable according to the supplied version information. It may or may not still be reachable." - } - case other: - print(f"[RW]: ERROR: Unexpected `reach` status \"{other}\"") - classification = "Unable to assess" - justification = { - "conclusion": "Error: internal tool failure" - } - - return { - "cve_id": self.sink.cve_id, - "classification": classification, - "justification": justification - } - -class ReachToolManager: - def __init__(self, facts_file: Path, src_id: str, tmp_reach_input_path: Path, reach_output_path: Path, reach_path: Path, reach_args: list[str]): - self.facts_file = facts_file - self.src_id = src_id - self.reach_output_path = reach_output_path - self.reach_path = reach_path - self.reach_args = reach_args - - self.tmp_reach_input_path = tmp_reach_input_path - self.tmp_reach_input_path.parent.mkdir(parents=True, exist_ok=True) # probably redundant - - def get_tool_input(self, results: list[ReachabilityResult]) -> dict[str, Any]: - return { - "cache": False, - "queries": [ - {"src": self.src_id, "dst": result.func_id} for result in results - ] - } - - def serialize_tool_input(self, results: list[ReachabilityResult]) -> None: - input = self.get_tool_input(results) - - with self.tmp_reach_input_path.open("w") as f: - json.dump(input, f, indent=4) - print(f"[RW]: Wrote {self.tmp_reach_input_path}") - - def invoke_reach(self) -> None: - cmd = [ - str(self.reach_path), - "-f", str(self.facts_file), - "-i", str(self.tmp_reach_input_path), - "-o", str(self.reach_output_path) - ] - cmd.extend(self.reach_args) - print(f"[RW]: Invoking reach '{' '.join(cmd)}'") - res = subprocess.run(cmd, capture_output=True, text=True) - - if(res.returncode != 0): - print(f"[RW]: ERROR: reach tool exited with code {res.returncode}") - print("[RW]: reach STDOUT:", res.stdout) - print("[RW]: reach STDERR:", res.stderr) - else: - print(f"[RW]: reach wrote output to {self.reach_output_path}") - - def get_tool_results(self, fact_parser: FactParser) -> ReachToolResults: - with open(self.reach_output_path, "r") as rf: - reach_file = json.load(rf) - print(f"[RW]: Read {self.reach_output_path}") - - # Convert list of KV pairs to map - def parse_result_path(nodes: list[list[int]], edges: list[str]): - return ReachToolResult(nodes=[fact_parser.nodes[tuple(id)] for id in nodes], edges=edges) - - return { - tuple(result["dst"]): [parse_result_path(**r) for r in result["paths"]] for result in reach_file["query_results"] - } - -class Orchestrator: - def __init__(self, facts_file: str, vuln_json_path: str, final_out_path: str, reach_bin_path: str|None, reach_args: list[str], cp_src_dir: str|None, graph_dir: str|None, entrypoint: str): - DEFAULT_REACH_PATH = "reach" - - self.reach_args = reach_args - self.facts_file = Path(facts_file) - self.vuln_json_path = Path(vuln_json_path) - self.final_out_path = ( - Path(final_out_path) - if final_out_path - else Path(vuln_json_path).with_suffix(".reach.json") - ) - self.reach_bin_path = Path(reach_bin_path) if reach_bin_path else DEFAULT_REACH_PATH - self.cp_src_dir = Path(cp_src_dir) if cp_src_dir else None - - self.fact_parser = FactParser(self.facts_file) - self.fact_parser.demangle_names() - - self.output_graph_path = graph_dir - - self.entrypoint = entrypoint - - # Load vulnerabilities.json - with open(self.vuln_json_path, "r") as vj: - vuln_json = json.load(vj) - - # Initialize results - sinks = [Sink.from_vuln_dict(vuln) for vuln in vuln_json["vulnerabilities"]] - self.results = [ReachabilityResult(sink) for sink in sinks] - - def parse_vulnerable_results(self): - # If we have a source code directory, populate the package version - if self.cp_src_dir is None: - print("[RW]: WARNING: No source code directory provided, package versions will not be populated.") - return - cp_src_dir = self.cp_src_dir - - def is_vulnerable(vuln_version: str, actual_version: str) -> bool: - vrange = GenericVersionRange.from_string(f"vers:generic/{vuln_version}") - vstr = SemverVersion(actual_version) - return vrange.contains(vstr) - - def get_version(package_name: str): - # First, check for overlay ports - vcpkg_json = cp_src_dir / "vcpkg-overlays/ports" / package_name / "vcpkg.json" - - # If no port, try root - if not vcpkg_json.exists(): - vcpkg_json = cp_src_dir / "vcpkg.json" - - with vcpkg_json.open("r") as f: - vcpkg_data = json.load(f) - - # Check that the name matches - if vcpkg_data.get("name", None) != package_name: - return None, vcpkg_json - - # get the version from the vcpkg.json - return vcpkg_data.get("version", None), vcpkg_json - - # for sink in sinks: - for result in self.results: - name = result.sink.package_name - vulnerable_version_string = result.sink.vulnerable_package_version - - vcpkg_version_string, vcpkg_json = get_version(name) - if vcpkg_version_string is None: - print(f"[RW]: WARNING: Could not find vcpkg.json for package '{name}' in {self.cp_src_dir / 'vcpkg-overlays/ports' / name}") - continue - - print(f"[RW]: Populated package version for '{name}' from {vcpkg_json}: {vcpkg_version_string}") - result.sink.package_version = vcpkg_version_string - - # check if there is a version match - if is_vulnerable(vulnerable_version_string, vcpkg_version_string): - print(f"[RW]: Package version '{vcpkg_version_string}' is considered vulnerable according to '{vulnerable_version_string}'") - else: - print(f"[RW]: Package version '{vcpkg_version_string}' is not considered vulnerable according to '{vulnerable_version_string}'") - result.reachability = Reachability.UNREACHABLE_NOT_VULNERABLE - - def get_unsolved_results(self): - return [result for result in self.results if result.reachability == Reachability.UNKNOWN] - - def parse_facts(self): - "Update results from fact_parser to get func_id" - - # NOTE: we assume that we can always enter - # the desired basic block from an 'fmain' - src = self.fact_parser.get_func_id(self.entrypoint) - assert src is not None, f"Could not find source function '{self.entrypoint}'" - - file_name = self.fact_parser.get_node_module_name(src) - - print(f"Found function '{self.entrypoint}' in module '{file_name}'") - self.src = src - for result in self.results: - result.update_from_fact_parser(self.fact_parser) - - def run_reach_tool(self): - "Run reach tool and update Results" - # TODO (optional): implement flags to specify the intermediate file placements - - """ - https://stackoverflow.com/a/48710609 - """ - src = self.src - def is_docker(): - def text_in_file(text: str, filename: str): - try: - with open(filename, encoding='utf-8') as lines: - return any(text in line for line in lines) - except OSError: - return False - cgroup = '/proc/self/cgroup' - return os.path.exists('/.dockerenv') or text_in_file('docker', cgroup) - - if os.getenv("CI") is not None or is_docker(): - tmp_in = Path("/tmp/reach_wrap_input.json") - tmp_out = Path("/tmp/reach_wrap_output.json") - else: - tmp_in = Path("reach_wrap_input.json") - tmp_out = Path("reach_wrap_output.json") - - self.input_manager = ReachToolManager(self.facts_file, src, tmp_in, tmp_out, self.reach_bin_path, self.reach_args) - - unsolved_results = self.get_unsolved_results() - self.input_manager.serialize_tool_input(unsolved_results) - - # - # HACK: On problems with large facts, reach wrapper uses - # a truly ridiculous amount of memory. In order to - # prevent the EBOSS CI runners from OOM-ing, we - # destroy FactParser before invoking reach, and - # then pay the cost of re-building it twice after - # reach exits. - # - self.fact_parser = None - gc.collect() - self.input_manager.invoke_reach() - self.fact_parser = FactParser(self.facts_file) - self.fact_parser.demangle_names() - - tool_results = self.input_manager.get_tool_results(fact_parser=self.fact_parser) - for result in unsolved_results: - result.update_from_tool_results(tool_results) - - def serialize_output(self): - "Print results as final output" - data = { - "reachability_results": [result.get_dict() for result in self.results] - } - self.final_out_path.parent.mkdir(parents=True, exist_ok=True) # probably redundant - with self.final_out_path.open("w") as f: - json.dump(data, f, indent=4) - print(f"[RW]: Wrote {self.final_out_path}.") - - def serialize_as_graph(self): - if not self.output_graph_path: - return - - os.makedirs(self.output_graph_path, exist_ok=True) - - # sink nodes - with open(Path(self.output_graph_path, "nodeprops.facts"), "w") as nodeprops_file: - for result in self.results: - if result.reachability is Reachability.UNREACHABLE_NOT_FOUND: - continue - - nodeprops_file.write(f"{result.func_id},\"vulnerability_id\",{result.sink.cve_id}\n") - nodeprops_file.write(f"{result.func_id},\"reachable\",{ True if result.reachability == Reachability.REACHABLE else False}\n") - - # Edges - with ( - open(Path(self.output_graph_path, "edges.facts"), "w") as edges_file, - open(Path(self.output_graph_path, "edgeprops.facts"), "w") as edgeprops_file, - ): - i = 0 - for result in self.results: - if result.reachability is not Reachability.REACHABLE: - continue - - assert result.paths is not None - for path in result.paths: - for source, edge, destination in path.as_edges(): - edges_file.write(f"{i},{'ReachablePath'},{source.id},{destination.id}\n") - edgeprops_file.write(f"{i},{'kind'},{edge}\n") - i+=1 - - def main(self): - self.parse_vulnerable_results() - self.parse_facts() - - self.run_reach_tool() - self.serialize_output() - self.serialize_as_graph() - -def main(): - parser = argparse.ArgumentParser( - description="Reach tool wrapper used to manipulate inputs and outputs to desired forms" - ) - - parser.add_argument( - "-i", - "--input", - type=str, - help="the vulnerabilities.json path", - required=True - ) - - parser.add_argument( - "-o", "--output", type=str, help="the path to write the output file to" - ) - - parser.add_argument( - "-f", - "--facts", - type=str, - help="the file containing the facts", - required=True - ) - - parser.add_argument( - "-r", - "--reach", - type=str, - help="the path to the reach binary", - default=None - ) - - parser.add_argument( - "-s", - "--src", - type=str, - help="the folder containing the source code for the cp, it should have a vcpkg-overlays folder", - default=None, - ) - - parser.add_argument( - "-a", - "--args", - type=str, - help="additional arguments passed verbatim to `reach`", - nargs=argparse.REMAINDER, - default=[] - ) - - parser.add_argument( - "-g", - "--graph", - type=str, - help="output a facts file in shared volume that can be imported into neo4j, this argument specifies the path", - default=None, - required=False - ) - - parser.add_argument( - "-e", - "--entry", - type=str, - help="The function to use as the entrypoint for reachability analysis. Defaults to `main`", - default="main", - required=False - ) - - args = parser.parse_args() - # reach_out = Path("/tmp/reach_out.json") - - Orchestrator(args.facts, args.input, args.output, args.reach, args.args, args.src, args.graph, args.entry).main() - -if __name__ == "__main__": - main() diff --git a/resolve-cli/src/resolve/reach/Cargo.lock b/resolve-cli/src/resolve/reach/Cargo.lock index 8a527d55b..742f4aae8 100644 --- a/resolve-cli/src/resolve/reach/Cargo.lock +++ b/resolve-cli/src/resolve/reach/Cargo.lock @@ -352,7 +352,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" [[package]] -name = "reach" +name = "resolve-reach" version = "0.1.0" dependencies = [ "clap", diff --git a/resolve-cli/src/resolve/reach/Cargo.toml b/resolve-cli/src/resolve/reach/Cargo.toml index f904a0879..3158dcabb 100644 --- a/resolve-cli/src/resolve/reach/Cargo.toml +++ b/resolve-cli/src/resolve/reach/Cargo.toml @@ -1,5 +1,5 @@ [package] -name = "reach" +name = "resolve-reach" version = "0.1.0" edition = "2024" build = "build.rs" diff --git a/resolve-cli/src/resolve/reach/build.rs b/resolve-cli/src/resolve/reach/build.rs index 00134f79f..7bab10d7b 100644 --- a/resolve-cli/src/resolve/reach/build.rs +++ b/resolve-cli/src/resolve/reach/build.rs @@ -5,7 +5,7 @@ fn main() { let library_dir = PathBuf::from( env::var_os("RESOLVE_LIBREACH_DIR").expect( - "RESOLVE_LIBREACH_DIR is not set; build through the CMake reach-rs target or set it to the native library directory", + "RESOLVE_LIBREACH_DIR is not set; build through the CMake resolve-reach target or set it to the native library directory", ), ); diff --git a/resolve-cli/src/resolve/reach/src/main.rs b/resolve-cli/src/resolve/reach/src/main.rs index 9b225511d..13905e20b 100644 --- a/resolve-cli/src/resolve/reach/src/main.rs +++ b/resolve-cli/src/resolve/reach/src/main.rs @@ -22,18 +22,22 @@ mod vcpkg; mod vulnerability; #[derive(Parser, Debug)] +#[command( + name = "resolve-reach", + about = "Analyze static reachability for known vulnerabilities" +)] struct Args { /// Input vulnerabilities.json #[arg(short, long)] input: PathBuf, - /// Files containing facts (ELF, .so, .facts) + /// Files or directories containing facts (ELF, .so, .facts) #[arg(short, long, required = true, num_args=1, action= ArgAction::Append)] facts: Vec, /// The file to write the final report into - #[arg(short, long, default_value = "reach.json")] // TODO: .reach.json - output: PathBuf, + #[arg(short, long)] + output: Option, /// Source tree containing vcpkg-overlays #[arg(short, long)] @@ -66,8 +70,50 @@ fn load_vuln_json(path: &Path) -> Result { .map_err(|error| format!("failed to parse '{}': {error}", path.display())) } -fn load_facts(paths: &[PathBuf]) -> Result { - FactsBuf::read_files(paths).map_err(|error| format!("failed to load facts: {error}")) +fn expand_facts_paths(paths: &[PathBuf]) -> Result, String> { + let mut files = Vec::new(); + + for path in paths { + if !path.is_dir() { + files.push(path.clone()); + continue; + } + + let mut directory_files = fs::read_dir(path) + .map_err(|error| format!("failed to read '{}': {error}", path.display()))? + .map(|entry| { + entry + .map(|entry| entry.path()) + .map_err(|error| format!("failed to read '{}': {error}", path.display())) + }) + .collect::, _>>()?; + directory_files.retain(|file| { + file.is_file() + && file + .extension() + .is_some_and(|extension| extension == "facts") + }); + directory_files.sort(); + + if directory_files.is_empty() { + return Err(format!( + "facts directory '{}' contains no .facts files", + path.display() + )); + } + files.extend(directory_files); + } + + Ok(files) +} + +fn load_facts(paths: &[PathBuf]) -> Result<(FactsBuf, usize), String> { + let files = expand_facts_paths(paths)?; + let count = files.len(); + let facts = + FactsBuf::read_files(&files).map_err(|error| format!("failed to load facts: {error}"))?; + + Ok((facts, count)) } fn load_dlsym_log(path: &Path) -> Result, String> { @@ -82,6 +128,9 @@ fn load_dlsym_log(path: &Path) -> Result, String> { fn run() -> Result<(), String> { let args = Args::parse(); let input = load_vuln_json(&args.input)?; + let output = args + .output + .unwrap_or_else(|| args.input.with_extension("reach.json")); let mut analyses: Vec = input.vulnerabilities.into_iter().map(Into::into).collect(); @@ -93,7 +142,7 @@ fn run() -> Result<(), String> { ); } - let facts = load_facts(&args.facts)?; + let (facts, facts_file_count) = load_facts(&args.facts)?; let module_count = facts .view() .modules() @@ -102,7 +151,7 @@ fn run() -> Result<(), String> { println!( "[REACH] Loaded {module_count} facts modules from {} input files.", - args.facts.len() + facts_file_count ); let functions = FunctionIndex::build(&facts)?; @@ -118,7 +167,7 @@ fn run() -> Result<(), String> { &args.entry, &graph_options, )?; - write_report(&args.output, &analyses, &facts, &functions)?; + write_report(&output, &analyses, &facts, &functions)?; Ok(()) } diff --git a/resolve-cli/uv.lock b/resolve-cli/uv.lock index def3e551c..082561001 100644 --- a/resolve-cli/uv.lock +++ b/resolve-cli/uv.lock @@ -636,15 +636,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/47/4f/4a617ee93d8208d2bcf26b2d8b9402ceaed03e3853c754940e2290fed063/ollama-0.6.1-py3-none-any.whl", hash = "sha256:fc4c984b345735c5486faeee67d8a265214a31cbb828167782dc642ce0a2bf8c", size = 14354, upload-time = "2025-11-13T23:02:16.292Z" }, ] -[[package]] -name = "packaging" -version = "26.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/65/ee/299d360cdc32edc7d2cf530f3accf79c4fca01e96ffc950d8a52213bd8e4/packaging-26.0.tar.gz", hash = "sha256:00243ae351a257117b6a241061796684b084ed1c516a08c48a3f7e147a9d80b4", size = 143416, upload-time = "2026-01-21T20:50:39.064Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/b7/b9/c538f279a4e237a006a2c98387d081e9eb060d203d8ed34467cc0f0b9b53/packaging-26.0-py3-none-any.whl", hash = "sha256:b36f1fef9334a5588b4166f8bcd26a14e521f2b55e6b9de3aaa80d3ff7a37529", size = 74366, upload-time = "2026-01-21T20:50:37.788Z" }, -] - [[package]] name = "propcache" version = "0.4.1" @@ -845,6 +836,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f7/07/34573da085946b6a313d7c42f82f16e8920bfd730665de2d11c0c37a74b5/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:76d0819de158cd855d1cbb8fcafdf6f5cf1eb8e470abe056d5d161106e38062b", size = 2139017, upload-time = "2025-11-04T13:42:59.471Z" }, ] +[[package]] +name = "pyelftools" +version = "0.32" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b9/ab/33968940b2deb3d92f5b146bc6d4009a5f95d1d06c148ea2f9ee965071af/pyelftools-0.32.tar.gz", hash = "sha256:6de90ee7b8263e740c8715a925382d4099b354f29ac48ea40d840cf7aa14ace5", size = 15047199, upload-time = "2025-02-19T14:20:05.549Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/af/43/700932c4f0638c3421177144a2e86448c0d75dbaee2c7936bda3f9fd0878/pyelftools-0.32-py3-none-any.whl", hash = "sha256:013df952a006db5e138b1edf6d8a68ecc50630adbd0d83a2d41e7f846163d738", size = 188525, upload-time = "2025-02-19T14:19:59.919Z" }, +] + [[package]] name = "requests" version = "2.33.1" @@ -860,15 +860,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d7/8e/7540e8a2036f79a125c1d2ebadf69ed7901608859186c856fa0388ef4197/requests-2.33.1-py3-none-any.whl", hash = "sha256:4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a", size = 64947, upload-time = "2026-03-30T16:09:13.83Z" }, ] -[[package]] -name = "pyelftools" -version = "0.32" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/b9/ab/33968940b2deb3d92f5b146bc6d4009a5f95d1d06c148ea2f9ee965071af/pyelftools-0.32.tar.gz", hash = "sha256:6de90ee7b8263e740c8715a925382d4099b354f29ac48ea40d840cf7aa14ace5", size = 15047199, upload-time = "2025-02-19T14:20:05.549Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/af/43/700932c4f0638c3421177144a2e86448c0d75dbaee2c7936bda3f9fd0878/pyelftools-0.32-py3-none-any.whl", hash = "sha256:013df952a006db5e138b1edf6d8a68ecc50630adbd0d83a2d41e7f846163d738", size = 188525, upload-time = "2025-02-19T14:19:59.919Z" }, -] - [[package]] name = "resolve-cli" version = "0.1.0" @@ -880,7 +871,6 @@ dependencies = [ { name = "ollama" }, { name = "pydantic" }, { name = "pyelftools" }, - { name = "univers" }, ] [package.metadata] @@ -890,27 +880,7 @@ requires-dist = [ { name = "google-genai", specifier = ">=1.69.0" }, { name = "ollama", specifier = ">=0.6.1" }, { name = "pydantic", specifier = ">=2.12.5" }, - { name = "pyelftools", specifier = ">=0.32" }, - { name = "univers", specifier = ">=31.1.0" }, -] - -[[package]] -name = "semantic-version" -version = "2.10.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/7d/31/f2289ce78b9b473d582568c234e104d2a342fd658cc288a7553d83bb8595/semantic_version-2.10.0.tar.gz", hash = "sha256:bdabb6d336998cbb378d4b9db3a4b56a1e3235701dc05ea2690d9a997ed5041c", size = 52289, upload-time = "2022-05-26T13:35:23.454Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/6a/23/8146aad7d88f4fcb3a6218f41a60f6c2d4e3a72de72da1825dc7c8f7877c/semantic_version-2.10.0-py2.py3-none-any.whl", hash = "sha256:de78a3b8e0feda74cabc54aab2da702113e33ac9d9eb9d2389bcf1f58b7d9177", size = 15552, upload-time = "2022-05-26T13:35:21.206Z" }, -] - -[[package]] -name = "semver" -version = "3.0.4" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/72/d1/d3159231aec234a59dd7d601e9dd9fe96f3afff15efd33c1070019b26132/semver-3.0.4.tar.gz", hash = "sha256:afc7d8c584a5ed0a11033af086e8af226a9c0b206f313e0301f8dd7b6b589602", size = 269730, upload-time = "2025-01-24T13:19:27.617Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/a6/24/4d91e05817e92e3a61c8a21e08fd0f390f5301f1c448b137c57c4bc6e543/semver-3.0.4-py3-none-any.whl", hash = "sha256:9c824d87ba7f7ab4a1890799cec8596f15c1241cb473404ea1cb0c55e4b04746", size = 17912, upload-time = "2025-01-24T13:19:24.949Z" }, ] [[package]] @@ -952,21 +922,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" }, ] -[[package]] -name = "univers" -version = "31.1.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "attrs" }, - { name = "packaging" }, - { name = "semantic-version" }, - { name = "semver" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/78/83/7304856c01eadc320147818aa8b53338955d636db66e2bfdddea8941b527/univers-31.1.0.tar.gz", hash = "sha256:5c617edd03657f02ddaa84db0b66a11134aa604fe04b06e7c828483f089c9da6", size = 294626, upload-time = "2025-09-11T13:31:02.265Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/f7/f1/80d9c7c72511c3b792b6bd637334e9ae76ddd2a6e5188876c265bc44aefc/univers-31.1.0-py3-none-any.whl", hash = "sha256:e299829882b058d9355e244609739b7c78ab72da5c786bf7f681ffc9db523665", size = 96759, upload-time = "2025-09-11T13:31:00.928Z" }, -] - [[package]] name = "urllib3" version = "2.6.3" diff --git a/resolve-facts/CMakeLists.txt b/resolve-facts/CMakeLists.txt index ac91088b9..80950c789 100644 --- a/resolve-facts/CMakeLists.txt +++ b/resolve-facts/CMakeLists.txt @@ -110,12 +110,10 @@ file(GLOB_RECURSE SRC # Build Targets add_library(resolve_facts_llvm STATIC libs/resolve_facts_llvm/binary_facts_llvm.cpp - libs/resolve_facts_llvm/resolve_facts_llvm.cpp ) target_include_directories(resolve_facts_llvm SYSTEM PUBLIC ${LLVM_INCLUDE_DIRS}) find_package(Threads REQUIRED) target_link_libraries(resolve_facts_llvm PUBLIC - resolve_facts facts_rs Threads::Threads ${CMAKE_DL_LIBS} @@ -143,12 +141,6 @@ install(DIRECTORY include/resolve_facts_llvm DESTINATION ${CMAKE_INSTALL_INCLUDE ###################################################################### # REACH -# Collect source files for checks -file(GLOB_RECURSE SRC - "${CMAKE_CURRENT_SOURCE_DIR}/src/reach/*.cpp" - "${CMAKE_CURRENT_SOURCE_DIR}/src/reach/*.hpp" -) - file(GLOB_RECURSE LIB "${CMAKE_CURRENT_SOURCE_DIR}/include/reach/*.h" "${CMAKE_CURRENT_SOURCE_DIR}/include/reach/*.hpp" @@ -186,17 +178,11 @@ target_link_libraries(libreach PUBLIC target_compile_features(libreach PUBLIC cxx_std_23) -# reach executable -add_executable(reach src/reach/main.cpp) - -target_link_libraries(reach PRIVATE libreach json argparse) - if(COMMAND resolve_add_check_targets) resolve_add_check_targets(libreach ${LIB}) - resolve_add_check_targets(reach ${SRC}) endif() -install(TARGETS reach libreach EXPORT resolve_facts_targets) +install(TARGETS libreach EXPORT resolve_facts_targets) install(DIRECTORY include/reach DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) ###################################################################### diff --git a/resolve-facts/README.md b/resolve-facts/README.md index c3e72624f..04b14d287 100644 --- a/resolve-facts/README.md +++ b/resolve-facts/README.md @@ -5,12 +5,12 @@ # Resolve facts -Tools for creating and querying RESOLVE binary metadata, including the `reach` tool, which provides fast graph reachability for RESOLVE. +Libraries for creating and querying RESOLVE binary metadata. The `libreach` library provides graph construction and path search for the `resolve reach` command. **Full documentation:** - Facts: -- `reach` tool: +- Reachability analysis: ## Future Improvements diff --git a/resolve-facts/include/reach/distmap.hpp b/resolve-facts/include/reach/distmap.hpp index 7e21a7448..4ed8ca0bc 100644 --- a/resolve-facts/include/reach/distmap.hpp +++ b/resolve-facts/include/reach/distmap.hpp @@ -10,6 +10,7 @@ #include #include "reach/facts.hpp" +#include "json/json.hpp" using NNodeId = resolve_facts::NamespacedNodeId; diff --git a/resolve-facts/include/reach/facts.hpp b/resolve-facts/include/reach/facts.hpp index 1a64108dc..2f406d751 100644 --- a/resolve-facts/include/reach/facts.hpp +++ b/resolve-facts/include/reach/facts.hpp @@ -5,15 +5,13 @@ #pragma once -#include -#include +#include +#include #include #include #include #include -#include "json/json.hpp" - #include "resolve_facts/resolve_facts.hpp" using NamespacedNodeId = resolve_facts::NamespacedNodeId; @@ -89,26 +87,6 @@ namespace dlsym { struct loaded_symbol { std::string symbol; std::string library; - bool operator==(const loaded_symbol &rhs) const { - return symbol == rhs.symbol && library == rhs.library; - }; -}; - -struct log { - std::vector loaded_symbols; }; -NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(loaded_symbol, symbol, library); -NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(log, loaded_symbols); - -inline std::optional -load_log_from_file(const std::filesystem::path &path) { - std::ifstream f(path); - if (!f.is_open()) { - return {}; - } - nlohmann::json j; - f >> j; - return j.template get(); -} } // namespace dlsym diff --git a/resolve-facts/include/resolve_facts_llvm/LLVMFacts.hpp b/resolve-facts/include/resolve_facts_llvm/LLVMFacts.hpp deleted file mode 100644 index 3f7750884..000000000 --- a/resolve-facts/include/resolve_facts_llvm/LLVMFacts.hpp +++ /dev/null @@ -1,212 +0,0 @@ -/* - * Copyright (c) 2025 Riverside Research. - * LGPL-3; See LICENSE.txt in the repo root for details. - */ - -#ifndef RESOLVE_LLVM_LLVMFACTS_HPP -#define RESOLVE_LLVM_LLVMFACTS_HPP - -#include "resolve_facts/resolve_facts.hpp" - -#include "llvm/IR/BasicBlock.h" -#include "llvm/IR/Constants.h" -#include "llvm/IR/Function.h" -#include "llvm/IR/GlobalVariable.h" -#include "llvm/IR/Instruction.h" -#include "llvm/IR/Module.h" -#include "llvm/Support/FileSystem.h" - -#include - -using ProgramFacts = resolve_facts::ProgramFacts; -using ModuleFacts = resolve_facts::ModuleFacts; -using Node = resolve_facts::Node; -using NodeId = resolve_facts::NodeId; -using NodeType = resolve_facts::NodeType; -using EdgeId = resolve_facts::EdgeId; - -class LLVMFacts { - ProgramFacts &facts; - NodeId next_node_id = 1; - - std::unordered_map moduleIDs; - std::unordered_map functionIDs; - std::unordered_map basicBlockIDs; - std::unordered_map argumentIDs; - std::unordered_map instructionIDs; - std::unordered_map globalVarIDs; - - void recordNewModule(const NodeId &id, const size_t size_hint) { - ModuleFacts mf{}; - // Try to avoid reallocations - mf.nodes.reserve(size_hint); - mf.edges.reserve(2 * size_hint); - - facts.modules[id] = mf; - } - - /// Record a node fact. - void recordNode(const NodeId &module, const NodeId &id, - const NodeType &type) { - Node node{.type = type}; - facts.modules.at(module).nodes.emplace(id, node); - } - - /// Record a node property. - template - void recordNodeProp(const NodeId &module, const NodeId &nodeID, - F &&update_func) { - auto &mf = facts.modules.at(module); - update_func(mf.nodes.at(nodeID)); - } - - /// Record an edge fact. - template - void recordEdge(const NodeId &module, const NodeId &srcID, - const NodeId &tgtID, F &&update_func) { - auto pair = EdgeId(srcID, tgtID); - auto &mf = facts.modules.at(module); - auto [it, exists] = mf.edges.try_emplace(pair); - update_func(it->second); - } - -public: - LLVMFacts(ProgramFacts &facts) : facts(facts) {} - - NodeId addNode(const llvm::Module &M) { - if (moduleIDs.find(&M) == moduleIDs.end()) { - - llvm::SmallString<128> src_path = llvm::StringRef(M.getSourceFileName()); - llvm::sys::fs::make_absolute(src_path); - - std::string src = (std::string)src_path; - size_t hash = std::hash{}(src); - auto id = (NodeId)hash; - - // llvm::errs() << "Creating new module: " << id << "\n"; - - moduleIDs[&M] = id; - - // Estimate how many total nodes we will be creating to prevent rehashes - auto instrs = M.getInstructionCount(); - recordNewModule(id, 2 * instrs); - recordNode(id, id, NodeType::Module); - return id; - } - return moduleIDs[&M]; - } - - NodeId getModuleId(const llvm::Module &m) { return addNode(m); } - - template NodeId getModuleId(const T &i) { - const llvm::Module *module; - - constexpr bool parent_is_module = - std::is_same_v; - constexpr bool is_argument = std::is_same_v; - if constexpr (parent_is_module) { - module = i.getParent(); - } else if constexpr (is_argument) { - module = i.getParent()->getParent(); - } else { - module = i.getModule(); - } - - assert(module); - return addNode(*module); - } - - template static std::size_t getIndexInParent(const T &item) { - const auto &parent = *item.getParent(); - return std::distance(parent.begin(), item.getIterator()); - } - - NodeId addNode(const llvm::GlobalVariable &GV) { - if (globalVarIDs.find(&GV) == globalVarIDs.end()) { - auto id = next_node_id; - next_node_id += 1; - auto module_id = getModuleId(GV); - - globalVarIDs[&GV] = id; - recordNode(module_id, id, NodeType::GlobalVariable); - return id; - } - return globalVarIDs[&GV]; - } - - NodeId addNode(const llvm::Function &F) { - if (functionIDs.find(&F) == functionIDs.end()) { - auto id = next_node_id; - next_node_id += 1; - auto module_id = getModuleId(F); - - functionIDs[&F] = id; - recordNode(module_id, id, NodeType::Function); - return id; - } - return functionIDs[&F]; - } - - NodeId addNode(const llvm::Argument &A) { - if (argumentIDs.find(&A) == argumentIDs.end()) { - auto id = next_node_id; - next_node_id += 1; - auto module_id = getModuleId(A); - - argumentIDs[&A] = id; - recordNode(module_id, id, NodeType::Argument); - return id; - } - return argumentIDs[&A]; - } - - NodeId addNode(const llvm::BasicBlock &BB) { - if (basicBlockIDs.find(&BB) == basicBlockIDs.end()) { - auto id = next_node_id; - next_node_id += 1; - auto module_id = getModuleId(BB); - - basicBlockIDs[&BB] = id; - recordNode(module_id, id, NodeType::BasicBlock); - return id; - } - return basicBlockIDs[&BB]; - } - - NodeId addNode(const llvm::Instruction &I) { - if (instructionIDs.find(&I) == instructionIDs.end()) { - auto id = next_node_id; - next_node_id += 1; - auto module_id = getModuleId(I); - - instructionIDs[&I] = id; - recordNode(module_id, id, NodeType::Instruction); - return id; - } - return instructionIDs[&I]; - } - - template - void addEdge(S &src, D &dst, F &&update_func) { - auto m1 = getModuleId(src); - auto m2 = getModuleId(dst); - assert(m1 == m2); - - addEdge(m1, addNode(src), addNode(dst), update_func); - } - - template - void addEdge(NodeId module, NodeId src, NodeId dst, F &&update_func) { - recordEdge(module, src, dst, update_func); - } - - template - void addNodeProp(const N &node, F &&update_func) { - auto module_id = getModuleId(node); - recordNodeProp(module_id, addNode(node), update_func); - } - - const std::string serialize() const { return facts.serialize(); } -}; - -#endif // RESOLVE_LLVM_LLVMFACTS_HPP diff --git a/resolve-facts/include/resolve_facts_llvm/resolve_facts_llvm.hpp b/resolve-facts/include/resolve_facts_llvm/resolve_facts_llvm.hpp deleted file mode 100644 index 9fa4da0c6..000000000 --- a/resolve-facts/include/resolve_facts_llvm/resolve_facts_llvm.hpp +++ /dev/null @@ -1,48 +0,0 @@ -/* - * Copyright (c) 2025 Riverside Research. - * LGPL-3; See LICENSE.txt in the repo root for details. - */ - -#include "resolve_facts/resolve_facts.hpp" -#include "resolve_facts_llvm/LLVMFacts.hpp" - -#include "llvm/ADT/SmallString.h" -#include "llvm/ADT/SmallVector.h" -#include "llvm/IR/BasicBlock.h" -#include "llvm/IR/CFG.h" -#include "llvm/IR/Constants.h" -#include "llvm/IR/DebugInfoMetadata.h" -#include "llvm/IR/Function.h" -#include "llvm/IR/GlobalVariable.h" -#include "llvm/IR/Instruction.h" -#include "llvm/IR/Instructions.h" -#include "llvm/IR/LLVMContext.h" -#include "llvm/IR/Module.h" -#include "llvm/IR/PassManager.h" -#include "llvm/Passes/PassBuilder.h" -#include "llvm/Passes/PassPlugin.h" -#include "llvm/Support/Compression.h" -#include "llvm/Support/FileSystem.h" -#include "llvm/Support/Path.h" -#include "llvm/Support/raw_ostream.h" -#include "llvm/Transforms/Utils/ModuleUtils.h" - -using namespace llvm; - -namespace resolve { -extern ProgramFacts all_facts; -extern LLVMFacts facts; - -std::string debugLocToString(DebugLoc dbgLoc); - -std::string typeToString(const Type &type); - -void getGlobalFacts(GlobalVariable &G); - -void getFunctionFacts(Function &F); - -void getModuleFacts(Module &M); - -// Embed the accumulated facts into custom ELF sections. -void embedFacts(Module &M); -} // namespace resolve diff --git a/resolve-facts/libs/resolve_facts_llvm/resolve_facts_llvm.cpp b/resolve-facts/libs/resolve_facts_llvm/resolve_facts_llvm.cpp deleted file mode 100644 index d309a60df..000000000 --- a/resolve-facts/libs/resolve_facts_llvm/resolve_facts_llvm.cpp +++ /dev/null @@ -1,183 +0,0 @@ -/* - * Copyright (c) 2025 Riverside Research. - * LGPL-3; See LICENSE.txt in the repo root for details. - */ - -#include "resolve_facts_llvm/resolve_facts_llvm.hpp" - -#include // For std::getenv -#include -#include -#include -#include -#include -#include - -using namespace llvm; - -using Linkage = resolve_facts::Linkage; -using CallType = resolve_facts::CallType; -using EdgeKind = resolve_facts::EdgeKind; - -ProgramFacts resolve::all_facts; -LLVMFacts resolve::facts(resolve::all_facts); - -std::string resolve::debugLocToString(DebugLoc dbgLoc) { - auto line = std::to_string(dbgLoc.getLine()); - auto col = std::to_string(dbgLoc.getCol()); - return line + ":" + col; -} - -std::string resolve::typeToString(const Type &type) { - std::string str; - llvm::raw_string_ostream out(str); - type.print(out); - return str; -} - -void resolve::getGlobalFacts(GlobalVariable &G) { - facts.addNode(G); - facts.addNodeProp(G, [&](auto& node) { - node.name = G.getName().str(); - node.linkage = (G.hasExternalLinkage() ? Linkage::ExternalLinkage : Linkage::Other); - }); -} - -static std::string getFunctionNameFromDebugInfo(Function &F) { - // Each function may have a DISubprogram attached - if (auto *SP = F.getSubprogram()) { - if (auto *File = SP->getFile()) { - return (File->getDirectory() + "/" + File->getFilename()).str(); - } - } - return ""; -} - -void resolve::getFunctionFacts(Function &F) { - facts.addNode(F); - facts.addNodeProp(F, [&](auto& node) { - node.name = F.getName().str(); - node.linkage = (F.hasExternalLinkage() ? Linkage::ExternalLinkage : Linkage::Other); - node.function_type = typeToString(*F.getFunctionType()); - auto name = getFunctionNameFromDebugInfo(F); - if (name != "") { - node.source_file = name; - } - if (F.hasAddressTaken()) { - node.address_taken = true; - } - }); - - if (F.isDeclaration()) - return; - - facts.addEdge(F, F.getEntryBlock(), [](auto& edge) { edge.kinds.push_back(EdgeKind::EntryPoint); }); - - for (Argument &A : F.args()) { - facts.addEdge(F, A, [&](auto& edge) { edge.kinds.push_back(EdgeKind::Contains); }); - facts.addNodeProp(A, [&](auto& node) { node.idx = A.getArgNo(); }); - } - - for (BasicBlock &BB : F) { - facts.addEdge(F, BB, [&](auto& edge) { edge.kinds.push_back(EdgeKind::Contains); }); - facts.addNodeProp(BB, [&](auto& node) { - node.idx = LLVMFacts::getIndexInParent(BB); - if (BB.hasName()) { - node.name = BB.getName().str(); - } - }); - - // Control flow Edges - for (BasicBlock *Succ : successors(&BB)) { - facts.addEdge(BB, *Succ, [&](auto& edge) { edge.kinds.push_back(EdgeKind::ControlFlowTo); }); - } - - for (Instruction &I : BB) { - facts.addEdge(BB, I, [&](auto& edge) { edge.kinds.push_back(EdgeKind::Contains); }); - facts.addNodeProp(I, [&](auto& node) { - node.opcode = I.getOpcodeName(); - if (auto dbgLoc = I.getDebugLoc()) { - node.source_loc = debugLocToString(dbgLoc); - } - }); - - // Data–flow edges: from each operand (if an instruction) to I. - for (Value *op : I.operands()) { - if (Instruction *opI = dyn_cast(op)) { - facts.addEdge(*opI, I, [&](auto& edge) { edge.kinds.push_back(EdgeKind::DataFlowTo); }); - } else if (Argument *opA = dyn_cast(op)) { - facts.addEdge(*opA, I, [&](auto& edge) { edge.kinds.push_back(EdgeKind::DataFlowTo); }); - } else if (GlobalVariable *opG = dyn_cast(op)) { - facts.addEdge(I, *opG, [&](auto& edge) { edge.kinds.push_back(EdgeKind::References); }); - } else if (Function *opF = dyn_cast(op)) { - facts.addEdge(I, *opF, [&](auto& edge) { edge.kinds.push_back(EdgeKind::References); }); - } - } - - // Call edge: record call relationship at the instruction level only. - if (auto *CB = dyn_cast(&I)) { - CallType ct; - if (Function *Callee = CB->getCalledFunction()) { - facts.addEdge(I, *Callee, [&](auto& edge) { edge.kinds.push_back(EdgeKind::Calls); }); - ct = CallType::Direct; - } else { - // Indirect call - ct = CallType::Indirect; - } - - facts.addNodeProp(I, [&](auto& node) { - node.call_type = ct; - node.function_type = typeToString(*CB->getFunctionType()); - }); - } - } - } -} - -void resolve::getModuleFacts(Module &M) { - facts.addNodeProp(M, [&](auto& node) { node.source_file = M.getSourceFileName(); }); - - for (GlobalVariable &G : M.globals()) { - facts.addEdge(M, G, [&](auto& edge) { edge.kinds.push_back(EdgeKind::Contains); }); - - getGlobalFacts(G); - } - - for (Function &F : M) { - facts.addEdge(M, F, [&](auto& edge) { edge.kinds.push_back(EdgeKind::Contains); }); - - getFunctionFacts(F); - } -} - -// Embed the accumulated facts into custom ELF sections. -void resolve::embedFacts(Module &M) { - LLVMContext &C = M.getContext(); - auto embedFactsSection = [&](StringRef sectionName, - const std::string &facts) { - ArrayRef inputData(reinterpret_cast(facts.data()), - facts.size()); - SmallVector compressedFacts; - - if (std::getenv("RESOLVE_IGNORE_COMPRESSION")) { - compressedFacts = SmallVector(inputData); - } else { - compression::Params params(compression::Format::Zstd); - compression::compress(params, inputData, compressedFacts); - } - - //errs() << "Embedding facts for " << sectionName << " with original size " << facts.size() << " and compressed size " << compressedFacts.size() << "\n"; - - Constant *dataArr = ConstantDataArray::get(C, compressedFacts); - GlobalVariable *gv = - new GlobalVariable(M, dataArr->getType(), - /*isConstant=*/true, GlobalValue::InternalLinkage, - dataArr, "resolve" + std::string(sectionName)); - gv->setAlignment(Align()); - gv->setSection(sectionName); - appendToCompilerUsed(M, {gv}); - }; - - // add a newline afterwards to help-distinguish between combined modules - embedFactsSection(".facts", facts.serialize() + "\n"); -} diff --git a/resolve-facts/src/reach/config.hpp b/resolve-facts/src/reach/config.hpp deleted file mode 100644 index 598d00690..000000000 --- a/resolve-facts/src/reach/config.hpp +++ /dev/null @@ -1,92 +0,0 @@ -/* - * Copyright (c) 2025 Riverside Research. - * LGPL-3; See LICENSE.txt in the repo root for details. - */ - -#pragma once - -#include -#include -#include -#include - -#include "json/json.hpp" - -#include "reach/facts.hpp" - -using NNodeId = resolve_facts::NamespacedNodeId; - -namespace conf { -struct query { - NNodeId src; - NNodeId dst; -}; - -struct candidate_node { - std::optional file; - std::string function_name; -}; - -struct config { - std::filesystem::path facts_path; - std::vector queries; - std::vector candidate_path; - bool dynlink = false; - std::optional out_path = {}; - std::optional dlsym_log_path = {}; - std::string graph_type = ""; - std::optional num_paths = {}; - bool validate_facts = false; - bool verbose = false; -}; - -// Generate JSON deserializers for config -NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(query, src, dst); -NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT(candidate_node, file, - function_name); -NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT(config, facts_path, queries, - candidate_path, dynlink, - out_path, dlsym_log_path, - graph_type, num_paths, - validate_facts, verbose); - -// Load config from JSON file -inline std::optional -load_config_from_file(const std::filesystem::path &path) { - std::ifstream f(path); - if (!f.is_open()) { - return {}; - } - nlohmann::json j; - f >> j; - return j.template get(); -} -} // namespace conf - -namespace output { -struct path { - std::vector nodes; - std::vector edges; -}; - -struct query_result { - double query_time; - NNodeId src; - NNodeId dst; - std::vector paths; -}; - -struct results { - double facts_load_time; - double graph_build_time; - std::vector query_results; -}; - -// Generate JSON serializers for results -NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_ONLY_SERIALIZE(path, nodes, edges) -NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_ONLY_SERIALIZE(query_result, query_time, src, - dst, paths); -NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_ONLY_SERIALIZE(results, facts_load_time, - graph_build_time, - query_results); -} // namespace output diff --git a/resolve-facts/src/reach/main.cpp b/resolve-facts/src/reach/main.cpp deleted file mode 100644 index d6e1c5cdc..000000000 --- a/resolve-facts/src/reach/main.cpp +++ /dev/null @@ -1,377 +0,0 @@ -/* - * Copyright (c) 2025 Riverside Research. - * LGPL-3; See LICENSE.txt in the repo root for details. - */ - -// reach - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "argparse/argparse.hpp" - -#include "config.hpp" -#include "reach/facts.hpp" -#include "reach/graph.hpp" -#include "reach/search.hpp" -#include "reach/util.hpp" - -using namespace std; -using namespace chrono; -namespace fs = filesystem; -using json = nlohmann::json; - -// Load config from input file if it was given, then allow any -// explicitly given command line arguments to override the input file. -conf::config load_config(const argparse::ArgumentParser &program) { - try { - const optional in_path = program.present("input"); - conf::config conf; - if (in_path.has_value()) { - const auto conf_opt = conf::load_config_from_file(in_path.value()); - if (conf_opt.has_value()) { - conf = conf_opt.value(); - } - } - if (program.present("facts")) { - conf.facts_path = program.get("facts"); - } - // TODO: argument passing with new id format - if (program.present("src") && program.present("dst")) { - auto src_str = program.get("src"); - auto dst_str = program.get("dst"); - auto srcs = util::split(src_str, ','); - auto dsts = util::split(dst_str, ','); - - conf.queries.push_back({{std::stoi(srcs[0]), std::stoi(srcs[1])}, - {std::stoi(dsts[0]), std::stoi(dsts[1])}}); - } - conf.dynlink = program.get("dynlink") || conf.dynlink; - if (program.present("output")) { - conf.out_path = program.present("output"); - } - if (program.present("dlsym-log")) { - conf.dlsym_log_path = program.present("dlsym-log"); - } - if (program.present("graph")) { - conf.graph_type = program.get("graph"); - } else if (conf.graph_type == "") { - conf.graph_type = "cfg"; - } - if (program.present("num-paths")) { - conf.num_paths = program.get("num-paths"); - } else if (!conf.num_paths.has_value()) { - conf.num_paths = 1; - } - conf.validate_facts = - program.get("validate-facts") || conf.validate_facts; - - if (program.present("path")) { - auto path = program.get("path"); - auto path_split = util::split(path, ','); - - for (const auto &p : path_split) { - // split on first ';' only to allow c++ namespaced names - // e.g. graph.cpp;graph::build_from_program_facts - auto idx = p.find(';'); - // No file specified - if (idx == std::string::npos) { - conf.candidate_path.emplace_back(std::nullopt, p); - } else { - auto path = p.substr(0, idx); - auto node = p.substr(idx + 1); - conf.candidate_path.emplace_back(path, node); - } - } - } - - conf.verbose = program.get("verbose") || conf.verbose; - return conf; - } catch (exception &e) { - throw runtime_error("argparse error: " + string(e.what())); - } -} - -optional> -build_loaded_syms(const optional &path) { - if (path.has_value()) { - const auto log_opt = dlsym::load_log_from_file(path.value()); - if (!log_opt.has_value()) { - return {}; - } - const auto log = log_opt.value(); - vector syms; - // Ensure no duplicate entries - for (const auto &sym : log.loaded_symbols) { - if (find(syms.begin(), syms.end(), sym) == syms.end()) { - syms.push_back(sym); - } - } - return {syms}; - } else { - return {}; - } -} - -void validate_config(const conf::config &conf) { - if (!fs::exists(conf.facts_path)) { - cerr << "CONFIG ERROR: facts_path " << conf.facts_path << " doesn't exist." - << endl; - exit(1); - } -} - -void print_config(const conf::config &conf) { - const json j = conf; - cout << setw(4) << j << endl; -} - -int main(int argc, char *argv[]) { - argparse::ArgumentParser program("reach"); - - program.add_argument("-f", "--facts").help("facts file path"); - program.add_argument("-s", "--src").help("source node in graph"); - program.add_argument("-d", "--dst").help("destination node in graph"); - program.add_argument("-i", "--input").help("JSON input path"); - program.add_argument("-o", "--output").help("JSON output path"); - program.add_argument("-dl", "--dynlink") - .help( - "treat functions with external linkage as having their address taken") - .flag(); - program.add_argument("-ds", "--dlsym-log") - .help("path to file containing dlsym log of loaded symbols"); - program.add_argument("-g", "--graph") - .help("graph type (\"simple\", \"cfg\", or \"call\"). Default \"cfg\""); - program.add_argument("-p", "--path") - .help("candidate path of comma-separated function_name or " - "file;function_name"); - program.add_argument("-n", "--num-paths") - .help("number of paths to generate (n shortest)") - .scan<'i', size_t>(); - program.add_argument("--validate-facts") - .help("validate facts database after loading") - .flag(); - program.add_argument("--verbose") - .help("print misc information to stdout") - .flag(); - - try { - program.parse_args(argc, argv); - } catch (const std::exception &err) { - cerr << err.what() << endl; - cerr << program; - exit(1); - } - - conf::config conf = load_config(program); - if (conf.verbose) { - cout << "Loaded config:" << endl; - print_config(conf); - } - validate_config(conf); - const auto loaded_syms = build_loaded_syms(conf.dlsym_log_path); - - // Execute reachability queries. - // First, build graph. - - typedef graph::T (*graph_builder)( - const resolve_facts::ProgramFacts &, bool, - const optional> &); - - const unordered_map graph_builders = { - {"cfg", graph::build_from_program_facts}, - }; - - if (!graph_builders.contains(conf.graph_type)) { - cerr << "unknown graph type: '" << conf.graph_type << endl; - exit(-1); - } - - time_point t0 = system_clock::now(); - ifstream facts(conf.facts_path); - const auto pf = resolve_facts::ProgramFacts::deserialize(facts); - facts.close(); - - duration facts_load_time = system_clock::now() - t0; - - if (conf.verbose) { - - auto nodes = 0; - auto edges = 0; - for (const auto &[_, m] : pf.modules) { - nodes += m.nodes.size(); - edges += m.edges.size(); - } - cout << "Loaded facts in " << facts_load_time.count() - << " seconds. # nodes = " << nodes << " # edges = " << edges << endl; - } - - t0 = system_clock::now(); - const auto g = - graph_builders.at(conf.graph_type)(pf, conf.dynlink, loaded_syms); - duration graph_build_time = system_clock::now() - t0; - - if (conf.verbose) { - auto edges = 0; - for (const auto &[_, e] : g.edges) { - edges += e.size(); - } - - cout << "Loaded graph in " << graph_build_time.count() - << " seconds. # edges = " << edges << endl; - } - if (!graph::wf(g.edges)) { - cerr << "WARNING: graph not well-formed" << endl; - } - - // Then execute queries against the graph and accumulate results. - - output::results res; - res.facts_load_time = facts_load_time.count(); - res.graph_build_time = graph_build_time.count(); - - std::vector candidate_ids; - - auto find_node = [&](const auto &node) -> std::optional { - for (const auto &[mid, m] : pf.modules) { - if (node.file && - !m.nodes.at(mid).source_file.value_or("").contains(*node.file)) { - continue; - } - - for (const auto &[nid, n] : m.nodes) { - if (n.type == NodeType::Function && n.name.has_value()) { - auto name = n.name.value(); - // Try an exact match on the function name - if (name == node.function_name) { - return std::optional{std::make_pair(mid, nid)}; - } - - // If that doesn't work attempt to demangle the name - // Sadly __cxa_demangle either requires a malloced pointer as input, - // or returns a (fresh) malloced pointer as a result. - // Calling free() in 2025 is sad. I tried to be fancy with a - // shared_ptr with a custom deallocator but was getting malloc - // corruption errors. - auto ret = abi::__cxa_demangle(name.c_str(), NULL, NULL, NULL); - - if (ret) { - std::string demangled{ret}; - free(ret); - - if (demangled.contains(node.function_name)) { - return std::optional{std::make_pair(mid, nid)}; - } - } - } - } - } - return std::nullopt; - }; - - for (const auto &p : conf.candidate_path) { - auto id = find_node(p); - if (!id.has_value()) { - cerr << "No matching node found for candidate path node (file: " - << p.file.value_or("") << ", function: " << p.function_name - << ")\n"; - } else { - candidate_ids.push_back(id.value()); - } - } - - if (conf.candidate_path.size() > 1 && candidate_ids.size() < 2) { - cerr << "Candidate path specified but not enough nodes found; path: \n"; - for (const auto &p : conf.candidate_path) { - cerr << "\t file:" << p.file.value_or("") - << ", name: " << p.function_name << "\n"; - } - } else if (candidate_ids.size() >= 2) { - for (auto i = 0; i < candidate_ids.size() - 1; i += 1) { - conf.queries.emplace_back(candidate_ids[i], candidate_ids[i + 1]); - } - } - - for (const auto &q : conf.queries) { - t0 = system_clock::now(); - output::query_result qres; - qres.src = q.src; - qres.dst = q.dst; - - auto print_missing = [&](auto node, auto type) { - cerr << "node " << type << " " << resolve_facts::to_string(node) - << " not found" << endl; - }; - - // The graph may not have any edges from the src as all may be of the form - // (dst -> src) If the explicit edge does not exist at least check that the - // id is found in the total list of nodes - auto has_src = g.edges.contains(q.src) || pf.containsNode(q.src); - auto has_dst = g.edges.contains(q.dst) || pf.containsNode(q.dst); - - if (!has_src) { - print_missing(q.src, "src"); - } - if (!has_dst) { - print_missing(q.dst, "dst"); - } - - // If both src and dst exist, try to find path. - if (has_src && has_dst) { - - const auto paths = - search::k_paths_yen(g.edges, q.dst, q.src, conf.num_paths.value()); - - duration query_time = system_clock::now() - t0; - qres.query_time = query_time.count(); - - vector weights; - for (const auto &p : paths) { - weights.push_back(graph::path_weight(p)); - } - if (!is_sorted(weights.begin(), weights.end())) { - cerr << "WARNING: paths not sorted by weight" << endl; - } - - for (const auto &path : paths) { - vector p_ids; - vector edges; - for (const auto &e : path) { - const auto id = e.node; - p_ids.push_back(id); - edges.push_back(EdgeType_to_string(e.type)); - } - - reverse(p_ids.begin(), p_ids.end()); - reverse(edges.begin(), edges.end()); - edges.pop_back(); - - qres.paths.push_back({p_ids, edges}); - } - } else { - exit(-1); - } - - res.query_results.push_back(qres); - } - - // Dump results object to out_path if it exists, else to stdout. - const json j = res; - if (conf.out_path.has_value()) { - ofstream f(conf.out_path.value()); - f << setw(4) << j << endl; - } else { - cout << setw(4) << j << endl; - } - - // for (const auto& qres : res.query_results) { - // cout << "# paths: " << qres.paths.size() << endl; - // } -} diff --git a/scripts/package-release.sh b/scripts/package-release.sh index 450f5bd93..0e8acf754 100755 --- a/scripts/package-release.sh +++ b/scripts/package-release.sh @@ -109,7 +109,7 @@ for file in "$STAGED_PREFIX"/bin/*; do done cmake -E make_directory "$STAGE_DIR/usr/local/bin" -for cmd in "$STAGED_PREFIX"/bin/resolve* "$STAGED_PREFIX"/bin/reach "$STAGED_PREFIX"/bin/resolve_read_props; do +for cmd in "$STAGED_PREFIX"/bin/resolve* "$STAGED_PREFIX"/bin/resolve_read_props; do [ -e "$cmd" ] || continue cmd_name=$(basename "$cmd") ln -sfn "$INSTALL_PREFIX/bin/$cmd_name" "$STAGE_DIR/usr/local/bin/$cmd_name"