Add archetype-validation-prober: an ingestion validation gate for Pub/Sub

archetype-validation-prober/PYTHON_COMPANION.md

@@ -0,0 +1,41 @@
+# Python Companion for the Archetype Gate
+
+This repository keeps the Java prober as the upstream reference, but the same
+idea can be implemented in a more compact and operationally friendly way with a
+Python companion.
+
+## What the companion adds
+
+- `Archetype` gate implemented as a total function in Python
+- `ValidationResult` and `ValidationError` as immutable value objects
+- `ClassifyingReceiver` without a builder or singleton boilerplate
+- `vertex_agent/` for ingestion, retrieval, and MCP exposure
+- real GCP resources:
+  - Pub/Sub topic and subscription
+  - BigQuery dataset and table
+  - Cloud Storage bucket for docs
+- smoke validation with OAuth/ADC only
+
+## Why this is useful
+
+The Java prober proves the gate behavior. The Python companion shows how the
+same architecture can be expanded into a full GCP-native workflow:
+
+1. validate deterministically before publish
+2. keep the retry path for transient failures only
+3. store knowledge in BigQuery
+4. expose the result through MCP for coding agents
+5. validate everything against real infrastructure
+
+## What was proven in practice
+
+- `uv run pytest` passed
+- `uv run python -m scripts.gcp_smoke` returned `smoke=ok`
+- Pub/Sub publish/pull round-trip succeeded
+- BigQuery insert/query succeeded
+
+## Security note
+
+No service account keys are stored in the repository. Real smoke tests use
+developer-owned OAuth/ADC credentials only.
+

archetype-validation-prober/README.md

@@ -0,0 +1,116 @@
+# Archetype Validation Prober
+
+A reference implementation of an **archetype validation gate** in front of Cloud Pub/Sub.
+
+It demonstrates a single principle: **only what minimally fits the process should ever be
+enqueued.** Everything that is deterministically invalid is rejected *synchronously at the gate* and
+never published, so it can never bounce in the queue, never inflate the dead-letter store, and never
+burn engine resources being retried.
+
+The prober is provided as-is for demonstration purposes only, with no SLA. It is not meant to be run
+as part of a production or critical workload.
+
+## Why
+
+A common failure mode of asynchronous messaging is treating a transport `200`/ack as if it meant the
+message was *accepted*. It does not. A payload can be delivered and still be rejected on content, and
+a message that never returns a positive functional ack **bounces forever** (or is silently dropped at
+retention). At two messages that is noise; at two hundred thousand it is an outage.
+
+The fix is to separate the concerns and validate the deterministic ones **before** enqueuing:
+
+| Outcome | Nature | Where it is caught | Action |
+| --- | --- | --- | --- |
+| Conformant (functional ACK) | Success | — | Publish / `ack()` |
+| Not JSON / wrong type / missing field / bad enum / bad pattern | **Deterministic** | **At the gate (this prober)** | **Reject synchronously, never enqueue** |
+| Byte-different but semantically equal (charset / Unicode NFD) | False reject | At the gate (canonicalized) | Normalize to NFC, then accept |
+| Downstream down (timeout, 5xx, connection) | Transient | Consumer | `nack()` → retry with backoff → dead-letter |
+| State-dependent (unknown id, duplicate, business rule) | Unpredictable | Consumer | Route to quarantine / dead-letter |
+
+The gate handles the deterministic and false-reject rows. The consumer
+([`ClassifyingReceiver`](src/main/java/com/google/cloud/pubsub/archetype/ClassifyingReceiver.java))
+handles the rest and, crucially, **only lets transient failures ride the redelivery machinery**;
+functional rejects are `ack()`-ed and parked in quarantine so they never bounce.
+
+## Components
+
+- [`Archetype`](src/main/java/com/google/cloud/pubsub/archetype/Archetype.java) — the gate. Loads a
+  versioned JSON Schema (the *archetype*) and, cheapest check first, canonicalizes (charset + Unicode
+  NFC), then validates syntactically and structurally. Returns accept + canonical form, or reject +
+  machine-readable reason codes. Never throws on bad input.
+- [`ValidationResult`](src/main/java/com/google/cloud/pubsub/archetype/ValidationResult.java) — the
+  terminal outcome (`ACCEPTED` with canonical payload, or `REJECTED` with reasons).
+- [`ClassifyingReceiver`](src/main/java/com/google/cloud/pubsub/archetype/ClassifyingReceiver.java) —
+  a subscriber that classifies each processing failure (accepted / transient / functional reject)
+  and acts accordingly, keeping poison messages out of the retry loop.
+- [`ArchetypeValidationGateway`](src/main/java/com/google/cloud/pubsub/archetype/ArchetypeValidationGateway.java)
+  — the runnable entry point. With no arguments it runs a self-contained, offline demo of the gate
+  over representative payloads.
+- [`archetype.schema.json`](src/main/resources/archetype.schema.json) — the canonical contract used
+  by the demo. Replace it with your own to model your payload.
+
+## Build
+
+These instructions assume [Maven](https://maven.apache.org/) 3 and Java 8.
+
+```
+cd archetype-validation-prober
+mvn package
+```
+
+The resulting jar is at `target/pubsub-archetype-validation-prober.jar`.
+
+## Run (offline demo)
+
+```
+java -jar target/pubsub-archetype-validation-prober.jar
+```
+
+Expected output (one line per sample): the conformant and the NFD-normalized payloads are
+`ACCEPTED (enqueued)`; every deterministic failure is `REJECT` with its reason codes and is never
+enqueued.
+
+### Options
+
+| Property | Type | Default | Description |
+| --- | --- | --- | --- |
+| `--archetype` | String | bundled `archetype.schema.json` | Path to the JSON Schema archetype to validate against. |
+| `--charset` | String | `UTF-8` | Declared charset of incoming payloads, used for canonicalization. |
+| `--help` | flag | — | Print usage and exit. |
+
+## Test
+
+```
+mvn test
+```
+
+## Smoke test (real Pub/Sub or emulator)
+
+The smoke tests are intentionally off by default. They validate the end-to-end
+flow against either a local emulator or a real GCP project.
+
+### Emulator
+
+```bash
+export PUBSUB_EMULATOR_HOST=localhost:8085
+gcloud beta emulators pubsub start
+mvn verify -Psmoke
+```
+
+### Real GCP
+
+```bash
+export GOOGLE_CLOUD_PROJECT=portfolioadvanced-llm
+gcloud auth login
+gcloud auth application-default login
+mvn verify -Psmoke
+```
+
+The smoke suite covers:
+
+- valid payload round-trip
+- invalid payload never reaches publish
+- encoding rejection before JSON parsing
+
+For the higher-level GCP-native companion that demonstrates the same idea in
+Python, see [PYTHON_COMPANION.md](PYTHON_COMPANION.md).

archetype-validation-prober/pom.xml

@@ -0,0 +1,137 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <groupId>com.google.cloud.pubsub.archetype</groupId>
+  <artifactId>archetype-validation-prober</artifactId>
+  <packaging>jar</packaging>
+  <version>1.0-SNAPSHOT</version>
+  <name>archetype-validation-prober</name>
+  <url>http://maven.apache.org</url>
+
+  <properties>
+    <maven.compiler.source>1.8</maven.compiler.source>
+    <maven.compiler.target>1.8</maven.compiler.target>
+    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+  </properties>
+
+  <dependencies>
+    <dependency>
+      <groupId>junit</groupId>
+      <artifactId>junit</artifactId>
+      <version>4.13.1</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>com.beust</groupId>
+      <artifactId>jcommander</artifactId>
+      <version>1.72</version>
+    </dependency>
+    <dependency>
+      <groupId>com.google.cloud</groupId>
+      <artifactId>google-cloud-pubsub</artifactId>
+      <version>1.116.3</version>
+    </dependency>
+    <!-- JSON archetype (schema) validation. 1.0.x line targets Java 8. -->
+    <dependency>
+      <groupId>com.networknt</groupId>
+      <artifactId>json-schema-validator</artifactId>
+      <version>1.0.87</version>
+    </dependency>
+    <dependency>
+      <groupId>com.fasterxml.jackson.core</groupId>
+      <artifactId>jackson-databind</artifactId>
+      <version>2.13.5</version>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-shade-plugin</artifactId>
+        <version>2.3</version>
+        <configuration>
+          <createDependencyReducedPom>true</createDependencyReducedPom>
+          <filters>
+            <filter>
+              <artifact>*:*</artifact>
+              <excludes>
+                <exclude>META-INF/*.SF</exclude>
+                <exclude>META-INF/*.DSA</exclude>
+                <exclude>META-INF/*.RSA</exclude>
+              </excludes>
+            </filter>
+          </filters>
+        </configuration>
+        <executions>
+          <execution>
+            <phase>package</phase>
+            <goals>
+              <goal>shade</goal>
+            </goals>
+            <configuration>
+              <finalName>pubsub-archetype-validation-prober</finalName>
+              <transformers>
+                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+                  <mainClass>com.google.cloud.pubsub.archetype.ArchetypeValidationGateway</mainClass>
+                </transformer>
+                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+              </transformers>
+            </configuration>
+          </execution>
+        </executions>
+      </plugin>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-compiler-plugin</artifactId>
+        <configuration>
+          <source>1.8</source>
+          <target>1.8</target>
+        </configuration>
+      </plugin>
+    </plugins>
+  </build>
+
+  <!--
+    Smoke profile — runs integration tests against a real or emulated Pub/Sub endpoint.
+    Intentionally OFF by default; activate with: mvn verify -Psmoke
+
+    Two modes:
+      Emulator (zero cost):  export PUBSUB_EMULATOR_HOST=localhost:8085
+                             gcloud beta emulators pubsub start
+      Real GCP:              export GOOGLE_CLOUD_PROJECT=<project>
+                             (Application Default Credentials must be valid)
+  -->
+  <profiles>
+    <profile>
+      <id>smoke</id>
+      <build>
+        <plugins>
+          <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-failsafe-plugin</artifactId>
+            <version>3.2.5</version>
+            <configuration>
+              <includes>
+                <include>**/*SmokeIT.java</include>
+              </includes>
+              <systemPropertyVariables>
+                <GOOGLE_CLOUD_PROJECT>${env.GOOGLE_CLOUD_PROJECT}</GOOGLE_CLOUD_PROJECT>
+                <PUBSUB_EMULATOR_HOST>${env.PUBSUB_EMULATOR_HOST}</PUBSUB_EMULATOR_HOST>
+              </systemPropertyVariables>
+            </configuration>
+            <executions>
+              <execution>
+                <goals>
+                  <goal>integration-test</goal>
+                  <goal>verify</goal>
+                </goals>
+              </execution>
+            </executions>
+          </plugin>
+        </plugins>
+      </build>
+    </profile>
+  </profiles>
+
+</project>