feat: add compat.re2 2022-04-01 + a upb feature on compat.protobuf - #148
Merged
Conversation
The index-side half of gRPC's P1: with these, a future grpc-m adaptation
repo has to vendor only gRPC's own source.
Before writing either, the biggest unknown was settled by compiling gRPC
1.83.0 TUs directly with mcpp's gcc@16.1.0 — no patches, no configure, no
codegen, include paths only:
OK src/core/lib/slice/slice.cc
OK src/core/lib/surface/call.cc (one of core's heaviest)
OK src/core/lib/promise/activity.cc
OK src/core/ext/upb-gen/google/protobuf/any.upb_minitable.c
OK src/cpp/common/channel_arguments.cc
OK src/cpp/client/channel_cc.cc (the C++ API layer)
and the SSL layer 4/4 against compat.openssl 3.5.1's headers rather than
BoringSSL. gRPC's tree also has no .h.in / config.h.cmake at all, confirming
it needs no configure step.
compat.re2 is pinned to 2022-04-01 because that IS gRPC 1.83.0's pin (its
third_party/re2 submodule is commit 0c5616d == that tag) and gRPC's xds
matchers are written against this API — RE2 swapped re2::StringPiece for
absl::string_view in its 2023 releases, so a newer one is not a drop-in.
Sources follow upstream's RE2_SOURCES: all of re2/*.cc (upstream keeps tests
one level down in re2/testing/) plus util/rune.cc and util/strutil.cc named
individually, because the other TUs in util/ define main() — a dependency's
objects all enter the consumer's link — and util/pcre.cc would drag in libpcre.
The `upb` feature needs no new download: upb has lived in protobuf's own
tarball since v22. Two things there are not guesswork:
* upb/**/*.c CANNOT be globbed. It pulls 14 extra files including TWO
ALTERNATIVE BUILDS of the descriptor tables (upb/reflection/stage0/ and
upb/reflection/cmake/); compiling more than one is a duplicate-symbol
failure. Hence a transcription of libupb_srcs (63 files).
* descriptor.upb_minitable.c must be added ON TOP of that list even though
it is not in it — upstream's cmake/libupb.cmake adds exactly that file as
bootstrap_sources, because the whole reflection layer reaches the
descriptor tables through upb/reflection/descriptor_bootstrap.h. Omitting
it compiles fine and fails only at link time.
include_dirs gains the tarball root and the bootstrap dir. Those are
unconditional because `features` can gate sources/defines/deps but NOT include
dirs, and the feature's own TUs need them; both are additive rather than
shadowing (the root supplies only upb/…, since protobuf's C++ headers live
under src/, and the bootstrap dir supplies google/protobuf/descriptor.upb*.h,
a different file name from the C++ descriptor.h beside it).
Verified cold with the pinned mcpp 2026.8.3.3, gcc@16.1.0,
MCPP_INDEX_MIRROR=GLOBAL, MCPP_BUILD_CACHE=local, target/ removed first:
abseil test result ok. 1 passed; 0 failed (17.84s)
protobuf test result ok. 1 passed; 0 failed (40.88s)
protobuf-gzip test result ok. 1 passed; 0 failed (51.99s)
protobuf-upb test result ok. 1 passed; 0 failed (70.20s)
re2 test result ok. 1 passed; 0 failed ( 2.35s)
The first three are regressions: this change touches compat.protobuf's
include_dirs, so the existing members had to be re-checked.
Object counts verified: re2 links 23 .o (22 sources + the test);
protobuf-upb links 296 = 151 abseil + 80 protobuf + 64 upb + 1 test. The upb
feature was verified negatively too — reverting the member to a bare
`protobuf = "35.1"` drops it to exactly 232 and fails the link with undefined
upb_Arena_Init / upb_Decode / upb_DefPool_Free AND
google__protobuf__FileDescriptorProto_msg_init, which is precisely the
bootstrap descriptor table the second bullet above is about.
The upb test drives BOTH runtimes over the same bytes: the C++ runtime builds
and serializes a FileDescriptorProto, then upb parses it, loads it into a
upb_DefPool and answers reflection queries — and asserts that a name never
declared does NOT resolve.
CN mirror published and closed-loop checked: mcpp-res/re2@2022-04-01 returns
http=200 and is byte-identical to GLOBAL.
…max() CI's windows leg failed with re2/regexp.cc:389: error: too few arguments provided to function-like macro invocation re2/regexp.cc:389: error: reference to non-static member function must be called which is not the usual std::min/std::max collision. RE2's util/mutex.h includes <windows.h>, and its min/max function-like macros then eat RE2's OWN accessors: regexp.cc compares `a->min() == b->min()` and `a->max() == b->max()` on Regexp. Upstream's CMake never spells NOMINMAX out because a CMake consumer normally sets it project-wide; in the index each package carries its own flags. linux re-verified (test result ok); the windows path is only checkable in CI.
This was referenced Aug 4, 2026
Sunrisepeak
added a commit
that referenced
this pull request
Aug 4, 2026
The end of the gRPC track that #147 (abseil, protobuf), #148 (re2, the protobuf `upb` feature) and #149 (c-ares) built toward. gRPC itself lands as a Form A descriptor pointing at mcpplibs/grpc-m. WHY THIS ONE CANNOT BE A COMPAT DESCRIPTOR. Every other heavy library here points at an upstream tarball. gRPC publishes none: v1.83.0 has no release assets at all, and its tag archive carries abseil, protobuf, re2, boringssl and zlib as EMPTY submodule placeholders — one directory entry each — so there is nothing for url+sha256 to address. grpc-m's release tarball IS that artifact: upstream's src/ and include/ vendored with zero patches, plus the two third_party pieces gRPC really ships (address_sorting, xxhash). What it does NOT vendor is why it belongs on this index rather than standing alone: abseil, protobuf(+upb), re2, c-ares, OpenSSL and zlib all come from the packages here, so a consumer that also uses protobuf or abseil directly links ONE copy instead of colliding with a second vendored set. No CMake, no Bazel, no configure step — checked, not assumed: gRPC's tree contains no .h.in or config.h.cmake, and its generated upb code is checked in upstream, so mcpp needs only include paths. All 1001 TUs are compiled by the resolved toolchain, so nothing inherits a foreign C++ ABI the way an install()-driven CMake build would (which is what ruled that route out for gRPC in the first place). The source list is upstream's own — the union of add_library(gpr), grpc, grpc++ and address_sorting — and grpc-m's tools/gen_sources.py --check runs in that repo's CI to prove the manifest has not drifted from the vendored tree. One file is excluded: src/core/ext/upb-gen/google/protobuf/descriptor.upb_minitable.c, byte-for-byte identical to the bootstrap copy compat.protobuf's `upb` feature compiles. LINUX + MACOS ONLY, and the reason is a dependency rather than gRPC: compat.openssl has no windows xpm entry, so resolution there fails with E_NOT_FOUND before anything compiles, and gRPC's secure build cannot drop TLS. The member is gated the way tests/examples/openssl already is — cfg-gated dependency, no-op main() on windows. #150 is the fix for the underlying gap. The member drives `import grpc;` only and carries no protoc output: gRPC's codegen needs host tools mcpp cannot hand a consumer, so grpc-m's own examples/helloworld covers the generated-stub path while this member covers the module surface. It asserts the error path too (a Status that is NOT ok), so an always-OK stub could not pass. Verified with the pinned mcpp 2026.8.3.3, gcc@16.1.0, MCPP_BUILD_CACHE=local, target/ and .mcpp/ removed first — the package downloads from the descriptor's real release tarball and sha: Compiling mcpplibs.grpc v1.83.0 grpc::Version() = 1.83.0 grpc module: OK test result ok. 1 passed; 0 failed; finished in 247.85s grpc-m's own CI is green on both platforms for the tagged commit (linux 54m21s, macOS 29m49s), where examples/helloworld stands a real server on a loopback port and makes a real unary RPC. CN mirror published and closed-loop checked: mcpp-res/grpc@1.83.0 returns http=200 and is byte-identical to GLOBAL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The index-side half of gRPC's P1 (follows #147). With these, a future
grpc-madaptation repo has to vendor only gRPC's own source.
First: the big unknown got settled
Before writing either descriptor, gRPC 1.83.0 TUs were compiled directly with mcpp's
gcc@16.1.0 — no patches, no configure, no codegen, include paths only:
…and the SSL layer 4/4 against
compat.openssl3.5.1's headers, not BoringSSL.gRPC's tree also contains no
.h.in/config.h.cmake, confirming it needs noconfigure step. The whole-source route is real.
compat.re2 — why 2022-04-01
That is an upstream RE2 release tag and gRPC 1.83.0's pin (
third_party/re2==commit
0c5616d== that tag). It matters: gRPC's xds matchers are written againstthis API, and RE2 swapped
re2::StringPieceforabsl::string_viewin its 2023releases, so a newer RE2 is not a drop-in. Newer ones can be added as additional
versions later (
compat.catch2precedent).Sources follow upstream's
RE2_SOURCES: all ofre2/*.cc(upstream keeps tests onelevel down in
re2/testing/), plusutil/rune.ccandutil/strutil.ccnamedindividually — the other TUs in
util/definemain(), and a dependency'sobjects all enter the consumer's link.
The
upbfeature — no new download, but two non-obvious trapsupb has lived in protobuf's own tarball since v22, so this is 64 more TUs out of a
package already in the index. Off by default (nothing in the C++ runtime uses it).
upb/**/*.ccannot be globbed. It pulls 14 extra files including twoalternative builds of the descriptor tables (
upb/reflection/stage0/andupb/reflection/cmake/). Compiling more than one is a duplicate-symbol failure.Hence a transcription of upstream's
libupb_srcs(63 files).descriptor.upb_minitable.cmust be added on top of that list even though itis not in it — upstream's
cmake/libupb.cmakeadds exactly that file asbootstrap_sources, because the reflection layer reaches the descriptor tablesthrough
upb/reflection/descriptor_bootstrap.h. Omitting it compiles fine andfails only at link time.
include_dirsgains the tarball root and the bootstrap dir, unconditionally:featurescan gate sources/defines/deps but not include dirs, and the feature'sown TUs need them. Both are additive, not shadowing.
Verification
Cold, CI configuration (pinned mcpp
2026.8.3.3, gcc@16.1.0,GLOBAL,MCPP_BUILD_CACHE=local),target/removed first:The first three are regressions — this touches
compat.protobuf'sinclude_dirs..o(22 sources + test); protobuf-upblinks 296 = 151 abseil + 80 protobuf + 64 upb + 1 test.
upbverified negatively: reverting the member to a bareprotobuf = "35.1"drops it to exactly 232 and fails the link with undefined
upb_Arena_Init/upb_Decode/upb_DefPool_Freeandgoogle__protobuf__FileDescriptorProto_msg_init— precisely the bootstrapdescriptor table trap feat: add mbedtls 3.6.1 descriptor #2 is about.
and serializes a
FileDescriptorProto, then upb parses it, loads it into aupb_DefPooland answers reflection queries — and asserts a name never declareddoes not resolve.
mcpp-res/re2@2022-04-01,http=200,byte-identical to GLOBAL.
Next (grpc-m), with two knowns already scouted
(
grpc_no_ares=true→GRPC_ARES=0), and only 10 files includeares.h, allunder globbable directories. helloworld over
localhostneeds no async DNS; anaresfeature +compat.c-ares@1.34.5can follow (gRPC already ships frozenconfig_{linux,darwin,windows}/ares_config.husable as the config snapshot).src/core/ext/upb-gen/google/protobuf/descriptor.upb_minitable.c, which defines thesame symbols as the bootstrap file this feature brings. grpc-m must exclude one.
Full write-up:
.agents/docs/2026-08-04-add-re2-and-protobuf-upb-plan.md.