Design document for idempotent storage of relocation metadata.

[smithp35]

#399 describes an example [1] where a combination of the pauthabielf64 and memtagabielf64 cannot be resolved in a system that resolves dynamic relocations more than once.

This design document is an exporation of the design space of possible solutions to the problem. The two most serious contenders are:

• An additional ELF section containing metadata in a one to one correspondence with the relocations. This metadata section augments the existing ELF relocation sections.

• A compressed relative relocation format that encodes the metadata in the relocation section. This would replace the existing relocation sections for relocations requiring additional metadata.

For the ABI the recommendation is to add an additional ELF metadata section to augment the existing relocation sections. This is the most general, and could apply to self-relocating dynamic linkers as well as executables.

The compressed relative relocation format is only likely to be used by self-relocating executables, this means that it probably doesn't need to be fully defined in the full ABI at this point as it can be a private contract between tool and relocation resolver.

[1] In summary both the memtagabielf64 and pauthabielf 64 require metadata in the place of the relocation. When a system can use the place then the reserved for addend field in the signing-schema can be used for the memtagabielf64 tag offset. A system that resolves relocations more than once, and encodes the signing-schema in the relocation addend has no more bits.

[EliaGeretto]

This is very hard with memtag-globals. All accesses to global variables need to be routed through the GOT because all of them need to use a tagged pointer. This essentially translates to no global variables in initialization code.

[smithp35]

I'll add that as an additional complicating factor. I expect that it can be worked around with care, but it is yet another example where to write the initialisation code needs expertise in two ABI extensions.

[EliaGeretto]

The difficulty with this is also the enforcement of this restriction at compile time, expecially in a project that uses LTO, since relocations are visible only after linking.

[smithp35]

It is definitely fragile. I expect that to make the "take care" work will require the footprint of the code covered by the restrictions to be a small enough and stable enough for a small team of experts to write and maintain.

[EliaGeretto]

Sorting them will make the grouping more efficient, but may compromise the effectiveness of the bitmaps used in the RELR encoding. I guess we would need statistics on a project that uses RELR already to figure this out.

[smithp35]

It is possible, and yes ideally we'll need some data to check the heuristic.

My expectation is that with the current encoding, changing state will almost always be more expensive than adding an additional non-bitmap address field to an existing state. There's also the problem that changing state mid-bitmap may require a new non-bitmap address field anyway.

I think it would need a different approach to make changing state cheaper, for example declaring all the metadata once, and then using a smaller index into it during the description of relocations.

Relocation Code
metadata 1
metadata 2
metadata 3
metadata 4
Relocations for code
Use metadata 1
Address
Bitmap
Use metadata2
Bitmap (assuming no overlap with previous bitmap

[EliaGeretto]

This may not be the case (same signing-schema) with:

• -fptrauth-function-pointer-type-discrimination and similar.
• manually specified discriminators (string with variable name).
• tag offsets which currently depend on the size of the object. We could tell the compiler to prefer -1 though.

Compared to RELA, the statement is true because we are at least not repeating r_addend, which is in the place, and r_info, since we state the relocation code only once.

[smithp35]

With the default signing-schema, my C++ test application ended with 50 unique signing-schemas out of 1500 relocations. I agree that if almost all relocations have different metadata then there will be better ways to encode. There's also a balance between how complex the encoding and decoding can be. For example, would using ULEB encoding like CREL, at the expense of non-4-byte aligned fields be acceptable?

[EliaGeretto]

That's good news. Is your C++ test a "real" project or is it artificial? It's fair to assume that with C++ inheritance you may end up with a lot of methods with the same signature and thus the same discriminator (I am not familiar with that part of the instrumentation). Do you think that transfers to C as well? I guess so if it is "object-oriented".

[smithp35]

Artificial; it is just a trivial program to test that I can throw and catch an exception. This drags in quite a lot libunwind (C) and libcxxabi (C++) in particular the itanium demangler for error messages which has quite a few virtual functions. In practice the vtables dominate the example, with a large .data.rel.ro, small .got and even smaller .data

I think C++ vtables would translate reasonably well to C function pointers with type discrimination enabled. Many function pointers will share the same type signature.

[EliaGeretto]

I think we should also discuss how much space we should reserve for metadata. Technically, if we have a full 64 bits addend, Memtag requires a 64 bits tag offset to match it (60 if we define it as a number of tag granules). This means that, to avoid any restrictions, PAuth + Memtag requires 64 + 32 bits of metadata (trimmable removing reserved bits, but still more than 64). To avoid running out of space again and also not waste metadata space in Memtag-only or PAuth-only binaries, we may consider making the metadata size dependent on the relocation code for the compressed relocation solution. Alternatively, we could have a few bits in the header of whatever metadata section we end up picking which say how many 64 bits words the metadata field will contain. In this way, when we end up wanting to do PAuth + Memtag + something else, increasing that value is enough.

[EliaGeretto]

We may need an additional code for the PAuth + tag offset case, if we want a full 64 bits tag offset to match the 64 bits addend.

[smithp35]

One option I had in mind was to make the number of additional meta-data items dependent on the relocation code, so there could be any amount of additional metadata. That extension could be made at a later date.

Given the maximum size of a small code binary is 2 GiB a 32-bit field for an offset would be enough. Even with the large code model I think there are few non-contrived use cases for a 64-bit sized tag offset.

[smithp35]

I think we should also discuss how much space we should reserve for metadata. Technically, if we have a full 64 bits addend, Memtag requires a 64 bits tag offset to match it (60 if we define it as a number of tag granules). This means that, to avoid any restrictions, PAuth + Memtag requires 64 + 32 bits of metadata (trimmable removing reserved bits, but still more than 64). To avoid running out of space again and also not waste metadata space in Memtag-only or PAuth-only binaries, we may consider making the metadata size dependent on the relocation code for the compressed relocation solution. Alternatively, we could have a few bits in the header of whatever metadata section we end up picking which say how many 64 bits words the metadata field will contain. In this way, when we end up wanting to do PAuth + Memtag + something else, increasing that value is enough.

There is a possibility of altering the sizes of the metadata entries, for dynamic linkers this could be done with a DT_ tag, as well as setting sh_entsize. A statically linked program would need a linker defined symbol, which may end up being user-defined rather than being set by the linker. The size of the entries could affect the encoding:

• (4 bytes, R_AARCH64_AUTH_RELATIVE) PAuthABI metadata only (shift left by 32) to recover the original signing-schema.
• (4 bytes, R_AARCH64_RELATIVE) 32-bit tag offset
• (8 bytes, R_AARCH64_AUTH_RELATIVE) = PAuthABI metadata with reserved for addend containing a 32-bit tag offset
• (8 bytes, R_AARCH64_RELATIVE) 64-bit tag offset
• (16 bytes, R_AARCH64_AUTH_RELATIVE) = 8-bytes of PAuthABI followed by 8-bytes of tag offset.
• (16 bytes, R_AARCH64_RELATIVE) = 8-bytes of zeros followed by 8-bytes of tag offset.

To keep it simple I only wrote up the 8-bytes case. I think an implementation could be extended to other two cases given some careful wording.

I'm not yet convinced that there's a non-contrived use case for 64-bit offsets at the moment. The small code-model already limits program size to 2GiB (32-bit offsets in exeception index tables). If I've understood the memtagabielf64 correctly it would need an extremely large array, or some kind of pointer-integer cast with some extremely large offsets.

[EliaGeretto]

I think we should also discuss how much space we should reserve for metadata. Technically, if we have a full 64 bits addend, Memtag requires a 64 bits tag offset to match it (60 if we define it as a number of tag granules). This means that, to avoid any restrictions, PAuth + Memtag requires 64 + 32 bits of metadata (trimmable removing reserved bits, but still more than 64). To avoid running out of space again and also not waste metadata space in Memtag-only or PAuth-only binaries, we may consider making the metadata size dependent on the relocation code for the compressed relocation solution. Alternatively, we could have a few bits in the header of whatever metadata section we end up picking which say how many 64 bits words the metadata field will contain. In this way, when we end up wanting to do PAuth + Memtag + something else, increasing that value is enough.

There is a possibility of altering the sizes of the metadata entries, for dynamic linkers this could be done with a DT_ tag, as well as setting sh_entsize. A statically linked program would need a linker defined symbol, which may end up being user-defined rather than being set by the linker. The size of the entries could affect the encoding:

• (4 bytes, R_AARCH64_AUTH_RELATIVE) PAuthABI metadata only (shift left by 32) to recover the original signing-schema.
• (4 bytes, R_AARCH64_RELATIVE) 32-bit tag offset
• (8 bytes, R_AARCH64_AUTH_RELATIVE) = PAuthABI metadata with reserved for addend containing a 32-bit tag offset
• (8 bytes, R_AARCH64_RELATIVE) 64-bit tag offset
• (16 bytes, R_AARCH64_AUTH_RELATIVE) = 8-bytes of PAuthABI followed by 8-bytes of tag offset.
• (16 bytes, R_AARCH64_RELATIVE) = 8-bytes of zeros followed by 8-bytes of tag offset.

To keep it simple I only wrote up the 8-bytes case. I think an implementation could be extended to other two cases given some careful wording.

I'm not yet convinced that there's a non-contrived use case for 64-bit offsets at the moment. The small code-model already limits program size to 2GiB (32-bit offsets in exeception index tables). If I've understood the memtagabielf64 correctly it would need an extremely large array, or some kind of pointer-integer cast with some extremely large offsets.

As long as the new design leaves the possibility for future expansion open without forcing extremely complex solutions, we should be fine. What you are proposing sounds reasonable to me.