AlignSec

Reproducible cross-lingual prompt-injection security evaluation for instruction-tuned LLMs.

This repository contains the technically frozen AlignSec v2/v2.1 study prepared for the Journal of Information Security and Applications (JISA). The earlier v1/v4/v5 pilot is retained only in Git history and is superseded for publication-facing results. Do not mix pilot outputs with v2/v2.1.

Final study design

• 4 pinned open-weight instruction models: Qwen2.5-3B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct, and Zephyr-7B-Beta.
• 5 language conditions: English, French, Arabic, Bengali, and Sindhi.
• 2 threat channels: direct user injection and indirect document injection.
• 3 prompt-level configurations: policy only, structured isolation, and spotlight marking.
• 24 independent semantic attack prototypes and 12 objective benign tasks.
• Synthetic canaries are held in system/trusted context and are absent from user-controlled content.
• NF4 is the primary precision; Qwen/Phi include a prespecified FP16 sensitivity subset.
• Confirmed leakage includes literal/normalized disclosure plus Base64, URL-Base64, and hex transforms; near-partial leakage is separate.
• Primary uncertainty is clustered by semantic prototype; paired McNemar tests use BH adjustment; exploratory GEE is clustered by prototype.

Output-length audit

The original security endpoint remains confirmed leakage within the prespecified 80-token NF4 generation window. A post-run audit found substantial output-length censoring, so v2.1 reran identical prompts under longer bounded windows: benign 256→512 tokens when needed and attack A01–A04 sensitivity 512→1024 tokens when needed.

All 2,160 v2.1 rendered prompts match the frozen v2 prompt hashes. The single prespecified rescue pass produced 289 generations. After rescue, 215 rows remained final-window capped (138 benign; 77 attack). These rows are reported explicitly: attack negatives mean no confirmed leakage within the tested window, and benign unparseable capped responses count as incorrect/unparseable within the tested window.

Translation validation

All 156 non-English items received exhaustive AI-assisted semantic/syntactic audit and correction. No native-speaker or human linguistic validation is claimed. See docs/TRANSLATION_VALIDATION.md.

Frozen identities

• v2 config: 7e70d005ff2dbda2e8b9adef9336b3e563f904e3e748d81c19dece47054d06bd
• v2.1 follow-up: f930e5900ab2108a99fe803dcd5746778d32988295689da2bff994f3b3b12dc3
• final v2.1 ZIP: 0a82a6d5026ce00a624687cc2b5bd48c0330490d1d11183ceb73e91e4c1a0834
• executed v2.1 notebook: 26fd9f9c0d9b819970cd65e4c1523dd1d6653a9272523e3800f5d4f95b9d21be

Repository layout

analysis/                 final source + executed notebooks
artifacts/v2/             unpacked frozen v2 results
artifacts/v2_1/           unpacked frozen v2.1 results
artifacts/frozen_packages immutable v2/v2.1 ZIP packages
docs/                     methods, review resolution, claim boundaries
translation/              translation-audit supporting record
scripts/                  repository metadata + integrity verification
metadata/                 repository-level manifest/checksums

Verify

python scripts/verify_repository.py

A valid technical-freeze checkout must end with Repository verification PASSED.

Read first

• docs/TECHNICAL_FREEZE.md
• docs/INTERNAL_REVIEW_RESOLUTION.md
• docs/REPRODUCIBILITY.md
• docs/CLAIM_BOUNDARIES.md
• docs/TRANSLATION_VALIDATION.md

License

Original repository code/documentation is governed by the root LICENSE. Third-party model weights are not redistributed and remain subject to upstream terms. See docs/LICENSE_SCOPE.md and THIRD_PARTY_NOTICES.md.