Update settings handling; add provenance and allow archiving unused settings

[dale-wahl]

What this does

4CAT's settings table has no record of where a setting came from, so settings left behind by renames and removed features accumulate forever and show up in the settings panel as junk. They could not be cleaned up safely, because an undeclared setting is indistinguishable from one whose extension is merely uninstalled, disabled, or failing to import.

This records which module declares each setting, uses that to tell those cases apart, and gives us a way to remove the ones that really are gone. And I added an archive to do that reversibly.

Why not the extension. prefix

The issue proposed prefixing extension settings so they could be identified. I went a different way, for a couple reasons.

Renaming a setting like selenium.browser means changing every config.get() call, every required_settings entry and every config block at the same time. Any extension not updated would back to defaults and we would lose the value, not just the row. Not ideal.

Additionally, a prefix says a setting is extension-owned. We still wouldn't know whether the owner is absent because it was uninstalled/deleted or removed, switched off (extension), crashed on import, etc. Crashed on import also applies to core processors too, so just knowing "is this an extension setting?" would miss that.

For collision-proofing, new settings should go under the extension's own id such as my_extension.api_key. Not extensions.<id> since that is one of the namespaces used for core settings (see below). Now those core prefixes are refused. The extensions README.md says so as well (for good readers).

How all this works

ModuleCollector now records which worker declared each setting instead of flattening them all together. After the modules load, the back-end writes information about each setting into settings_declarations and stores them in the database (common/config_manager.py, backend/bootstrap.py). Every stored setting can then be sorted into one of five states, only one of which is ever offered for removal:

• dormant — extension installed but switched off; kept, so switching it on restores its configuration
• absent_extension — extension uninstalled; kept indefinitely for the same reason
• vanished — last declared by core, and gone for longer than the grace period; the only removable class
• recently_absent — gone, but not for long enough to be sure
• unknown — nothing ever recorded declaring it; no evidence either way, so never removed automatically (all our accumulated old settings are here; we can remove them via migrate scripts if we know they are old)

Age is measured against the last start-up on which every module imported, not against the clock (4cat.declarations_last_clean_scan). While an import is broken that marker stops advancing, so nothing can age into looking removed just because 4CAT currently cannot see it (mostly in case we ship something broken and do not notice it immediately).
Age is measured from the first complete start-up that found a setting missing, recorded per setting in settings_declarations.absent_since. There is a last seen, but the gap between two start-ups is however long the server happened to be up, so measuring from that would make anything an upgrade removes look long gone the moment it went. absent_since is only ever written on a start-up where every module imported (4cat.declarations_last_scan_complete), so a broken import still cannot age anything out.

A daily worker reports what it finds to admins (backend/workers/audit_settings.py), behind 4cat.report_orphan_settings, which is off by default. It remembers which set it last reported in 4cat.declarations_reported, so dismissing the notification does not bring it back on the next run. I discovered canonical_id means "came from the phone-home server" so could not be used and I had to match the notification text itself (this might be worth an improvement on the notifications table). This is a tool for us (and hypothetical other developers), but probably not interesting to general admins.

Undeclared settings are no longer rendered in the settings panel at all. A line at the foot of the page links to a new Unused settings page (webtool/templates/controlpanel/settings-unused.html) listing them with why each is kept, and an Archive button for the ones that qualify. This is for everyone.

Nothing is destroyed

Archiving moves a setting's values (the global one and every tag) into settings_archive, and they can be restored from the same page. The guard lives in ConfigManager.archive_setting(), not in the view, so a request naming a setting directly cannot get past it (I think/hope). We could allow archiving other settings too if we want (e.g. extensions and even offer to restore them if enabled/reinstalled but... that's mostly just db cleanup).

Two related fixes fell out of this:

A module could previously override a core setting's definition, because module config was merged over core with a plain dict.update(). That is now refused rather than merged, and namespaces belonging to core (privileges., flask., 4cat., path., datasources., extensions., logging.) are refused outright so an extension cannot claim a name a later 4CAT version might use. This matters because a definition controls a setting's global flag which meant a module using the same name plus global on a core privilege would have made every per-tag restriction silently stop applying. Probably not a risk so much as a mistake (installing an extension by nature has lots of risks anyway).

module_config.bin is now written atomically. It was truncated and rewritten in place, which a reader in the other container could catch mid-write; the retry that was meant to survive that but probably would have failed, because it re-read the same file handle after a failed unpickle. This should all but avoid that happening (I think it could still try to read right as a file was being replaced so I left the retry). I accidentally ran into this because I originally dumped the provenance stuff in with the rest of the module_config.bin stuff (turned out that was bad in general).

Needs an upgrade for the database

VERSION goes to 1.57. migrate-1.56-1.57.py creates the two tables and archives 18 core settings for features 4CAT no longer has (due to renames, the Reddit datasource, the removed scheduler, the old per-processor proxy settings, and so on). The list is written out in the migration with a note on what became of each one, since none of it can be figured out automatically. Now removed settings won't be a problem!

Extension settings are not touched, including ones an extension has renamed itself. I'm leaving that to the extension author's call, not ours, and an older version of it may still be in use somewhere. I am not sure if or how we could encourage extensions to remove old settings, but we can track them now and could combine that with versioning of extensions.

On the my DevCAT I tested against: 182 settings attributed, 27 undeclared, 18 archived, 9 deliberately left (extensions and future feature settings I've been toying with).

Two smaller settings bugs found

api.youtube.key was declared in both config_definition.py and youtube_metadata.py. Since module config was merged after core, the processor's copy had been silently winning. Its help text is now the one in the core declaration so
nothing changes in the interface, and the duplicate is gone.

api.openai.api_key is read in processors/machine_learning/audio_to_text.py but declared nowhere, so it cannot be set
from the interface and the OpenAI Whisper fallback never fires. Left alone here — it either wants re-declaring or retiring in favor of the LLM settings. Probably LLM, but need to sort out what that looks like. Out of this PR anyway.