Skip to content

feat: Agentforce Scorers Authoring Command#456

Open
nabilnaffar-sf wants to merge 7 commits into
salesforcecli:mainfrom
nabilnaffar-sf:nnaffar/scorers
Open

feat: Agentforce Scorers Authoring Command#456
nabilnaffar-sf wants to merge 7 commits into
salesforcecli:mainfrom
nabilnaffar-sf:nnaffar/scorers

Conversation

@nabilnaffar-sf

Copy link
Copy Markdown

What does this PR do?

Adds a new sf agent scorer create command that generates AiAgentScorerDefinition metadata XML files for Agentforce scorers. The command supports:

  • Interactive interview mode — prompts for data type, output values, engine type, agent association, and all other scorer fields step-by-step
  • YAML spec-driven mode (--spec) — reads a YAML file for repeatable/automated scorer creation
  • Preview mode (--preview) — outputs generated XML without writing to disk

What issues does this PR fix or reference?

no issues, this is a new feature

@salesforce-cla

salesforce-cla Bot commented Jul 1, 2026

Copy link
Copy Markdown

Thanks for the contribution! It looks like @nabilnaffar-sf is an internal user so signing the CLA is not required. However, we need to confirm this.

@nabilnaffar-sf nabilnaffar-sf marked this pull request as ready for review July 1, 2026 17:00
@nabilnaffar-sf nabilnaffar-sf requested a review from a team as a code owner July 1, 2026 17:00
@nabilnaffar-sf nabilnaffar-sf marked this pull request as draft July 2, 2026 07:22
@nabilnaffar-sf nabilnaffar-sf marked this pull request as ready for review July 2, 2026 07:22
@jeniok

jeniok commented Jul 12, 2026

Copy link
Copy Markdown

Command shape looks great and the interview + spec YAML flow is a nice touch. However, I ran the branch locally and hit several issues that I think should block merge. Highlights: the unit-test suite doesn't
actually run (48/53 fail), --spec-schema and --preview require an org they don't need, and the documented "non-interactive all-flags" example crashes.

Blocking

  1. Unit tests are effectively broken — 48 of 53 fail. All 48 failures share one cause:

Error: Parsing --spec
No file found at test.yaml

The command declares spec: Flags.file({ exists: true }) (src/commands/agent/scorer/create.ts:225), so oclif runs a real fs.stat() before esmock can intercept node:fs. Tests pass fake paths ('test.yaml',
'open-scorer.yaml', etc.) which all fail the existence check. Only 5 tests pass: 3 that write real files to a tmp dir, plus 2 that don't use --spec.

GitHub CI on this PR shows only SAST + CLA checks — the unit-test job never ran, so this went undetected.

Fix options: drop exists: true and validate in code, or refactor tests to write to a real tmp dir. Note that writtenFiles assertions can't observe writes inside @salesforce/agents/lib/agentScorer.js either,
because esmock only rewrites the command module's imports — the mocking pattern needs a rethink regardless.

  1. --spec-schema requires --target-org.

$ sf agent scorer create --spec-schema
NoDefaultEnvError: No default environment found.

target-org: Flags.requiredOrg() is unconditional. --spec-schema shouldn't need auth (it's a static file dump), and --preview shouldn't either unless --agent-api-name is missing. The first example in
messages/agent.scorer.create.md breaks out of the box.

  1. Documented "non-interactive" example crashes. From the message file:

sf agent scorer create --api-name Expert_Analysis --data-type Text
--engine-type Manual --label Expert_Analysis --agent-api-name My_Agent --status Available

Running this triggers ExitPromptError — inquirer still prompts for description (create.ts:324), semanticType (:330), agent-association inputScope (:458), and Text outputEnumValues (:402). The jsonEnabled() gate
at create.ts:301-309 only covers four flags marked required; it doesn't cover the fields the interview actually asks for.

Should fix

  1. --data-type OpenEnded from the flag path can't complete non-interactively. promptForDataTypeDetails('OpenEnded') still prompts for lightningType, but there's no CLI flag for it. The non-interactive OpenEnded
    path is unusable.

  2. Flag/schema mismatch: --data-type OpenEnded → dataType: LightningType. create.ts:346 — the CLI's OpenEnded enum resolves to LightningType in the XML. This surprise translation isn't in the flag help; users
    seeing LightningType in the preview will be confused.

  3. JSON schema doesn't enforce conditional requirements. schemas/agent-scorer-create__spec.json requires apiName, dataType, label, engineType, agentAssociation, but not specification for Number, lightningType
    for LightningType, or outputEnumValues for Text. Use draft-07 if/then/else or oneOf per dataType — otherwise invalid specs pass schema validation and blow up later.

  4. Duplicated apiName regex. create.ts:60 and agents/src/agentScorer.ts:124 both encode the API-name rule. If one drifts, error messages diverge. Have the CLI defer to the library's validator.

  5. --json mode silently overwrites existing files. create.ts:276 short-circuits the confirm prompt when jsonEnabled(). No --force flag for scripting. At minimum, log to stderr that an overwrite occurred.

  6. --spec-schema returns a fake AgentScorerCreateResult. create.ts:251 returns { path: '', apiName: '', contents: '' } — violates the semantic contract. Machine consumers get an empty triple. Widen the return
    type or split --spec-schema into its own command.

Nice-to-have

  1. Unchecked cast on YAML parse — create.ts:257: YAML.parse(...) as ScorerSpec. Since you ship a JSON schema for this exact file, run it through ajv first. Better error messages than "specification is
    required" from deep inside validateScorerSpec.

  2. promptForNumberSpecification throws mid-interview on min >= max — create.ts:164 — loses prior answers. Prefer soft-validation via inquirer's validate callback.

  3. "Exactly one fallback" only checked post-hoc. promptForOutputEnumValues lets users mark multiple fallbacks; the constraint is enforced only later in validateScorerSpec. Enforce inline so the user fixes it
    during entry.

  4. export type ScorerSpecFile = ScorerSpec (create.ts:45) is marked deprecated and only used by the (broken) test file. Once tests are fixed, remove.

  5. --api-version flag is defined but only used to build the connection for Agent.listRemote. No metadata deploy, so it looks vestigial. If intentional (SOQL API version override), add a comment; otherwise drop
    it.

  6. eslint-disable-next-line complexity on run(). The method genuinely is too complex. Extracting resolveSpec() and writeOrPreview() helpers would also make the tests salvageable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants