Skip to content

fix(parser): retain single-letter language skills #855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
qtjg wants to merge 1 commit into from
Closed

fix(parser): retain single-letter language skills #855

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/lib/heuristics/corpus.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ const TRUTH_ANNOTATED_FIELD_FLOOR = 150;
* `npm run check:baselines` on every run, and bounded here — undescribed debt may
* not GROW. File the issue and flip the entry to `open`; then lower this.
*/
const UNFILED_TRUTH_CEILING = 9;
const UNFILED_TRUTH_CEILING = 8;
Comment thread
qtjg marked this conversation as resolved.

/** Generator category = the fixture root's immediate subdirectory. */
function categoryOf(repoRelPdfPath: string): string {
Expand Down
6 changes: 6 additions & 0 deletions src/lib/heuristics/extract/skills.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,12 @@ describe("tokenizeSkillLine", () => {
expect(tokenizeSkillLine(",,,;;;")).toEqual([]);
});

it("keeps the defensible single-letter languages and rejects stray glyphs (#832)", () => {
const result = tokenizeSkillLine("C, R, D, X");
expect(result).toEqual(expect.arrayContaining(["C", "R", "D"]));
expect(result).not.toContain("X");
});
Comment on lines +82 to +86

@s-annam s-annam Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secondary (finding 2). The behaviour is right — I ran #832's whole AC token set against this head and every row passes. The gap is that only the X rejection is asserted.

J is the one worth pinning: #832 explicitly says "Do NOT add J, K, Q, or F", so a test holding J as rejected is what stops SINGLE_LETTER_SKILLS being widened later without an argument. And the mixed line is a positive control — without it, a regression that made this return [] for any line containing a stray glyph would still pass.

Verified outputs, so the suggestion below is asserting real behaviour rather than a guess:

"Languages: C, R, D"      -> ["C","R","D"]
"Skills: x, J, •, ("      -> []
"Skills: Python, x, J, C" -> ["Python","C"]
Suggested change
it("keeps the defensible single-letter languages and rejects stray glyphs (#832)", () => {
const result = tokenizeSkillLine("C, R, D, X");
expect(result).toEqual(expect.arrayContaining(["C", "R", "D"]));
expect(result).not.toContain("X");
});
it("keeps the defensible single-letter languages (#832)", () => {
expect(tokenizeSkillLine("Languages: C, R, D")).toEqual(
expect.arrayContaining(["C", "R", "D"]),
);
});
it("still rejects other single-character noise tokens (#832)", () => {
// `J` is a real language name but also a plausible stray glyph; #832
// deliberately keeps it out of the allowlist. Pin that so a later widening
// of SINGLE_LETTER_SKILLS has to argue for itself.
expect(tokenizeSkillLine("Skills: x, J, •, (")).toEqual([]);
});
it("keeps an allowlisted letter alongside real skills on one line (#832)", () => {
// Positive control: distinguishes rejecting noise from rejecting everything.
expect(tokenizeSkillLine("Skills: Python, x, J, C")).toEqual(["Python", "C"]);
});


it("drops the whole cell when a URL is present in a comma-separated list", () => {
// tokenizeCell's looksLikeContactLink check fires on the ENTIRE cleaned
// cell before the split. "github.com/janesmith" matches the path-slash
Expand Down
7 changes: 7 additions & 0 deletions src/lib/heuristics/extract/skills.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,12 @@ const PROFILE_HOST_RE =
* "Socket.io", "ASP.NET") that has no slash. */
const URLISH_RE = /(https?:\/\/|www\.|\b[a-z0-9-]+\.[a-z]{2,}\/\S)/i;

/** One-character tokens that are real, commonly-listed languages. The length
* floor in `isSkillToken` is a noise guard against stray glyphs left by column
* splitting; these are the only single characters that are not noise, so they
* are allowlisted rather than lowering the floor. */
const SINGLE_LETTER_SKILLS = new Set(["c", "r", "d"]);

/** True when a candidate skill token is really a professional-profile link
* (GitHub / LinkedIn / portfolio, etc.) or its bare heading word. Such links
* belong only in the contact/profile section, never in Skills. */
Expand All @@ -139,6 +145,7 @@ function looksLikeContactLink(tok: string): boolean {
}

function isSkillToken(tok: string): boolean {
if (tok.length === 1 && SINGLE_LETTER_SKILLS.has(tok.toLowerCase())) return true;

@s-annam s-annam Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirming this line rather than flagging it: the early return true is exactly what #832 asked for, and it earns its keep — an allowlisted letter skips the numeric / URL / date-range / word-count checks below, none of which a bare letter can trip, so the early return states the intent instead of relying on them to be no-ops. toLowerCase() gives the case-insensitivity the issue's step 2 wanted, and I verified "Languages: c, r, d" survives as ["c","r","d"]. No change requested here.

if (tok.length < 2 || tok.length > 40) return false;
if (/^\d+$/.test(tok)) return false;
// A professional-profile link (or its bare "GitHub" / "LinkedIn" heading) is
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/google-docs/google-docs-skia-proxy-role-first-experience.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"phoneIsValid",
"skills"
],
"skillsCount": 11,
"skillsCount": 12,
"experienceCount": 2,
"educationCount": 1,
"projectsCount": 0,
Expand Down Expand Up @@ -74,7 +74,7 @@
"sectionSource": "regex",
"pageCount": 1,
"rawCharCount": 1428,
"extractedCharCount": 1146,
"extractedCharCount": 1147,
"sections": [
{
"name": "profile",
Expand All @@ -101,7 +101,7 @@
"hasSummary": false,
"experienceCount": 2,
"educationCount": 1,
"skillsCount": 11
"skillsCount": 12
},
"linkAnnotationCount": 0,
"disagreements": []
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/google-docs/google-docs-skia-proxy-role-first-experience.truth.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@
"note": "Role 2's employer line reads “Multicultural Engineering Program – State Polytechnic University”; `company` comes back as just the university — the program half is not lost, the parser puts it on `team`, but `experience.company` scores the `company` field alone. The identical shape is measured on unknown/single-column-title-below-anchor. Possibly a defensible org/team split rather than a defect — recorded rather than assumed, because ground truth's job is to state what the page says and let a human adjudicate."
},
"skills": {
"issue": null,
"status": "unfiled",
"note": "Two independent disagreements on one field: the single-letter token “C” is DROPPED from the Programming Languages row (the identical drop is measured on latex/multi-degree-coursework, so it is not fixture-specific), and “Fluent in Spanish” is admitted as a skill from the “Language:” row."
"issue": 833,
"status": "open",
"note": "The independent disagreement — “Fluent in Spanish” is admitted as a skill from the “Language:” row — is tracked by #833. The separate single-letter “C” drop was fixed by #832."
}
}
}
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/latex/deedy-resume-macfonts.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"skills",
"website_url"
],
"skillsCount": 22,
"skillsCount": 23,
"experienceCount": 6,
"educationCount": 3,
"projectsCount": 0,
Expand Down Expand Up @@ -83,7 +83,7 @@
"sectionSource": "markdown",
"pageCount": 1,
"rawCharCount": 3205,
"extractedCharCount": 2156,
"extractedCharCount": 2157,
"sections": [
{
"name": "profile",
Expand Down Expand Up @@ -114,7 +114,7 @@
"hasSummary": false,
"experienceCount": 6,
"educationCount": 3,
"skillsCount": 22
"skillsCount": 23
},
"linkAnnotationCount": 9,
"disagreements": []
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/latex/deedy-resume-openfonts.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"skills",
"website_url"
],
"skillsCount": 22,
"skillsCount": 23,
"experienceCount": 6,
"educationCount": 3,
"projectsCount": 0,
Expand Down Expand Up @@ -83,7 +83,7 @@
"sectionSource": "markdown",
"pageCount": 1,
"rawCharCount": 3207,
"extractedCharCount": 2158,
"extractedCharCount": 2159,
"sections": [
{
"name": "profile",
Expand Down Expand Up @@ -114,7 +114,7 @@
"hasSummary": false,
"experienceCount": 6,
"educationCount": 3,
"skillsCount": 22
"skillsCount": 23
},
"linkAnnotationCount": 9,
"disagreements": []
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/latex/multi-degree-coursework.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"website_url",
"work_authorization"
],
"skillsCount": 17,
"skillsCount": 18,
"experienceCount": 4,
"educationCount": 2,
"projectsCount": 3,
Expand Down Expand Up @@ -77,7 +77,7 @@
"sectionSource": "markdown",
"pageCount": 1,
"rawCharCount": 3428,
"extractedCharCount": 2625,
"extractedCharCount": 2626,
"sections": [
{
"name": "profile",
Expand Down Expand Up @@ -108,7 +108,7 @@
"hasSummary": false,
"experienceCount": 4,
"educationCount": 2,
"skillsCount": 17
"skillsCount": 18
},
"linkAnnotationCount": 6,
"disagreements": []
Expand Down
8 changes: 1 addition & 7 deletions tests/fixtures/pdfs/latex/multi-degree-coursework.truth.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,5 @@
"Raspberry Pi",
"iOS"
],
"knownWrong": {
"skills": {
"issue": null,
"status": "unfiled",
"note": "The single-letter token “C” is DROPPED from the Languages row while “C++” survives. Second independent measurement of the same drop (see google-docs/google-docs-skia-proxy-role-first-experience)."
}
}
"knownWrong": {}
Comment on lines 60 to +61

@s-annam s-annam Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (finding 5), non-blocking. With the entry resolved, this leaves the only empty knownWrong in the corpus — 10 of the 16 truth sidecars omit the key entirely and none carries {}. Dropping the key matches the neighbours.

Suggested change
],
"knownWrong": {
"skills": {
"issue": null,
"status": "unfiled",
"note": "The single-letter token “C” is DROPPED from the Languages row while “C++” survives. Second independent measurement of the same drop (see google-docs/google-docs-skia-proxy-role-first-experience)."
}
}
"knownWrong": {}
]

}
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/unknown/pdflib-leading-glyph-skills-header.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"phoneIsValid",
"skills"
],
"skillsCount": 8,
"skillsCount": 9,
"experienceCount": 1,
"educationCount": 1,
"projectsCount": 0,
Expand Down Expand Up @@ -74,7 +74,7 @@
"sectionSource": "markdown",
"pageCount": 1,
"rawCharCount": 462,
"extractedCharCount": 289,
"extractedCharCount": 290,
"sections": [
{
"name": "profile",
Expand All @@ -101,7 +101,7 @@
"hasSummary": false,
"experienceCount": 1,
"educationCount": 1,
"skillsCount": 8
"skillsCount": 9
},
"linkAnnotationCount": 0,
"disagreements": []
Expand Down