v3: variable_details.csv smoking Func:: names and DerivedVar feeders out of sync with refactored R functions

[DougManuel]

Problem

inst/extdata/variable_details.csv on v3 (d98e9d6) references smoking derivation functions that do not exist in the package, so rec_with_table() fails (or silently skips) when harmonizing through the worksheet path:

Error in get(custom_function_name) : object 'calculate_SMKG040' not found

Worksheet recEnd	Actual function in R/smoke-start.R
Func::calculate_SMKG040	calculate_SMKG040_cont()
Func::calculate_SMKG203_continuous	calculate_SMKG203_cont()
Func::calculate_SMKG203_from_combined	calculate_SMKG203_cont()
Func::calculate_SMKG207_continuous	calculate_SMKG207_cont()
Func::calculate_SMKG207_from_combined	calculate_SMKG207_cont()

The DerivedVar:: feeder lists in the same rows also no longer match the function signatures:

Variable (block)	Worksheet feeders	Function signature expects
SMKG040_cont (PUMF 2001–2003, 2005–2014)	[SMKG203_pre2005, SMKG207_pre2005] / [SMKG203_2005plus, SMKG207_2005plus] (categorical)	calculate_SMKG040_cont(SMKG203_cont, SMKG207_cont) (continuous midpoints)
SMKG203_cont (PUMF 2015+)	[SMK_005, SMKG040] (grouped category)	calculate_SMKG203_cont(SMK_005, SMKG040_cont)
SMKG207_cont (PUMF 2015+)	[SMK_005, SMK_030, SMKG040] — three feeders	calculate_SMKG207_cont(SMK_030, SMKG040_cont) — two parameters (third feeder lands in output_format)
SMKG207_cont / SMK_207 (Master 2015+)	[SMK_005, SMK_030, SMK_040] — three feeders	same two-parameter signature

Impact

• SMKG040_cont cannot be derived for PUMF 2001–2014 or Master 2001–2014, which breaks the age_start_smoking chain through rec_with_table().
• SMKG203_cont / SMKG207_cont (2015+) and SMK_040 / SMK_203 / SMK_207 (Master 2015+) hit the same phantom-name failure.

The test suite passes because it calls the calculate_*() functions directly with vectors; Func:: resolution from the worksheet is only exercised by an actual rec_with_table() run.

Proposed fix (validated locally)

Exact cell replacements in variable_details.csv:

• Func::calculate_SMKG040 → Func::calculate_SMKG040_cont
• Func::calculate_SMKG203_continuous and Func::calculate_SMKG203_from_combined → Func::calculate_SMKG203_cont
• Func::calculate_SMKG207_continuous and Func::calculate_SMKG207_from_combined → Func::calculate_SMKG207_cont
• DerivedVar::[SMKG203_pre2005, SMKG207_pre2005] and DerivedVar::[SMKG203_2005plus, SMKG207_2005plus] → DerivedVar::[SMKG203_cont, SMKG207_cont]
• DerivedVar::[SMK_005, SMKG040] → DerivedVar::[SMK_005, SMKG040_cont]
• DerivedVar::[SMK_005, SMK_030, SMKG040] → DerivedVar::[SMK_030, SMKG040_cont]
• DerivedVar::[SMK_005, SMK_030, SMK_040] → DerivedVar::[SMK_030, SMK_040]

This matches the era design documented in calculate_SMKG040_cont() (2001–2014: combine 203/207 midpoints into 040; 2015+: derive 203/207 from directly-recoded SMKG040_cont via the SMK_005/SMK_030 gates). The packaged .rda worksheets would need rebuilding from the CSVs afterward. We have these fixes applied locally and are validating them against the CSHM (cshgm-dev) harmonization pipeline; happy to open a PR.

Context

Found while integrating cchsflow v3 into the CSHM pipeline, which passes the v3 CSV worksheets explicitly to rec_with_table().

[DougManuel]

A third, related defect found while validating: time_quit_smoking_daily cannot be derived for any PUMF cycle through rec_with_table().

Its worksheet rows use a single block covering all PUMF and Master databases with DerivedVar::[SMKDSTY_cat5, SMK_09A_cont, SMK_09C]. SMK_09C is Master-only (no PUMF rows in variable_details.csv), so on PUMF databases the feeder can never be recoded and recode_derived_variables() skips the variable with a "could not be derived" warning — even though calculate_time_quit_smoking_daily() explicitly supports PUMF via its SMK_09C = NULL default.

Fix validated locally: split the three rows (Func, NA::a, NA::b) into two blocks —

• PUMF databases (cchs2001_p … cchs2019_2020_p): DerivedVar::[SMKDSTY_cat5, SMK_09A_cont] (function falls back to the midpoint feeder)
• Master databases (cchs2001_m … cchs2021_m, cchs2023_m): keep DerivedVar::[SMKDSTY_cat5, SMK_09A_cont, SMK_09C]

Other cessation variables are unaffected: time_quit_smoking ([SMK_09A_cont, SMK_06A_cont]) and time_quit_smoking_complete use feeders with PUMF coverage. The same audit pattern may be worth applying package-wide: any DerivedVar feeder that has no rows for a database the derived variable claims should be flagged at build time.

[DougManuel]

Two more defects in the same class, found by auditing every DerivedVar feeder list against the corresponding function formals (feeders are passed positionally by rec_with_table()):

4. cigs_per_day — feeder order reversed. Worksheet: DerivedVar::[SMK_204, SMK_208, SMKDSTY_original]; signature: calculate_cigs_per_day(SMKDSTY_original, SMK_204, SMK_208). The function therefore gates on SMK_204 (cigarettes/day) as if it were smoking status — only respondents smoking exactly 1 cigarette/day pass the status == 1 branch. Empirically (CCHS 2001 PUMF, 1% sample): SMK_204 is healthy (22.9% non-missing, median 15 cpd among daily smokers) but cigs_per_day came out 0.4% non-missing — 5 of 301 daily smokers. Fix: reorder to DerivedVar::[SMKDSTY_original, SMK_204, SMK_208].

5. pack_years_der — wrong variables in the occasional-smoker slots. Worksheet feeds SMK_204/SMK_208 (cigs/day, current-daily and former-daily) into the cigs_occasional/days_per_month parameters, which the status 2/3 branches use as cigarettes-per-occasion × days-smoked-per-month. The correct CCHS items are SMK_05B (cigs/day on days smoked, occasional) and SMK_05C (days smoked/month) — as in v2's pack_years_fun(). Result is silently wrong pack-years for occasional smokers (e.g. a former-daily-now-occasional smoker gets their former daily CPD treated as days/month). Fix: replace SMK_204, SMK_208 with SMK_05B, SMK_05C in both pack_years_der feeder lists (PUMF and Master).

Both fixes are applied in our local checkout and under validation. Suggest the package-wide build-time check mentioned above also compare each DerivedVar list length/order against formals() of the named function — all five defects in this issue would have been caught.

[DougManuel]

Validation against real PUMF data surfaced three more, distinct from the Func/feeder issues above:

6. SMKDSTY_cat5 — 2015+ block misapplies the pre-2015 collapse rules. Post-2015 SMKDVSTY codes (DDI): 1 current daily, 2 current occasional, 3 former daily, 4 former occasional, 5 experimental, 6 lifetime abstainer. The 2015+ block has [2,3]→2 and [4,5]→4 with no rule producing cat5 3 (former daily) and no rule for raw 6 (never → falls to else → NA(b)). Effect in CCHS 2015–16: 28,202 former-daily smokers classified as "occasional", 44,002 never-smokers as missing — and time_quit_smoking_daily (which gates on cat5 == 3) derives for nobody in 2015+. Correct 2015+ map: 1→1, 2→2, 3→3, [4,5]→4, 6→5.

7. SMKG040 / SMKG040_cont — false 2019-20 PUMF claim. Raw SMKG040 exists only in the 2015-16 and 2017-18 PUMF (confirmed in DDI and in the release files). The 2015+ [SMKG040] blocks list cchs2019_2020_p, which can never recode. age_start_smoking is genuinely unavailable in the 2019-20 PUMF; the worksheet (and the variables.csv "PUMF: 2001-2021" notes) should say so.

8. SMKDSTY_cat3 — no PUMF value rows at all. The PUMF block contains only 6→3 plus NA rows; the [1,3]→1 / [4,5]→2 (pre-2015) and [1,2]→1 / [3,5]→2 (2015+) rows exist only for Master databases. On any PUMF cycle, every smoker falls to else → NA(b).

Fixes for 6 and 7 are applied in our local checkout and validated; happy to fold all of these into the same PR.

Separately: the 2015-16 file in cchsflow-data has a corrupted SMKG035 (valid skip/not stated collapsed into code 11, no NAs) — filed in that repo since it is a data-release defect, not a worksheet one.

[DougManuel]

One more, with an unusual root cause:

9. SMKG01C_cont — cchs2015_2016_p needs a code-11 exception (StatCan-shipped PUMF defect). The official 2015-16 PUMF (confirmed in StatCan's own data dictionary, p. 123) pools valid skip and not-stated into SMKG035 code 11 ("age 50 or older"): 11 = 45,737 (46.3% weighted!) with no 96/99 rows, versus ~90–95 genuine "50+" respondents in adjacent cycles. The worksheet's uniform 11 → 55 midpoint therefore assigned age 55 at first cigarette to ~44k never-smokers in that cycle, cascading into age_first_cigarette (came out 100% non-missing). Not in the StatCan errata; full forensics in Big-Life-Lab/cchsflow-data#3.

Fix (in the local branch): split the code-11 row — cchs2015_2016_p maps 11 → NA(a) with a documenting note (sacrifices the ~95 unrecoverable genuine 50+ responses); all other cycles keep 11 → 55. Validated: 2015-16 age_first_cigarette returns to a plausible non-missing rate in line with 2001 and 2019-20.