From 98bbbe59a07d615c3b8f69f29ac82ac4624504b2 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Wed, 10 Jun 2026 11:58:03 -0400 Subject: [PATCH 1/6] Protocol v0.3.0: missing data and imputation plan, Appendix D Specify the imputation approach in section 3.4.1: where-matrix MICE restricted to item non-response (NA(b)), m = 5; survey design variables (cycle, sampling weight) and sociodemographic predictors in the imputation model for congeniality with the APC analysis; derived variables recomputed from imputed feeders. Tables 1a/1b specified as a single table with unweighted n and survey-weighted statistics (1a disclosing missing data by NA type; 1b averaged across the m completed datasets), with a fully unweighted appendix variant. Add Appendix D (docs/protocol/appendix-imputation.qmd), adapting the DemPoRTv2 imputation assessment framework: CSHM-specific missing-data context (tagged-NA taxonomy, PUMF cycle-level gaps for 2019-20 and 2022, the 2015-16 SMKG035 recode), the model specification with the three improvements over the DemPoRTv2 implementation, the Table 1a/1b presentation, the four-phase diagnostic and sensitivity plan (including delta-adjustment for cycle-absent variables with the RDC Master files as an external check), and an implementation checklist tied to issues #3 and #4. --- docs/protocol/appendix-imputation.qmd | 150 ++++++++++++++++++++++++++ docs/protocol/full-protocol.qmd | 17 ++- 2 files changed, 163 insertions(+), 4 deletions(-) create mode 100644 docs/protocol/appendix-imputation.qmd diff --git a/docs/protocol/appendix-imputation.qmd b/docs/protocol/appendix-imputation.qmd new file mode 100644 index 0000000..2f3cae5 --- /dev/null +++ b/docs/protocol/appendix-imputation.qmd @@ -0,0 +1,150 @@ +--- +title: "Appendix D — Missing data and imputation plan" +subtitle: "Detailed specification and assessment plan for multiple imputation in the CSHM" +--- + +## Overview + +This appendix specifies the missing-data strategy for the Canadian Smoking Histories Model (CSHM) and the diagnostic and sensitivity analyses that will accompany it. It adapts the imputation assessment framework developed for DemPoRTv2 (`DemPoRT-V2-dev`, `appendix-imputation-assessment.qmd`), with three methodological improvements identified during CSHM development: protection of structural missingness through an explicit `where` matrix, inclusion of survey design variables in the imputation model, and deterministic recomputation of derived variables after imputation. + +**Method:** Multiple imputation by chained equations (MICE) + +**Number of imputations:** *m* = 5, with *maxit* = 5 iterations (production configuration; *m* = 1 in the `dev` profile for fast iteration). The adequacy of *m* will be assessed by examining the fraction of missing information (FMI) for key APC model coefficients; if FMI exceeds 0.5 for any coefficient, *m* will be increased. + +**Key dependencies:** mice, haven, naniar, VIM + +## Missing data context + +The CSHM pipeline preserves the reason for missingness throughout, using `haven::tagged_na()` for continuous variables and explicit factor levels for categorical variables: + +| Tag | Meaning | Treatment | +|-----|---------|-----------| +| NA(a) | Not applicable (e.g., initiation age for never-smokers) | Structural — never imputed | +| NA(b) | Don't know / refused / not stated | Item non-response — imputed under MAR | +| NA(c) | Not asked in this survey cycle | Cycle-level absence — imputed under a stronger MAR assumption, with sensitivity analysis | + +Three sources of missingness are specific to this study: + +1. **Item non-response (NA(b)).** Conventional missing data on smoking history questions among respondents in the question's universe. The MAR assumption is reasonable, and these cells are the primary imputation target. + +2. **Cycle-level absence (NA(c)).** Key smoking variables are unavailable in recent PUMF cycles because Statistics Canada moved source questions to Master-only collection or dropped them: + + | Variable | PUMF cycles missing | Reason | + |----------|--------------------|--------| + | `age_start_smoking` | 2019–20, 2022 | Raw SMKG040 absent from the 2019–20 PUMF onward | + | `SMKDSTY_original` (and downstream `cigs_per_day`, `pack_years_der`) | 2022 | SMK_05D moved to Master-only collection | + | `time_quit_smoking_daily` | 2022 | Feeder variables Master-only in 2022 | + + Imputing across cycle-level absence assumes the relationships between observed predictors and the missing variable are stable across cycles — a stronger form of MAR that will be evaluated explicitly in Phase 4. Master file analyses at the Research Data Centre, where these variables are observed, provide a direct check. + +3. **The 2015–16 SMKG035 defect.** The official 2015–16 PUMF pools valid-skip and not-stated responses into SMKG035 category 11 ("age 50 or older"); see Big-Life-Lab/cchsflow-data#3. The harmonization recodes category 11 to NA(a) for that cycle. The practical consequence for imputation: true never-smokers are correctly NA(a) and excluded, while the ~1,400 ever-smokers whose responses are hidden in the pooled category become identifiable — through the smoking gate questions (`smoked_100_lifetime`, `SMKDSTY_original`) — as ever-smokers with missing `age_first_cigarette`, and are imputed. The ~95 genuine "age 50+" responses are imputed rather than assigned the category midpoint; this is an accepted loss (~0.2% of ever-smokers in the cycle) that avoids assigning age 55 at first cigarette to roughly 44,000 never-smokers. + +## Imputation model specification + +### Cells eligible for imputation: the `where` matrix + +MICE will be called with an explicit `where` matrix in which a cell is `TRUE` only if its value is tagged NA(b) (or NA(c), for variables imputed across cycle-level absence). This differs from the DemPoRTv2 implementation, which converts all tagged NAs to plain `NA` before imputation: because `haven::tagged_na()` values *are* `NA` to `mice()`, that approach silently imputes structural missingness — for example, inventing quit times for never-smokers. The `where` matrix makes the universe logic automatic: a never-smoker's NA(a) on `age_first_cigarette` is never a candidate cell, while an ever-smoker's NA(b) on the same variable is. + +For categorical variables, the `"NA(a)"` and `"NA(c)"` factor levels are removed from the modelling frame before imputation (and restored afterward) so that polytomous regression cannot assign a structural-missing label as an imputed value. + +### Variables in the imputation model + +Predictors are selected by the `imputation-predictor` role in `worksheets/cshm-variables.csv`, following the principle of congeniality: the imputation model is at least as rich as the analysis model. The prespecified set: + +- **Survey design and demographics:** survey cycle, sampling weight (`WTS_M`), sex, age (`DHHGAGE_cont`), province, education +- **Smoking history:** smoking status (`SMKDSTY_original`), `age_first_cigarette`, `age_start_smoking`, `time_quit_smoking_daily`, `cigs_per_day` + +Including the sampling weight and survey cycle as *predictors* makes the imputation design-consistent without requiring a weighted imputation algorithm — a second improvement over the DemPoRTv2 implementation, which omits design variables. Age, sex, and cycle jointly encode the cohort dimension of the APC analysis model, preserving congeniality with the Stage 8 models. + +### Derived variables + +Derived smoking variables (`pack_years_der`, and any variable with a cchsflow derivation chain) are not imputed directly. Their feeder variables are imputed, and the derived values are recomputed deterministically from the completed feeders using the same cchsflow derivation functions used at harmonization. This replaces the DemPoRTv2 approach of hardcoded post-imputation consistency fixes (e.g., setting pack-years to zero for non-smokers) with the worksheet-encoded dependency chain, guaranteeing internal consistency by construction. + +### Imputed data storage and downstream use + +All *m* completed datasets are retained (long format keyed by imputation number). + +- **Table 1b** statistics are computed within each completed dataset and averaged across imputations. +- **APC models (Stage 8)** are initially fitted on the first completed dataset, with a documented upgrade path to fitting per imputation and pooling coefficients by Rubin's rules. FMI from the pooled fits (Phase 2.3) determines whether the pooled analysis is required for the primary results. + +## Descriptive tables 1a and 1b + +Baseline characteristics are reported twice, with complementary purposes: + +- **Table 1a (pre-imputation)** discloses the missing data: each variable shows its observed categories plus separate rows for NA(a), NA(b), and NA(c), so the amount and type of missingness is visible before any imputation. +- **Table 1b (post-imputation)** describes the analysis population: the same layout after imputation (averaged across the *m* completed datasets), in which NA(b) rows are resolved and structural NA(a)/NA(c) rows remain visible. + +Both tables present **unweighted n with survey-weighted percentages** (and weighted median/IQR for continuous variables) in a single table: the unweighted n discloses the information content of each cell — including the missing-data cells in Table 1a — while the weighted statistics describe the Canadian population. A fully unweighted variant is produced as an appendix table. Weighted statistics use the CCHS sampling weight (`WTS_M`); no variance estimation is required in Table 1, so bootstrap weights (not released with the PUMF) are not needed. Tables are stratified by sex and survey cycle, and examined for discontinuities across CCHS design eras (2001–2005, 2007–2014, 2015–2021, 2022+). + +## Phase 1: Missing data diagnostics (pre-imputation) + +### 1.1 Quantify missingness + +Largely delivered by Table 1a. A supplementary table summarizes missingness for all imputation-model variables, separated by type: NA(b) item non-response as a proportion of the variable's universe, and NA(c) cycle-level absence as a proportion of the cohort contributed by non-collecting cycles. + +**Criteria:** flag variables with more than 50% total missingness for specialized handling. + +### 1.2 Visualize patterns + +Co-occurrence of missing values via `naniar::gg_miss_upset()` or `VIM::aggr()`, with attention to the block structure driven by cycle-level absence (the 2019–20 and 2022 PUMF blocks identified above). + +### 1.3 Mechanism assessment + +Associations between missingness indicators and observed variables (age, sex, education, cycle) to support the MAR assumption for item non-response. For cycle-level absence, document the predictors available in collecting cycles and assess the plausibility of stable relationships across cycles. + +## Phase 2: Technical validation + +### 2.1 Convergence + +Trace plots (`mice::plot()`) for the mean and SD of all imputed variables; chains should intermingle without trends. Variables imputed across cycle-level absence receive particular attention; increase `maxit` if needed. + +### 2.2 Imputation-model health + +Review `mice` logged events (`loggedEvents`) for dropped collinear or constant variables — surfaced as an explicit pipeline check rather than silently suppressed (see cshm-dev issue #3). + +### 2.3 Between-imputation diagnostics + +FMI and relative increase in variance for key APC coefficients from `mice::pool()`. FMI > 0.5 triggers an increase in *m*. + +## Phase 3: Plausibility checks + +### 3.1 Distributional similarity + +Density plots of observed versus imputed values (`mice::densityplot()`) for `age_first_cigarette`, `age_start_smoking`, `time_quit_smoking_daily`, and `cigs_per_day`; category-share comparisons for `SMKDSTY_original`. Imputed values must respect logical bounds (initiation age within the configured floor and current age; non-negative quit times). + +### 3.2 Bivariate relationships + +Verify that observed-data relationships persist in imputed records: initiation age by birth cohort and sex (the core APC relationships), quit time by age. For cycle-absent variables, compare joint distributions in collecting cycles against imputed values in non-collecting cycles. + +### 3.3 Derived-variable consistency + +Recomputed `pack_years_der` distributions are compared pre- and post-imputation; never-smokers have zero pack-years by construction (recomputation, not post-hoc fixing). + +## Phase 4: Sensitivity analyses + +### 4.1 Complete-case comparison + +Fit the initiation and cessation APC models on the multiply imputed data and on the complete-case subset. Discrepancies quantify the bias that listwise deletion would introduce under MAR. + +### 4.2 Imputation-method robustness + +For the variables with the highest missingness, compare predictive mean matching against random forest imputation for continuous variables, and polytomous regression against random forest for `SMKDSTY_original`. + +### 4.3 Sensitivity to MAR for cycle-absent variables + +Delta-adjustment analysis for `age_start_smoking` in 2019–20 (and the 2022 smoking block): shift imputed values by ±0.5 SD, refit the APC models, and assess robustness of the initiation-rate trends. Because these cycles never measured the variable in the PUMF, MAR cannot be verified empirically there — but the **Master files observe these variables**, so the RDC analysis provides a direct external check unavailable to most delta-adjustment settings. + +### 4.4 Era-stratified stability + +Fit the APC models on (a) cycles where all key variables are observed (2001–2017/18 PUMF) and (b) the full imputed cohort, and compare coefficient estimates. This doubles as the design-era discontinuity check specified in §3.4.1 of the protocol. + +## Implementation checklist + +- [ ] Assign `imputation-predictor` roles in `worksheets/cshm-variables.csv` (prespecified set above) +- [ ] Rewrite `impute_data()` with the `where` matrix, factor-level protection, tag restoration on write-back, and `loggedEvents` surfacing (cshm-dev issue #3) +- [ ] Extend the descriptive engine with a `weight_var` argument (weighted %, weighted median/IQR); Tables 1a/1b present unweighted n with weighted statistics +- [ ] Recompute derived variables from imputed feeders via the cchsflow derivation chain +- [ ] Retain all *m* completed datasets; average Table 1b across imputations +- [ ] Add an imputation-diagnostics target and Quarto report (trace plots, density plots, FMI table) +- [ ] Golden-record and tagged-NA preservation tests (cshm-dev issue #4) +- [ ] Delta-adjustment sensitivity analysis for cycle-absent variables; RDC cross-check at the Master-file stage diff --git a/docs/protocol/full-protocol.qmd b/docs/protocol/full-protocol.qmd index d1d0e7f..90c7750 100644 --- a/docs/protocol/full-protocol.qmd +++ b/docs/protocol/full-protocol.qmd @@ -2,9 +2,12 @@ title: "A Canadian Smoking Histories Model: a study protocol to generate smoking cohorts from 1940 and project to 2050" status: "Draft" version-summary: - date: "2026-02-24" - version: "0.2.5" + date: "2026-06-11" + version: "0.3.0" version-history: + - version: "0.3.0" + date: "2026-06-11" + description: "Missing data and imputation plan: where-matrix MICE restricted to item non-response (m = 5), survey design variables in the imputation model, derived variables recomputed from imputed feeders. Tables 1a/1b specified as unweighted n with survey-weighted statistics in a single table. Added Appendix D (detailed imputation plan and four-phase assessment, adapting the DemPoRTv2 framework). Documented PUMF cycle-level smoking variable gaps (2019-20, 2022) and the 2015-16 SMKG035 recode." - version: "0.2.0" date: "2026-03-09" description: "Update to include reporting guideline placeholders. Added potential measurement error and sensitivity analyses. Equity subgroup analysis removed from primary analysis. Added total cigarettes smoked in Canada by year as an expected output. Added sensitivity analyses for spline type (NSP vs RCS) and mortality adjustment method (MPoRT vs Peto)." @@ -139,8 +142,8 @@ The CSHM uses a two-stage analytical framework. In the first stage, we estimate Analysis-ready datasets will be produced through a standardized pipeline. 1. **Cleaning:** Distribution checks and truncation of extreme values for continuous variables (e.g., smoking intensity). -2. **Imputation:** Missing data on smoking status, initiation age, and cessation age will be addressed using Multiple Imputation by Chained Equations (MICE). The imputation model will include sociodemographic predictors (age, sex, province, education, income) and health status indicators. -3. **Descriptive statistics:** Baseline characteristics will be reported pre-imputation (Table 1a) and post-imputation (Table 1b). Tables 1 will include stratification for sex and survey year. Descriptive statistics will be examined for potential discontinuities across CCHS design eras (2001–2005, 2007–2014, 2015–2021, 2022+), which reflect major changes to the survey frame, recruitment, and collection mode [@gagne2017; @backinger2008]. Although smoking variables can be harmonized across cycles, changes to the sampling frame and mode may introduce measurement differences that are not fully correctable through harmonization alone [@chen2020joinpoint]. +2. **Imputation:** Item non-response (don't know/refused; tagged NA(b)) on smoking status, initiation age, and cessation timing will be addressed using Multiple Imputation by Chained Equations (MICE) with *m* = 5 imputations. Imputation is restricted to item non-response through an explicit `where` matrix; structural missingness — not-applicable (NA(a), e.g., initiation age for never-smokers) and not-asked-this-cycle (NA(c)) — is preserved and never imputed directly. The imputation model includes the survey design variables (cycle, sampling weight) and sociodemographic predictors (age, sex, province, education) alongside the smoking history variables, preserving congeniality with the APC analysis models. Derived variables (e.g., pack-years) are recomputed from their imputed feeder variables rather than imputed directly. The full specification, including the diagnostic and sensitivity-analysis plan, is given in [Appendix D: Missing data and imputation plan](appendix-imputation.qmd). +3. **Descriptive statistics:** Baseline characteristics will be reported pre-imputation (Table 1a), which discloses the amount and type of missing data, and post-imputation (Table 1b), averaged across the *m* completed datasets. Both tables present unweighted n with survey-weighted percentages (and weighted median/IQR for continuous variables) — the unweighted n discloses the information content of each cell while the weighted statistics describe the population; a fully unweighted variant is provided as an appendix table. Tables 1 will include stratification for sex and survey year. Descriptive statistics will be examined for potential discontinuities across CCHS design eras (2001–2005, 2007–2014, 2015–2021, 2022+), which reflect major changes to the survey frame, recruitment, and collection mode [@gagne2017; @backinger2008]. Although smoking variables can be harmonized across cycles, changes to the sampling frame and mode may introduce measurement differences that are not fully correctable through harmonization alone [@chen2020joinpoint]. ### 3.4.2 Age-period-cohort modelling @@ -335,6 +338,12 @@ All code will be hosted on GitHub under the MIT licence. The repository will inc > STRESS: Implementation details; GATHER: Access to data and analytics +## Appendix D: Missing data and imputation plan + +The detailed missing-data specification — the tagged-NA taxonomy, the `where`-matrix MICE design, cycle-level absence handling, the Table 1a/1b weighting presentation, and the four-phase diagnostic and sensitivity-analysis plan — is maintained as a standalone document: [Appendix D: Missing data and imputation plan](appendix-imputation.qmd). It adapts the DemPoRTv2 imputation assessment framework with improvements developed for this study. + +> STROBE: Statistical methods (missing data); GATHER: Data processing + :::: :::: {.section-body page-break="true"} From a3a7e7c7dfaf3d8c035dba2d783293ce9b9b1b28 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Wed, 10 Jun 2026 15:29:55 -0400 Subject: [PATCH 2/6] Implement where-matrix imputation and weighted Tables 1a/1b MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 5 (R/imputation.R): MICE with an explicit where matrix marking only NA(b) cells; tagged NA(a)/NA(c) survive untouched because the write-back copies only where-matrix cells into the original data, and the stage stops if MICE declines to impute a flagged cell (constant/ collinear) rather than silently degrading an NA(b) tag to plain NA. Factor NA(x) levels are removed from the modelling frame and restored implicitly. All m completed datasets are retained, with an imputed-cells audit and surfaced loggedEvents. Descriptive engine: weight_var argument adds survey-weighted percent (categories and NA-type rows) and weighted median/IQR via a midpoint-ECDF interpolating quantile; unchanged output when NULL. get_cshm_desc_data_mi() averages Table 1b statistics across the m imputations. Table cells render unweighted n with weighted percent / weighted median (IQR). Table rows are now selected by the table1 role as documented (was predictor); SDCFIMM/SDCGCGT gain the table1 role. Worksheet: imputation-predictor roles per protocol Appendix D, plus 15 auxiliary covariates (marital status, alcohol, BMI, general and mental health, stress, belonging, six chronic conditions, energy expenditure — all cchsflow-v3-covered to 2019-20) with 16 feeder rows; 72 rows total. Coverage validation passes (known 2022 gap only). _targets.R: weighted Table 1a; MI-averaged Table 1b; APC fits imputation 1 (Appendix D documents the Rubin-pooling upgrade path). Tests: 41 new (where-matrix contract, tagged-NA round trip, MICE declined-cell guard, weighted quantiles, multi-role fixtures); 85 pass total. --- R/create-descriptive-tables.R | 33 +++- R/descriptive-data.R | 64 +++++++- R/get-descriptive-data.R | 57 ++++++- R/imputation.R | 201 ++++++++++++++++------- _targets.R | 16 +- tests/testthat/test-descriptive-tables.R | 8 +- tests/testthat/test-imputation.R | 131 +++++++++++++++ worksheets/cshm-variables.csv | 57 +++++-- 8 files changed, 471 insertions(+), 96 deletions(-) create mode 100644 tests/testthat/test-imputation.R diff --git a/R/create-descriptive-tables.R b/R/create-descriptive-tables.R index 497a119..daf6bf4 100644 --- a/R/create-descriptive-tables.R +++ b/R/create-descriptive-tables.R @@ -5,30 +5,49 @@ NA_c_label <- "Missing from survey" categorical_predictor_footnote <- paste0( - "For categorical variables the values are displayed as N (percent). ", + "For categorical variables the values are displayed as unweighted N ", + "(survey-weighted percent where weights were supplied; otherwise unweighted percent). ", "Percents may not sum to 100 due to missingness." ) continuous_predictor_footnote <- paste0( - "For continuous predictors, the values are displayed as min - max, median (IQR)." + "For continuous predictors, the values are displayed as min - max, median (IQR); ", + "median and IQR are survey-weighted where weights were supplied." ) # ---- Formatting helpers ----------------------------------------------------- +# When the descriptive data carry weighted columns (weight_var supplied to the +# engine), cells show unweighted n with weighted percent / weighted median (IQR) +# per protocol v0.3.0 §3.4.1. Without weights they fall back to unweighted stats. format_cat_descriptive_data <- function(descriptive_data_row) { if (is.na(descriptive_data_row[1, "n"]) || descriptive_data_row[1, "n"] == 0) { return("No data") } + pct <- if ("wtd_percent" %in% colnames(descriptive_data_row) && + !is.na(descriptive_data_row[1, "wtd_percent"])) { + descriptive_data_row[1, "wtd_percent"] + } else { + descriptive_data_row[1, "percent"] + } formatted_n <- format(descriptive_data_row[1, "n"], big.mark = ",") - paste0(formatted_n, "\n (", round(descriptive_data_row[1, "percent"] * 100, 1), ")") + paste0(formatted_n, "\n (", round(pct * 100, 1), ")") } format_cont_descriptive_data <- function(descriptive_data_row) { - if (descriptive_data_row[1, "n"] == 0) return("No data") + if (is.na(descriptive_data_row[1, "n"]) || descriptive_data_row[1, "n"] == 0) { + return("No data") + } + weighted <- "wtd_median" %in% colnames(descriptive_data_row) && + !is.na(descriptive_data_row[1, "wtd_median"]) + med <- if (weighted) descriptive_data_row[1, "wtd_median"] + else descriptive_data_row[1, "median"] + p25 <- if (weighted) descriptive_data_row[1, "wtd_percentile25"] + else descriptive_data_row[1, "percentile25"] + p75 <- if (weighted) descriptive_data_row[1, "wtd_percentile75"] + else descriptive_data_row[1, "percentile75"] paste0( descriptive_data_row[1, "min"], " - ", descriptive_data_row[1, "max"], ",\n", - descriptive_data_row[1, "median"], - " (", descriptive_data_row[1, "percentile25"], - " - ", descriptive_data_row[1, "percentile75"], ")" + med, " (", p25, " - ", p75, ")" ) } diff --git a/R/descriptive-data.R b/R/descriptive-data.R index e299307..7c49e97 100644 --- a/R/descriptive-data.R +++ b/R/descriptive-data.R @@ -1,19 +1,30 @@ # descriptive-data.R # CSHM-specific wrapper for calculating descriptive statistics. # Pipeline targets: table_1a_data, table_1b_data +# +# Presentation per protocol v0.3.0 §3.4.1: single table with unweighted n and +# survey-weighted statistics (weighted % for categories and NA-type rows; +# weighted median/IQR for continuous). Table 1b averages across the m +# completed imputation datasets. #' Calculate descriptive statistics for the CSHM study population #' -#' Computes statistics for all predictor variables stratified by model-stratifier. -#' No row stratification is applied in the base table. +#' Computes statistics for all predictor variables stratified by +#' model-stratifier. No row stratification is applied in the base table. #' #' @param data Cleaned or imputed study data frame #' @param variables_sheet Variables worksheet data frame #' @param variable_details_sheet Variable details worksheet data frame +#' @param weight_var Survey weight column name (e.g. "WTS_M"). NULL = unweighted. #' @return Data frame of descriptive statistics (input to create_descriptive_table) -get_cshm_desc_data <- function(data, variables_sheet, variable_details_sheet) { - predictor_vars <- select_vars_by_role("predictor", variables_sheet) +get_cshm_desc_data <- function(data, variables_sheet, variable_details_sheet, + weight_var = NULL) { + # Table rows are selected by the table1 role (the documented wiring); + # columns are stratified by model-stratifier (sex). table1-stratifier + # drives the cycle-specific appendix table, not this base table. + predictor_vars <- select_vars_by_role("table1", variables_sheet) sex_stratifier <- select_vars_by_role("model-stratifier", variables_sheet)[1] + stopifnot(!is.na(sex_stratifier)) # Only describe variables that were actually harmonized into data # (some variables in the sheet may be absent if no variable_details rows matched) @@ -32,6 +43,49 @@ get_cshm_desc_data <- function(data, variables_sheet, variable_details_sheet) { variables_sheet = variables_sheet, variables_details_sheet = variable_details_sheet, variables = available, - stratify_config = stratify_config + stratify_config = stratify_config, + weight_var = weight_var ) } + +#' Descriptive statistics averaged across multiple imputations (Table 1b) +#' +#' Computes the descriptive statistics within each completed dataset and +#' averages the estimates across imputations (means for central statistics +#' and proportions; min of minima, max of maxima). With m = 1 this reduces +#' to a single get_cshm_desc_data() call. +#' +#' @param imputation_result Output of impute_data() (list with $datasets) +#' @param variables_sheet Variables worksheet data frame +#' @param variable_details_sheet Variable details worksheet data frame +#' @param weight_var Survey weight column name. NULL = unweighted. +#' @return Data frame of descriptive statistics, averaged across imputations +get_cshm_desc_data_mi <- function(imputation_result, variables_sheet, + variable_details_sheet, weight_var = NULL) { + datasets <- imputation_result$datasets + stopifnot(length(datasets) >= 1) + + per_imp <- lapply(datasets, function(d) { + get_cshm_desc_data(d, variables_sheet, variable_details_sheet, weight_var) + }) + if (length(per_imp) == 1) return(per_imp[[1]]) + + stacked <- dplyr::bind_rows(per_imp, .id = ".imp") + + mean_cols <- intersect( + c("median", "percentile25", "percentile75", "n", "percent", + "wtd_percentile25", "wtd_median", "wtd_percentile75", "wtd_percent"), + colnames(stacked) + ) + key_cols <- setdiff(colnames(stacked), c(".imp", mean_cols, "min", "max")) + + stacked |> + dplyr::group_by(dplyr::across(dplyr::all_of(key_cols))) |> + dplyr::summarise( + dplyr::across(dplyr::all_of(mean_cols), ~ mean(.x, na.rm = FALSE)), + min = if (all(is.na(min))) NA else min(min, na.rm = TRUE), + max = if (all(is.na(max))) NA else max(max, na.rm = TRUE), + .groups = "drop" + ) |> + as.data.frame() +} diff --git a/R/get-descriptive-data.R b/R/get-descriptive-data.R index b17e512..2f674fa 100644 --- a/R/get-descriptive-data.R +++ b/R/get-descriptive-data.R @@ -1,14 +1,42 @@ # get-descriptive-data.R # Calculate descriptive statistics for the study population. -# Ported from DemPoRT-V2-dev (origin/dev). +# Ported from DemPoRT-V2-dev (origin/dev); extended with survey-weighted +# statistics (weight_var) per protocol v0.3.0 §3.4.1 / Appendix D. + +#' Weighted quantile (midpoint-ECDF with linear interpolation) +#' +#' Interpolates linearly between order statistics positioned at the midpoints +#' of their cumulative weight intervals (the standard weighted analogue of a +#' continuous sample quantile). With equal weights the weighted median equals +#' the type-7 sample median; other quantiles are close but not identical to +#' type 7. Values beyond the first/last midpoint take the boundary value. +#' +#' @param x Numeric vector (NAs removed by caller) +#' @param w Numeric weights aligned with x +#' @param probs Probabilities in [0, 1] +#' @return Numeric vector of weighted quantiles +weighted_quantile <- function(x, w, probs) { + if (length(x) == 0) return(rep(NA_real_, length(probs))) + if (length(x) == 1) return(rep(x, length(probs))) + ord <- order(x) + x <- x[ord]; w <- w[ord] + cw <- cumsum(w) + midpoints <- (cw - w / 2) / sum(w) + stats::approx(midpoints, x, xout = probs, rule = 2, ties = "ordered")$y +} get_descriptive_data <- function( data, variables_sheet, variables_details_sheet, variables, - stratify_config + stratify_config, + weight_var = NULL ) { + use_weights <- !is.null(weight_var) + if (use_weights && !weight_var %in% colnames(data)) { + stop("weight_var '", weight_var, "' not found in data.") + } descriptive_data <- data.frame( variable = c(), cat = c(), @@ -60,7 +88,8 @@ get_descriptive_data <- function( current_stratifier_info$stratifier_combination[[strat]][1] } vals <- current_stratifier_info$data[[variable]] - vals <- vals[!is.na(vals)] + keep <- !is.na(vals) + vals <- vals[keep] s <- summary(vals) new_row$median <- s[[3]] new_row$percentile25 <- s[[2]] @@ -68,6 +97,14 @@ get_descriptive_data <- function( new_row$min <- s[[1]] new_row$max <- s[[6]] new_row$n <- length(vals) + if (use_weights) { + w <- current_stratifier_info$data[[weight_var]][keep] + wq <- weighted_quantile(vals, w, c(0.25, 0.5, 0.75)) + new_row$wtd_percentile25 <- wq[1] + new_row$wtd_median <- wq[2] + new_row$wtd_percentile75 <- wq[3] + new_row$wtd_percent <- NA_real_ + } descriptive_data <<- rbind(descriptive_data, new_row) } ) @@ -99,6 +136,13 @@ get_descriptive_data <- function( ) new_row$n <- nrow(filtered) new_row$percent <- new_row$n / nrow(current_stratifier_info$data) + if (use_weights) { + new_row$wtd_percentile25 <- NA_real_ + new_row$wtd_median <- NA_real_ + new_row$wtd_percentile75 <- NA_real_ + new_row$wtd_percent <- sum(filtered[[weight_var]], na.rm = TRUE) / + sum(current_stratifier_info$data[[weight_var]], na.rm = TRUE) + } descriptive_data <<- rbind(descriptive_data, new_row) } ) @@ -135,6 +179,13 @@ get_descriptive_data <- function( ) new_row$n <- nrow(filtered) new_row$percent <- new_row$n / nrow(current_stratifier_info$data) + if (use_weights) { + new_row$wtd_percentile25 <- NA_real_ + new_row$wtd_median <- NA_real_ + new_row$wtd_percentile75 <- NA_real_ + new_row$wtd_percent <- sum(filtered[[weight_var]], na.rm = TRUE) / + sum(current_stratifier_info$data[[weight_var]], na.rm = TRUE) + } descriptive_data <<- rbind(descriptive_data, new_row) } ) diff --git a/R/imputation.R b/R/imputation.R index 8096a57..ab05189 100644 --- a/R/imputation.R +++ b/R/imputation.R @@ -1,106 +1,187 @@ # imputation.R # Multiple imputation via MICE for missing smoking history variables. # Pipeline target: analysis_data +# +# Specification: docs/protocol/appendix-imputation.qmd (protocol v0.3.0). +# Three design points distinguish this from the DemPoRT-V2 implementation: +# 1. An explicit `where` matrix restricts imputation to item non-response +# (NA(b)); structural missingness (NA(a) not-applicable, NA(c) not asked) +# is never imputed. haven::tagged_na() values are plain NA to mice(), +# so without the where matrix MICE would impute structural cells. +# 2. Survey design variables (cycle, weight) enter the imputation model as +# predictors, making the imputation design-consistent. +# 3. All m completed datasets are retained; Table 1b averages across them +# and the APC stage starts from imputation 1 (upgrade path: Rubin pooling). #' Impute missing values via MICE #' -#' Applies multiple imputation to predictor variables with random missingness -#' (NA(b) = don't know/refused). Structural missingness — NA(a) (not -#' applicable) and NA(c) (not asked this cycle) — is preserved and not imputed. -#' -#' Number of imputations and iterations are controlled by cfg$imputation_m and -#' cfg$imputation_maxit. Use m=1, maxit=1 in draft/dev for fast iteration. +#' Imputes NA(b) (don't know/refused) cells for variables carrying the +#' `imputation-predictor` role. Structural missingness — NA(a) (not +#' applicable) and NA(c) (not asked this cycle) — is preserved untouched: +#' only cells flagged in the `where` matrix are ever written back, so tagged +#' NAs survive by construction. #' #' @param cleaned_data Output of clean_study_data() #' @param variables_sheet Variables worksheet data frame -#' @param cfg Config object from config::get() -#' @return Data frame with NA(b) values imputed (first completed dataset) +#' @param cfg Config object from config::get() (uses imputation_m, imputation_maxit) +#' @return List: +#' \describe{ +#' \item{datasets}{List of m completed data frames (full columns; only +#' NA(b) cells differ from cleaned_data)} +#' \item{m}{Number of imputations} +#' \item{imputed_cells}{Data frame: variable, n_imputed — audit of which +#' cells were filled} +#' \item{logged_events}{mice loggedEvents data frame (NULL if none)} +#' } impute_data <- function(cleaned_data, variables_sheet, cfg) { - # Identify variables for the MICE predictor matrix - # "imputation-predictor" role covers all variables that should inform imputation impute_vars <- select_vars_by_role("imputation-predictor", variables_sheet) - impute_vars <- intersect(impute_vars, colnames(cleaned_data)) - - # Check which variables actually have NA(b) — only those need imputation - has_na_b <- vapply(impute_vars, function(var) { - x <- cleaned_data[[var]] - if (is.factor(x)) { - "NA(b)" %in% as.character(x) - } else { - any(haven::is_tagged_na(x, "b")) - } - }, logical(1)) + missing_from_data <- setdiff(impute_vars, colnames(cleaned_data)) + if (length(missing_from_data) > 0) { + stop( + "Imputation-predictor variables absent from data: ", + paste(missing_from_data, collapse = ", "), + " — check harmonization output before imputing." + ) + } - vars_to_impute <- impute_vars[has_na_b] + # Build the MICE modelling frame and the where matrix (TRUE = NA(b) cell) + prep <- prepare_for_mice(cleaned_data, impute_vars) + n_imputable <- colSums(prep$where) + imputed_cells <- data.frame( + variable = names(n_imputable), + n_imputed = unname(n_imputable), + stringsAsFactors = FALSE + ) message( - "Imputation: ", length(vars_to_impute), - " variables with NA(b) missingness:\n ", - paste(vars_to_impute, collapse = ", ") + "Imputation targets (NA(b) cells):\n", + paste(sprintf( + " %s: %d", imputed_cells$variable[imputed_cells$n_imputed > 0], + imputed_cells$n_imputed[imputed_cells$n_imputed > 0] + ), collapse = "\n") ) - if (length(vars_to_impute) == 0) { - message("No NA(b) values found — returning cleaned data unchanged.") - return(cleaned_data) + if (sum(prep$where) == 0) { + message("No NA(b) cells found — returning cleaned data unchanged (m = 1).") + return(list( + datasets = list(cleaned_data), m = 1L, + imputed_cells = imputed_cells, logged_events = NULL + )) } - # Prepare data for MICE: pass all impute_vars so auxiliary predictors (those - # without NA(b)) are included in the predictor matrix, improving imputation - # quality. Only vars_to_impute will actually have NAs for MICE to fill. - mice_input <- prepare_for_mice(cleaned_data, impute_vars) + m <- cfg$imputation_m %||% 5 + maxit <- cfg$imputation_maxit %||% 5 + message("Running MICE: m=", m, ", maxit=", maxit) - # Run MICE - message( - "Running MICE: m=", cfg$imputation_m, - ", maxit=", cfg$imputation_maxit - ) mice_result <- mice::mice( - mice_input, - m = cfg$imputation_m, - maxit = cfg$imputation_maxit, + prep$data, + m = m, + maxit = maxit, + where = prep$where, printFlag = FALSE ) - # Complete: use first imputation dataset - completed <- mice::complete(mice_result, action = 1) + # Surface silently dropped/altered variables (constant, collinear, etc.) + logged <- mice_result$loggedEvents + if (!is.null(logged) && nrow(logged) > 0) { + warning( + "MICE logged events (variables dropped or altered in the imputation model):\n", + paste(capture.output(print(logged)), collapse = "\n"), + call. = FALSE + ) + } - # Write imputed values back; all other columns are unchanged - result <- cleaned_data - for (var in vars_to_impute) { - result[[var]] <- completed[[var]] + # Convergence check: mice() does not error on pathological chains, so a + # quick guard on non-finite chain means catches degenerate fits. + if (any(!is.finite(mice_result$chainMean), na.rm = FALSE) && + any(is.nan(mice_result$chainMean))) { + warning("Non-finite MICE chain means detected — inspect convergence ", + "(mice::plot) before using these imputations.", call. = FALSE) } - message("Imputation complete.") - result + # Write back: for each imputation, copy ONLY where-matrix cells into the + # original data. Everything else — including tagged NA(a)/NA(c) and the + # original factor levels — is untouched by construction. + datasets <- lapply(seq_len(m), function(i) { + completed <- mice::complete(mice_result, action = i) + out <- cleaned_data + for (var in colnames(prep$where)) { + cells <- prep$where[, var] + if (!any(cells)) next + imputed_vals <- completed[[var]][cells] + # mice declines to impute variables it flags constant/collinear + # (method set to ""), returning plain NA for their where-TRUE cells. + # Writing those back would replace the NA(b) tag with untagged NA — + # silent corruption of the missing-data accounting. Fail loudly instead. + if (anyNA(imputed_vals)) { + stop( + "MICE left ", sum(is.na(imputed_vals)), " where-TRUE cell(s) of '", + var, "' unimputed (see logged events above: likely constant or ", + "collinear). These NA(b) cells would lose their tag — adjust the ", + "imputation model before proceeding.", call. = FALSE + ) + } + if (is.factor(out[[var]])) { + # Imputed values are levels of the reduced factor; assign as character + # into the original factor (levels unchanged; "NA(b)" cells resolved) + out[[var]][cells] <- as.character(imputed_vals) + } else { + out[[var]][cells] <- imputed_vals + } + } + out + }) + + message("Imputation complete: ", m, " datasets.") + list( + datasets = datasets, m = m, + imputed_cells = imputed_cells, logged_events = logged + ) } -#' Prepare variables for MICE by converting NA(b) to regular NA +#' Prepare the MICE modelling frame and where matrix #' -#' Only NA(b) (random missing) is converted to regular NA for MICE to impute. -#' NA(a) (not applicable) and NA(c) (not asked this cycle) are left as-is, -#' which prevents MICE from imputing structural missingness. +#' Builds a copy of `vars` in which every tagged NA is plain NA (mice cannot +#' handle haven tags or "NA(x)" factor levels), plus a logical `where` matrix +#' marking exactly the NA(b) cells — the only cells MICE may impute. +#' Structural NA(a)/NA(c) cells are plain NA with where = FALSE: they are +#' excluded row-wise from the conditional models that use those variables, +#' which matches the question universes (e.g., never-smokers do not inform +#' the initiation-age model). +#' +#' For factors, the "NA(a)"/"NA(b)"/"NA(c)" levels are dropped from the +#' modelling frame so polytomous regression cannot assign a structural-missing +#' label as an imputed value. #' #' @param data Data frame -#' @param vars Character vector of variable names to pass to MICE -#' @return Data frame subset with only `vars`, NA(b) converted to regular NA +#' @param vars Character vector of variable names to include in the model +#' @return List: `data` (modelling frame), `where` (logical matrix, same dim) prepare_for_mice <- function(data, vars) { out <- data[, vars, drop = FALSE] + where <- matrix( + FALSE, nrow = nrow(out), ncol = length(vars), + dimnames = list(NULL, vars) + ) + + na_levels <- c("NA(a)", "NA(b)", "NA(c)") for (var in vars) { x <- out[[var]] if (is.factor(x)) { - # Replace "NA(b)" level with regular NA; keep "NA(a)" and "NA(c)" as levels char_x <- as.character(x) - char_x[char_x == "NA(b)"] <- NA_character_ - out[[var]] <- factor(char_x, levels = setdiff(levels(x), "NA(b)")) + where[, var] <- !is.na(char_x) & char_x == "NA(b)" + char_x[char_x %in% na_levels] <- NA_character_ + out[[var]] <- factor(char_x, levels = setdiff(levels(x), na_levels)) } else if (is.numeric(x)) { - # Replace tagged NA(b) with regular NA; tagged NA(a) and NA(c) pass through - x[haven::is_tagged_na(x, "b")] <- NA_real_ + where[, var] <- haven::is_tagged_na(x, "b") + # strip tags: all tagged NAs become plain NA in the modelling frame; + # NA(a)/NA(c) cells have where = FALSE and are never imputed + x[is.na(x)] <- NA_real_ out[[var]] <- x } } - out + list(data = out, where = where) } diff --git a/_targets.R b/_targets.R index 95c21a2..1f5e41c 100644 --- a/_targets.R +++ b/_targets.R @@ -44,28 +44,36 @@ list( ), # Stage 4: Descriptive statistics — pre-imputation (Table 1a source data) + # Unweighted n + survey-weighted statistics in one table (protocol §3.4.1) tar_target(table_1a_data, - get_cshm_desc_data(cleaned_data, variables_sheet, variable_details_sheet) + get_cshm_desc_data(cleaned_data, variables_sheet, variable_details_sheet, + weight_var = survey_var(cfg, "weight")) ), # Stage 5: Multiple imputation - # Imputed analysis data — primary reproducibility artifact (stored in the _targets/ store) + # Returns list(datasets = m completed data frames, m, imputed_cells, + # logged_events) — primary reproducibility artifact (in the _targets/ store). + # where-matrix MICE: only NA(b) cells imputed (protocol Appendix D). tar_target(analysis_data, impute_data(cleaned_data, variables_sheet, cfg), format = "rds" ), # Stage 6: Descriptive statistics — post-imputation (Table 1b source data) + # Averaged across the m completed datasets tar_target(table_1b_data, - get_cshm_desc_data(analysis_data, variables_sheet, variable_details_sheet) + get_cshm_desc_data_mi(analysis_data, variables_sheet, variable_details_sheet, + weight_var = survey_var(cfg, "weight")) ), # Stage 7: Prepare APC datasets (numerator + denominator combined, by sex) # Single target; denominator construction is the expensive step. # Keeping as one target lets {targets} cache the full APC data independently # from Stage 8 model parameters (knots, constraints). + # Fitted on imputation 1 (protocol Appendix D documents the upgrade path + # to per-imputation fits pooled by Rubin's rules). tar_target(apc_data, - prepare_apc_data(analysis_data, cfg) + prepare_apc_data(analysis_data$datasets[[1]], cfg) ), # Stage 8: Fit APC models — four independent targets for parallel execution diff --git a/tests/testthat/test-descriptive-tables.R b/tests/testthat/test-descriptive-tables.R index 9d64625..01dc7d4 100644 --- a/tests/testthat/test-descriptive-tables.R +++ b/tests/testthat/test-descriptive-tables.R @@ -11,7 +11,7 @@ make_variables_sheet <- function() { section = c("Sociodemographics", "Sociodemographics", "Sociodemographics", "Health behaviour"), units = c("N/A", "N/A", "Years", "N/A"), - role = c("design", "model-stratifier", "predictor", "predictor"), + role = c("design", "model-stratifier", "predictor, table1", "predictor, table1"), stringsAsFactors = FALSE ) } @@ -64,9 +64,9 @@ test_that("select_vars_by_role returns correct variables", { vs <- make_variables_sheet() expect_equal(select_vars_by_role("predictor", vs), c("DHH_AGE", "SMKDSTY")) expect_equal(select_vars_by_role("model-stratifier", vs), "DHH_SEX") - expect_equal(select_vars_by_role("design", vs), c("SurveyCycle", "WTS_M")[ - c("SurveyCycle", "WTS_M") %in% select_vars_by_role("design", vs) - ]) + expect_equal(select_vars_by_role("design", vs), "SurveyCycle") + # comma-separated multi-role rows are matched on the individual role + expect_equal(select_vars_by_role("table1", vs), c("DHH_AGE", "SMKDSTY")) expect_length(select_vars_by_role("nonexistent", vs), 0) }) diff --git a/tests/testthat/test-imputation.R b/tests/testthat/test-imputation.R new file mode 100644 index 0000000..7793188 --- /dev/null +++ b/tests/testthat/test-imputation.R @@ -0,0 +1,131 @@ +# Tests for where-matrix MICE imputation (protocol v0.3.0, Appendix D) +# The contract under test: only NA(b) cells are imputed; tagged NA(a)/NA(c) +# survive the round trip untouched; factor levels are preserved. + +make_imputation_test_data <- function(n = 80, seed = 42) { + set.seed(seed) + age <- round(runif(n, 20, 80)) + weight <- round(runif(n, 50, 500)) + + # Continuous: years since quit — NA(a) for never-smokers, some NA(b) + years_quit <- round(runif(n, 0, 30)) + years_quit[1:20] <- haven::tagged_na("a") # never-smokers: not applicable + years_quit[21:25] <- haven::tagged_na("b") # don't know / refused + years_quit[26:27] <- haven::tagged_na("c") # not asked this cycle + + # Categorical: smoking status with structural and random missing levels + status <- sample(c("1", "4", "6"), n, replace = TRUE) + status[28:31] <- "NA(b)" + status[32:33] <- "NA(a)" + status <- factor(status, levels = c("1", "4", "6", "NA(a)", "NA(b)")) + + data.frame( + age = age, weight = weight, + years_quit = years_quit, status = status, + stringsAsFactors = FALSE + ) +} + +make_imputation_variables_sheet <- function() { + data.frame( + variable = c("age", "weight", "years_quit", "status"), + variableType = c("Continuous", "Continuous", "Continuous", "Categorical"), + role = c("imputation-predictor", "imputation-predictor", + "predictor, imputation-predictor", + "predictor, imputation-predictor"), + source = "both", + stringsAsFactors = FALSE + ) +} + +test_that("prepare_for_mice builds the where matrix on exactly the NA(b) cells", { + d <- make_imputation_test_data() + prep <- prepare_for_mice(d, c("age", "weight", "years_quit", "status")) + + expect_equal(sum(prep$where[, "years_quit"]), 5) # rows 21:25 + expect_true(all(which(prep$where[, "years_quit"]) == 21:25)) + expect_equal(sum(prep$where[, "status"]), 4) # rows 28:31 + expect_equal(sum(prep$where[, "age"]), 0) + expect_equal(sum(prep$where[, "weight"]), 0) + + # Modelling frame: no NA(x) levels remain; tagged NAs are plain NA + expect_false(any(c("NA(a)", "NA(b)", "NA(c)") %in% levels(prep$data$status))) + expect_false(any(haven::is_tagged_na(prep$data$years_quit))) + # Structural cells are NA in the frame (excluded from imputation by where = FALSE) + expect_true(all(is.na(prep$data$years_quit[1:20]))) +}) + +test_that("impute_data fills NA(b) and preserves tagged NA(a)/NA(c) untouched", { + d <- make_imputation_test_data() + vs <- make_imputation_variables_sheet() + cfg <- list(imputation_m = 2, imputation_maxit = 1) + + result <- suppressWarnings(impute_data(d, vs, cfg)) + + expect_equal(result$m, 2) + expect_length(result$datasets, 2) + + for (out in result$datasets) { + # NA(b) cells resolved to real values + expect_false(any(is.na(out$years_quit[21:25]))) + expect_false(any(as.character(out$status[28:31]) == "NA(b)", na.rm = TRUE)) + expect_false(any(is.na(out$status[28:31]))) + + # Structural missingness untouched — tags survive the round trip + expect_true(all(haven::is_tagged_na(out$years_quit[1:20], "a"))) + expect_true(all(haven::is_tagged_na(out$years_quit[26:27], "c"))) + expect_true(all(as.character(out$status[32:33]) == "NA(a)")) + + # Factor levels preserved (including the structural-missing levels) + expect_equal(levels(out$status), levels(d$status)) + + # Imputed categorical values are never structural-missing labels + expect_true(all(as.character(out$status[28:31]) %in% c("1", "4", "6"))) + + # Complete columns are byte-identical + expect_identical(out$age, d$age) + expect_identical(out$weight, d$weight) + + # Observed values are unchanged + expect_identical(out$years_quit[40:80], d$years_quit[40:80]) + } + + # Audit output names the imputed cells + expect_equal( + result$imputed_cells$n_imputed[result$imputed_cells$variable == "years_quit"], 5 + ) +}) + +test_that("impute_data with no NA(b) returns the data unchanged", { + d <- make_imputation_test_data() + d$years_quit[21:25] <- 10 # remove the NA(b) cells + d$status[28:31] <- "1" + vs <- make_imputation_variables_sheet() + cfg <- list(imputation_m = 2, imputation_maxit = 1) + + result <- impute_data(d, vs, cfg) + expect_equal(result$m, 1L) + expect_identical(result$datasets[[1]], d) +}) + +test_that("impute_data stops when an imputation-predictor variable is absent", { + d <- make_imputation_test_data() + d$status <- NULL + vs <- make_imputation_variables_sheet() + cfg <- list(imputation_m = 1, imputation_maxit = 1) + expect_error(impute_data(d, vs, cfg), "absent from data") +}) + +test_that("weighted_quantile interpolates and matches type-7 median under equal weights", { + x <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) + w <- rep(1, 10) + # Midpoint-ECDF interpolation: equal-weight median equals the type-7 median + expect_equal(weighted_quantile(x, w, 0.5), unname(quantile(x, 0.5))) + # Upweighting the top value pulls the median toward it + w2 <- c(rep(1, 9), 100) + expect_gt(weighted_quantile(x, w2, 0.5), 9.5) + # Boundary probabilities clamp to the extremes + expect_equal(weighted_quantile(x, w, c(0, 1)), c(1, 10)) + expect_equal(weighted_quantile(numeric(0), numeric(0), 0.5), NA_real_) + expect_equal(weighted_quantile(7, 3, c(0.25, 0.75)), c(7, 7)) +}) diff --git a/worksheets/cshm-variables.csv b/worksheets/cshm-variables.csv index 86e3605..592732b 100644 --- a/worksheets/cshm-variables.csv +++ b/worksheets/cshm-variables.csv @@ -1,19 +1,19 @@ "variable","label","labelLong","variableType","databaseStart","variableStart","subject","section","units","notes","description","purpose","version","lastUpdated","reviewNotes","status","versionNotes","role","source" -"SurveyCycle","Survey cycle","CCHS survey cycle","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","DerivedVar::[data_name]","demographics","Sociodemographics","N/A",NA,"Derived from dataset name; identifies CCHS cycle for each respondent","Identifies the CCHS cycle for each respondent. Used to derive survey year for the period component of the APC model and to label cycle-specific results.","0.1.0","2025-06-01","","",NA,"design, table1-stratifier","both" -"DHHGAGE_cont","Age (grouped continuous, years)","Age - continuous","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","[DHH_AGE]","demographics","Sociodemographics","Years",NA,"Continuous age; preferred over DHHGAGE_A for APC model where available","Continuous age in years (midpoint-estimated in PUMF). The primary age input to the APC model. Cohort is derived as survey_year - age.","0.1.0","2025-06-01","","",NA,"predictor, table1, apc-denominator","pumf" -"DHH_SEX","Sex","Sex","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","cchs2001_p::DHHA_SEX, cchs2003_p::DHHA_SEX, cchs2005_p::DHHA_SEX, [DHH_SEX]","demographics","Sociodemographics","N/A",NA,"Sex (1=Male 2=Female). APC models run separately by sex","Sex. APC initiation and cessation models are fit separately for men and women because smoking trends differ substantially by sex. Also used to stratify Table 1.","0.1.0","2025-06-01","","",NA,"model-stratifier, table1, apc-denominator, table1-stratifier","pumf" -"GEOGPRV","Province of residence","Province of residence","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","[GEOGPRV]","demographics","Sociodemographics","N/A",NA,"Province/territory of residence. Used for provincial APC estimates","Province of residence. Used for provincial APC stratification (Stage 8 subgroup). Territories are pooled due to small sample sizes.","0.1.0","2025-06-01","","",NA,"predictor, table1","pumf" -"WTS_M","Master survey weight","Survey sampling weight","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","cchs2001_p::WTSAM, cchs2003_p::WTSC_M, cchs2005_p::WTSE_M, [WTS_M]","demographics","Sociodemographics","N/A",NA,"CCHS sampling weight. Required for all prevalence estimates and weighted APC models","Survey sampling weight. Applied in APC logistic regression as a case weight to produce nationally representative estimates.","0.1.0","2025-06-01","","",NA,"design, apc-denominator","pumf" -"SDCFIMM","Immigrant status (2007-2014 only)","Immigrant status (D)","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p","[SDCFIMM]","demographics","Sociodemographics","N/A",NA,"Immigrant status derived (1=non-immigrant 2=immigrant 3=non-permanent resident). Used in MPoRT mortality adjustment","Immigrant status. Used in the MPoRT mortality correction to adjust for survival bias. Only available 2007-2014.","0.1.0","2025-06-01","","",NA,"predictor","pumf" -"SDCGCGT","Cultural/racial origin (2007-2014 only)","Cultural or racial origin (D/G)","Categorical","cchs2001_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p","cchs2001_p::SDCA_COB, [SDCGCGT]","demographics","Sociodemographics","N/A",NA,"Cultural/racial origin. Not in all cycles (NA(c) where absent). Used for subgroup analysis; not a primary APC model variable","Cultural/racial origin. Retained for potential subgroup analyses. Only available in select cycles; will be NA(c) elsewhere.","0.1.0","2025-06-01","Not available all cycles","",NA,"predictor","pumf" -"EDUDR03","Education (3-cat)","Highest education level - 3 categories","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p","[EDUDR03]","demographics","Sociodemographics","N/A",NA,"Highest education (3-category: less than high school / high school graduate / post-secondary). Used for subgroup analysis. cchsflow harmonized from EDUADR04/EDUCDR04/EDUEDR04/EHG2DVR3.","Education (3-category). Included for descriptive purposes and potential subgroup analysis. Not available in 2019-20 or 2022 PUMF.","0.1.0","2025-06-01","Not available 2019-20 or 2022","",NA,"predictor, table1","pumf" +"SurveyCycle","Survey cycle","CCHS survey cycle","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","DerivedVar::[data_name]","demographics","Sociodemographics","N/A",NA,"Derived from dataset name; identifies CCHS cycle for each respondent","Identifies the CCHS cycle for each respondent. Used to derive survey year for the period component of the APC model and to label cycle-specific results.","0.1.0","2026-06-11","","",NA,"design, table1-stratifier, imputation-predictor","both" +"DHHGAGE_cont","Age (grouped continuous, years)","Age - continuous","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","[DHH_AGE]","demographics","Sociodemographics","Years",NA,"Continuous age; preferred over DHHGAGE_A for APC model where available","Continuous age in years (midpoint-estimated in PUMF). The primary age input to the APC model. Cohort is derived as survey_year - age.","0.1.0","2026-06-11","","",NA,"predictor, table1, apc-denominator, imputation-predictor","pumf" +"DHH_SEX","Sex","Sex","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","cchs2001_p::DHHA_SEX, cchs2003_p::DHHA_SEX, cchs2005_p::DHHA_SEX, [DHH_SEX]","demographics","Sociodemographics","N/A",NA,"Sex (1=Male 2=Female). APC models run separately by sex","Sex. APC initiation and cessation models are fit separately for men and women because smoking trends differ substantially by sex. Also used to stratify Table 1.","0.1.0","2026-06-11","","",NA,"model-stratifier, table1, apc-denominator, table1-stratifier, imputation-predictor","pumf" +"GEOGPRV","Province of residence","Province of residence","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","[GEOGPRV]","demographics","Sociodemographics","N/A",NA,"Province/territory of residence. Used for provincial APC estimates","Province of residence. Used for provincial APC stratification (Stage 8 subgroup). Territories are pooled due to small sample sizes.","0.1.0","2026-06-11","","",NA,"predictor, table1, imputation-predictor","pumf" +"WTS_M","Master survey weight","Survey sampling weight","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p","cchs2001_p::WTSAM, cchs2003_p::WTSC_M, cchs2005_p::WTSE_M, [WTS_M]","demographics","Sociodemographics","N/A",NA,"CCHS sampling weight. Required for all prevalence estimates and weighted APC models","Survey sampling weight. Applied in APC logistic regression as a case weight to produce nationally representative estimates.","0.1.0","2026-06-11","","",NA,"design, apc-denominator, imputation-predictor","pumf" +"SDCFIMM","Immigrant status (2007-2014 only)","Immigrant status (D)","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p","[SDCFIMM]","demographics","Sociodemographics","N/A",NA,"Immigrant status derived (1=non-immigrant 2=immigrant 3=non-permanent resident). Used in MPoRT mortality adjustment","Immigrant status. Used in the MPoRT mortality correction to adjust for survival bias. Only available 2007-2014.","0.1.0","2026-06-11","","",NA,"predictor, table1","pumf" +"SDCGCGT","Cultural/racial origin (2007-2014 only)","Cultural or racial origin (D/G)","Categorical","cchs2001_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p","cchs2001_p::SDCA_COB, [SDCGCGT]","demographics","Sociodemographics","N/A",NA,"Cultural/racial origin. Not in all cycles (NA(c) where absent). Used for subgroup analysis; not a primary APC model variable","Cultural/racial origin. Retained for potential subgroup analyses. Only available in select cycles; will be NA(c) elsewhere.","0.1.0","2026-06-11","Not available all cycles","",NA,"predictor, table1","pumf" +"EDUDR03","Education (3-cat)","Highest education level - 3 categories","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2011_2012_p, cchs2013_2014_p, cchs2015_2016_p, cchs2017_2018_p","[EDUDR03]","demographics","Sociodemographics","N/A",NA,"Highest education (3-category: less than high school / high school graduate / post-secondary). Used for subgroup analysis. cchsflow harmonized from EDUADR04/EDUCDR04/EDUEDR04/EHG2DVR3.","Education (3-category). Included for descriptive purposes and potential subgroup analysis. Not available in 2019-20 or 2022 PUMF.","0.1.0","2026-06-11","Not available 2019-20 or 2022","",NA,"predictor, table1, imputation-predictor","pumf" "SMK_01A","Smoked 100+ cigs","Ever smoked 100 or more cigarettes in lifetime","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p, cchs2023_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","cchs2001_p::SMKA_01A, cchs2003_p::SMKC_01A, cchs2005_p::SMKE_01A, cchs2015_2016_p::SMK_020, cchs2017_2018_p::SMK_020, cchs2019_2020_p::SMK_020, cchs2022_p::CSS_15, cchs2023_p::CSS_15, cchs2001_m::SMKA_01A, cchs2003_m::SMKC_01A, cchs2005_m::SMKE_01A, cchs2015_2016_m::SMK_020, cchs2017_2018_m::SMK_020, cchs2019_2020_m::SMK_020, cchs2021_m::SMK_020, cchs2022_m::CSS_15, cchs2023_m::CSS_15, [SMK_01A]","smoking","Health behaviour","N/A",NA,"Gate question: ever smoked >=100 cigarettes lifetime. Combined with SMK_01B and SMK_202 for never/current/former classification","Intermediate: input to SMKDSTY_A derivation. Gate variable: ever smoked 100+ cigarettes. Combined with SMK_01B and SMK_202 to classify respondents as never/current/former smoker for the APC numerator.","0.1.0","2026-06-09","","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","intermediate","both" "SMK_01B","Smoked 1 whole cig","Ever smoked a whole cigarette","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p, cchs2023_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","cchs2001_p::SMKA_01B, cchs2003_p::SMKC_01B, cchs2005_p::SMKE_01B, cchs2015_2016_p::SMK_025, cchs2017_2018_p::SMK_025, cchs2019_2020_p::SMK_025, cchs2022_p::CSS_05, cchs2023_p::CSS_05, cchs2001_m::SMKA_01B, cchs2003_m::SMKC_01B, cchs2005_m::SMKE_01B, cchs2015_2016_m::SMK_025, cchs2017_2018_m::SMK_025, cchs2019_2020_m::SMK_025, cchs2021_m::SMK_025, cchs2022_m::CSS_05, cchs2023_m::CSS_05, [SMK_01B]","smoking","Health behaviour","N/A",NA,"Second gate for never-smoker definition: never smoked a whole cigarette AND <100 lifetime cigarettes = never smoker","Intermediate: input to SMKDSTY_A derivation. Gate variable: ever smoked a whole cigarette. Used with SMK_01A to define never-smoker status.","0.1.0","2026-06-09","","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","intermediate","both" "SMK_202","Smoking type","Type of smoker presently (daily/occasional/not at all)","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m","cchs2001_p::SMKA_202, cchs2003_p::SMKC_202, cchs2005_p::SMKE_202, cchs2015_2016_p::SMK_005, cchs2017_2018_p::SMK_005, cchs2019_2020_p::SMK_005, cchs2001_m::SMKA_202, cchs2003_m::SMKC_202, cchs2005_m::SMKE_202, cchs2015_2016_m::SMK_005, cchs2017_2018_m::SMK_005, cchs2019_2020_m::SMK_005, cchs2021_m::SMK_005, [SMK_202]","smoking","Health behaviour","N/A",NA,"Current smoking frequency. Combined with SMK_01A/SMK_01B to derive 3-category status: never / current / former","Intermediate: input to SMKDSTY_A derivation. Current smoking frequency (daily/occasional/not at all). Combined with SMK_01A/SMK_01B to derive 3-category smoking status.","0.1.0","2026-06-09","","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","intermediate","both" -"SMKDSTY_original","Smoking status (6-cat)","Type of smoker derived - 6-category (cchsflow v3, original StatCan scheme)","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m","cchs2001_p::SMKADSTY, cchs2003_p::SMKCDSTY, cchs2005_p::SMKEDSTY, cchs2007_2008_p::SMKDSTY, cchs2009_2010_p::SMKDSTY, cchs2010_p::SMKDSTY, cchs2011_2012_p::SMKDSTY, cchs2012_p::SMKDSTY, cchs2013_2014_p::SMKDSTY, cchs2001_m::SMKADSTY, cchs2003_m::SMKCDSTY, cchs2005_m::SMKEDSTY, cchs2007_2008_m::SMKDSTY, cchs2009_2010_m::SMKDSTY, cchs2009_m::SMKDSTY, cchs2010_m::SMKDSTY, cchs2011_2012_m::SMKDSTY, cchs2012_m::SMKDSTY, cchs2013_2014_m::SMKDSTY, cchs2014_m::SMKDSTY, DerivedVar::[SMK_202, SMK_05D, SMK_01A]","smoking","Health behaviour","N/A",NA,"cchsflow v3 harmonized 6-cat smoking status: 1=daily, 2=occ(fmr daily), 3=always occasional, 4=former daily, 5=former occasional, 6=never. Consistent categories across all cycles.","Primary smoking classification for APC numerator construction and Table 1. cchsflow v3: 2001-2014 pass-through from SMKDSTY; 2015-2021 derived from SMK_202, SMK_05D, SMK_01A. Not supported PUMF 2022/2023.","0.1.0","2026-06-09","Renamed from SMKDSTY_A per CEP-002 year-based naming","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1, apc-numerator","both" -"age_first_cigarette","Age 1st cig (unified)*","Age smoked first whole cigarette - unified (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","DerivedVar::[SMKG01C_cont, SMK_01C]","smoking","Health behaviour","Years","Unified variable (cchsflow v3). Universe: ever smoked 100+ cigarettes. Priority: SMK_01C (Master exact) > SMKG01C_cont (PUMF midpoint). PUMF 2001-2021; Master 2001-2023.","Unified variable (cchsflow v3 PR #163). Master: exact age (SMK_01C); PUMF: midpoint-estimated (SMKG01C_cont). Primary initiation age input","Age at first whole cigarette (unified cchsflow v3 variable). Primary input for the initiation APC numerator. Routes to exact values (Master) or midpoint estimates (PUMF) automatically.","0.1.0","2026-06-09","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1, apc-numerator","both" -"age_start_smoking","Age daily (unified)*","Age started smoking daily - unified (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","DerivedVar::[SMKG040_cont, SMK_040]","smoking","Health behaviour","Years","Raw SMKG040 absent from 2019-20 PUMF (DDI-confirmed): age_start_smoking unavailable for cchs2019_2020_p and cchs2022_p; Master covers 2001-2023. See cchsflow#185.","Unified variable (cchsflow v3 PR #163). Master: exact age (SMK_040); PUMF: midpoint-estimated (SMKG040_cont). Primary daily initiation age","Age started smoking daily (unified cchsflow v3 variable). Used in the initiation APC model as an alternative or supplementary age measure.","0.1.0","2026-06-10","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1, apc-numerator","both" -"time_quit_smoking_daily","Yrs quit daily (unified)*","Years since stopped smoking daily - unified (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2023_m","DerivedVar::[SMKDSTY_cat5, SMK_09A_cont, SMK_09C]","smoking","Health behaviour","Years","Unified variable (cchsflow v3). Universe: former daily smokers. DerivedVar::[SMKDSTY_cat5, SMK_09A_cont, SMK_09C]: Master priority via SMK_09C exact years; PUMF fallback via SMK_09A_cont midpoint. Not supported 2022 or PUMF 2023.","Unified variable (cchsflow v3). PUMF: midpoint from SMK_09A_cont; Master: exact from SMK_09C. Former daily smokers only.","Primary input for the cessation APC numerator. Covers former daily smokers.","0.1.0","2026-06-09","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1, apc-numerator","both" +"SMKDSTY_original","Smoking status (6-cat)","Type of smoker derived - 6-category (cchsflow v3, original StatCan scheme)","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m","cchs2001_p::SMKADSTY, cchs2003_p::SMKCDSTY, cchs2005_p::SMKEDSTY, cchs2007_2008_p::SMKDSTY, cchs2009_2010_p::SMKDSTY, cchs2010_p::SMKDSTY, cchs2011_2012_p::SMKDSTY, cchs2012_p::SMKDSTY, cchs2013_2014_p::SMKDSTY, cchs2001_m::SMKADSTY, cchs2003_m::SMKCDSTY, cchs2005_m::SMKEDSTY, cchs2007_2008_m::SMKDSTY, cchs2009_2010_m::SMKDSTY, cchs2009_m::SMKDSTY, cchs2010_m::SMKDSTY, cchs2011_2012_m::SMKDSTY, cchs2012_m::SMKDSTY, cchs2013_2014_m::SMKDSTY, cchs2014_m::SMKDSTY, DerivedVar::[SMK_202, SMK_05D, SMK_01A]","smoking","Health behaviour","N/A",NA,"cchsflow v3 harmonized 6-cat smoking status: 1=daily, 2=occ(fmr daily), 3=always occasional, 4=former daily, 5=former occasional, 6=never. Consistent categories across all cycles.","Primary smoking classification for APC numerator construction and Table 1. cchsflow v3: 2001-2014 pass-through from SMKDSTY; 2015-2021 derived from SMK_202, SMK_05D, SMK_01A. Not supported PUMF 2022/2023.","0.1.0","2026-06-11","Renamed from SMKDSTY_A per CEP-002 year-based naming","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1, apc-numerator, imputation-predictor","both" +"age_first_cigarette","Age 1st cig (unified)*","Age smoked first whole cigarette - unified (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","DerivedVar::[SMKG01C_cont, SMK_01C]","smoking","Health behaviour","Years","Unified variable (cchsflow v3). Universe: ever smoked 100+ cigarettes. Priority: SMK_01C (Master exact) > SMKG01C_cont (PUMF midpoint). PUMF 2001-2021; Master 2001-2023.","Unified variable (cchsflow v3 PR #163). Master: exact age (SMK_01C); PUMF: midpoint-estimated (SMKG01C_cont). Primary initiation age input","Age at first whole cigarette (unified cchsflow v3 variable). Primary input for the initiation APC numerator. Routes to exact values (Master) or midpoint estimates (PUMF) automatically.","0.1.0","2026-06-11","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1, apc-numerator, imputation-predictor","both" +"age_start_smoking","Age daily (unified)*","Age started smoking daily - unified (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","DerivedVar::[SMKG040_cont, SMK_040]","smoking","Health behaviour","Years","Raw SMKG040 absent from 2019-20 PUMF (DDI-confirmed): age_start_smoking unavailable for cchs2019_2020_p and cchs2022_p; Master covers 2001-2023. See cchsflow#185.","Unified variable (cchsflow v3 PR #163). Master: exact age (SMK_040); PUMF: midpoint-estimated (SMKG040_cont). Primary daily initiation age","Age started smoking daily (unified cchsflow v3 variable). Used in the initiation APC model as an alternative or supplementary age measure.","0.1.0","2026-06-11","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1, apc-numerator, imputation-predictor","both" +"time_quit_smoking_daily","Yrs quit daily (unified)*","Years since stopped smoking daily - unified (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2023_m","DerivedVar::[SMKDSTY_cat5, SMK_09A_cont, SMK_09C]","smoking","Health behaviour","Years","Unified variable (cchsflow v3). Universe: former daily smokers. DerivedVar::[SMKDSTY_cat5, SMK_09A_cont, SMK_09C]: Master priority via SMK_09C exact years; PUMF fallback via SMK_09A_cont midpoint. Not supported 2022 or PUMF 2023.","Unified variable (cchsflow v3). PUMF: midpoint from SMK_09A_cont; Master: exact from SMK_09C. Former daily smokers only.","Primary input for the cessation APC numerator. Covers former daily smokers.","0.1.0","2026-06-11","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1, apc-numerator, imputation-predictor","both" "SMK_09A_cont","Yrs quit daily (PUMF)","Years since stopped smoking daily - former daily (PUMF continuous)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2023_m","cchs2001_p::SMKA_09A, cchs2003_p::SMKC_09A, cchs2005_p::SMKE_09A, cchs2015_2016_p::SMK_080, cchs2017_2018_p::SMK_080, cchs2019_2020_p::SMK_080, cchs2001_m::SMKA_09A, cchs2003_m::SMKC_09A, cchs2005_m::SMKE_09A, cchs2015_2016_m::SMK_080, cchs2017_2018_m::SMK_080, cchs2019_2020_m::SMK_080, cchs2021_m::SMK_080, cchs2023_m::SPU_25, [SMK_09A]","smoking","Health behaviour","Years","Midpoint-imputed years since quit, former daily smokers. Feeder for time_quit_smoking and time_quit_smoking_daily (cchsflow v3). Not available 2022 or PUMF 2023 (SPU_25 is Master-only).","PUMF-derived continuous years since quit. Superseded by time_quit_smoking once cchsflow v3 merges. Keep for pre-v3 fallback","Intermediate: PUMF/Master midpoint feeder for the unified cessation variables.","0.1.0","2026-06-09","","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","intermediate","both" "SMK_06A_cont","Yrs quit occ (PUMF)","Years since stopped smoking - former occasional smokers (PUMF continuous)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2023_m","cchs2001_p::SMKA_06A, cchs2003_p::SMKC_06A, cchs2005_p::SMKE_06A, cchs2015_2016_p::SMK_060, cchs2017_2018_p::SMK_060, cchs2019_2020_p::SMK_060, cchs2001_m::SMKA_06A, cchs2003_m::SMKC_06A, cchs2005_m::SMKE_06A, cchs2015_2016_m::SMK_060, cchs2017_2018_m::SMK_060, cchs2019_2020_m::SMK_060, cchs2021_m::SMK_060, cchs2023_m::SPU_10, [SMK_06A]","smoking","Health behaviour","Years","Midpoint-imputed years since quit, former occasional smokers. Feeder for time_quit_smoking (cchsflow v3 falls back to it when SMK_09A_cont is not applicable). Not available 2022 or PUMF 2023.","PUMF-derived years since quit for former occasional smokers. Not covered by time_quit_smoking (daily only)","Intermediate: occasional-smoker feeder for time_quit_smoking (all former smokers).","0.1.0","2026-06-09","","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","intermediate","both" "SMKDGSTP_cont","Yrs since quit (all)","Years since quit smoking completely - all former smokers (continuous)","Continuous","cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","cchs2003_p::SMKCGSTP, cchs2005_p::SMKEGSTP, cchs2007_2008_p::SMKGSTP, cchs2009_2010_p::SMKGSTP, cchs2010_p::SMKGSTP, cchs2011_2012_p::SMKGSTP, cchs2012_p::SMKGSTP, cchs2013_2014_p::SMKGSTP, cchs2003_m::SMKCDSTP, cchs2005_m::SMKEDSTP, cchs2007_2008_m::SMKDSTP, cchs2009_2010_m::SMKDSTP, cchs2009_m::SMKDSTP, cchs2010_m::SMKDSTP, cchs2011_2012_m::SMKDSTP, cchs2012_m::SMKDSTP, cchs2013_2014_m::SMKDSTP, cchs2014_m::SMKDSTP, cchs2015_2016_m::SMKDVSTP, cchs2017_2018_m::SMKDVSTP, cchs2019_2020_m::SMKDVSTP, cchs2021_m::SMKDVSTP, cchs2022_m::SMKDVSTP, cchs2023_m::SMKDVSTP, [SMKDGSTP]","smoking","Health behaviour","Years",NA,"StatCan derived continuous years since quit (all former smokers). Available 2009+. Preferred where available; cross-validate with SMK_09A_cont","StatsCan derived years since quit (all former smokers). Available from 2007 onward. Cross-validates SMK_09A_cont estimates.","0.1.0","2026-06-09","","","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","intermediate","both" @@ -38,5 +38,36 @@ "SMK_05B","Cigs/day (occ)","Number of cigarettes smoked daily - occasional smokers","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p, cchs2023_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","cchs2001_p::SMKA_05B, cchs2003_p::SMKC_05B, cchs2005_p::SMKE_05B, cchs2015_2016_p::SMK_050, cchs2017_2018_p::SMK_050, cchs2019_2020_p::SMK_050, cchs2022_p::CSS_30, cchs2023_p::CSS_30, cchs2001_m::SMKA_05B, cchs2003_m::SMKC_05B, cchs2005_m::SMKE_05B, cchs2015_2016_m::SMK_050, cchs2017_2018_m::SMK_050, cchs2019_2020_m::SMK_050, cchs2021_m::SMK_050, cchs2022_m::CSS_30, cchs2023_m::CSS_30, [SMK_05B]","smoking","Health behaviour","cigarettes",NA,"Number of cigarettes smoked daily - occasional smokers","Intermediate: cigarettes per day on days smoked (occasional smokers); occasional-period feeder for pack_years_der.","0.1.0","2026-06-10","Added for cchsflow v3 derivation chain; Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","active",NA,"intermediate","both" "SMK_05C","Days smoked/month","Days smoked at least 1 cigarette in past month","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2022_p, cchs2023_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","cchs2001_p::SMKA_05C, cchs2003_p::SMKC_05C, cchs2005_p::SMKE_05C, cchs2015_2016_p::SMK_055, cchs2017_2018_p::SMK_055, cchs2019_2020_p::SMK_055, cchs2022_p::CSS_35, cchs2023_p::CSS_35, cchs2001_m::SMKA_05C, cchs2003_m::SMKC_05C, cchs2005_m::SMKE_05C, cchs2015_2016_m::SMK_055, cchs2017_2018_m::SMK_055, cchs2019_2020_m::SMK_055, cchs2021_m::SMK_055, cchs2022_m::CSS_35, cchs2023_m::CSS_35, [SMK_05C]","smoking","Health behaviour","days",NA,"Days smoked at least 1 cigarette in past month","Intermediate: days smoked per month (occasional smokers); occasional-period feeder for pack_years_der.","0.1.0","2026-06-10","Added for cchsflow v3 derivation chain; Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","active",NA,"intermediate","both" "DHH_AGE","Age","Age","Continuous","cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m, cchs2022_m, cchs2023_m","cchs2001_m::DHHA_AGE, cchs2003_m::DHHC_AGE, cchs2005_m::DHHE_AGE, cchs2022_m::AWCAGE, cchs2023_m::AWCAGE, [DHH_AGE]","demographics","Sociodemographics","Years",NA,"Age","Intermediate: transitive feeder for pack_years_der (Master exact age) in the cchsflow v3 derivation chain.","0.1.0","2026-06-09","Added for cchsflow v3 derivation chain; Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","active",NA,"intermediate","master" -"cigs_per_day","Cigs/day (unified)*","Cigarettes per day - unified daily + former daily (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m","DerivedVar::[SMK_204, SMK_208, SMKDSTY_original]","smoking","Health behaviour","Cigarettes/day","Unified cigs/day (cchsflow v3). DerivedVar::[SMK_204, SMK_208, SMKDSTY_original]. Universe: ever-daily smokers. Not supported PUMF 2022/2023 (SMK_208 is Master-only via SPU in those cycles).","Unified cigs/day (cchsflow v3). Combines SMK_204 (current daily) and SMK_208 (former daily) automatically.","Smoking intensity for descriptive tables and future dose-response models.","0.1.0","2026-06-09","Replaces separate SMK_204/SMK_208","active","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac) - removed cchs2022_p (not supported in v3 final)","predictor, table1","both" +"cigs_per_day","Cigs/day (unified)*","Cigarettes per day - unified daily + former daily (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m","DerivedVar::[SMK_204, SMK_208, SMKDSTY_original]","smoking","Health behaviour","Cigarettes/day","Unified cigs/day (cchsflow v3). DerivedVar::[SMK_204, SMK_208, SMKDSTY_original]. Universe: ever-daily smokers. Not supported PUMF 2022/2023 (SMK_208 is Master-only via SPU in those cycles).","Unified cigs/day (cchsflow v3). Combines SMK_204 (current daily) and SMK_208 (former daily) automatically.","Smoking intensity for descriptive tables and future dose-response models.","0.1.0","2026-06-11","Replaces separate SMK_204/SMK_208","active","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac) - removed cchs2022_p (not supported in v3 final)","predictor, table1, imputation-predictor","both" "pack_years_der","Pack-years (unified)*","Cumulative pack-years - derived (cchsflow v3)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m, cchs2021_m","DerivedVar::[SMKDSTY_original, DHHGAGE_cont, DHH_AGE, age_start_smoking, cigs_per_day, time_quit_smoking, SMK_204, SMK_208, age_first_cigarette, smoked_100_lifetime]","smoking","Health behaviour","Pack-years","Cumulative pack-years (cchsflow v3). PUMF feeders: SMKDSTY_original, DHHGAGE_cont, age_start_smoking, cigs_per_day, time_quit_smoking, SMK_204, SMK_208, age_first_cigarette, smoked_100_lifetime. Not supported PUMF 2022/2023.","Cumulative pack-years (cchsflow v3). Derived from cigs_per_day and years smoked.","Cumulative smoking exposure measure for descriptive tables.","0.1.0","2026-06-09","PUMF gap: not available 2022","active","Synced to cchsflow v3 (smoking merged 2026-04-29, commit bd0df3ac)","predictor, table1","both" +"DHHGMS","Marital status","Marital status - (G)","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p","cchs2001_p::DHHAGMS, cchs2003_p::DHHCGMS, cchs2005_p::DHHEGMS, [DHHGMS]","Marital Status","Sociodemographics","N/A",NA,"Marital status - (G)","Marital status: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","pumf" +"ALCDTTM","Drinker type (last 12 months)","Type of drinker (12 months)","Categorical","cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2015_2016_p::ALCDVTTM, cchs2017_2018_p::ALCDVTTM, cchs2019_2020_p::ALCDVTTM, cchs2015_2016_m::ALCDVTTM, cchs2017_2018_m::ALCDVTTM, cchs2019_2020_m::ALCDVTTM, [ALCDTTM]","Alcohol","Health behaviour","N/A",NA,"Type of drinker (12 months)","Drinker type (12 months): auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"ALWDWKY","Drinks last week","Weekly consumption of alcohol","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::ALCADWKY, cchs2003_p::ALCCDWKY, cchs2005_p::ALCEDWKY, cchs2015_2016_p::ALWDVWKY, cchs2017_2018_p::ALWDVWKY, cchs2019_2020_p::ALWDVWKY, cchs2001_m::ALCADWKY, cchs2003_m::ALCCDWKY, cchs2005_m::ALCEDWKY, cchs2015_2016_m::ALWDVWKY, cchs2017_2018_m::ALWDVWKY, cchs2019_2020_m::ALWDVWKY, [ALWDWKY]","Alcohol","Health behaviour","drinks/week",NA,"Weekly consumption of alcohol","Drinks last week: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"HWTGBMI_der","Derived BMI","Derived Body Mass Index","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p","DerivedVar::[HWTGHTM, HWTGWTK]","BMI","Health status","kg/m2",NA,"Derived Body Mass Index","BMI (derived): auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","pumf" +"GEN_01","Self-perceived health","Self-perceived health","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::GENA_01, cchs2003_p::GENC_01, cchs2005_p::GENE_01, cchs2015_2016_p::GEN_005, cchs2017_2018_p::GEN_005, cchs2019_2020_p::GEN_005, cchs2001_m::GENA_01, cchs2003_m::GENC_01, cchs2005_m::GENE_01, cchs2015_2016_m::GEN_005, cchs2017_2018_m::GEN_005, cchs2019_2020_m::GEN_005, [GEN_01]","Self-perceived health","Health status","N/A",NA,"Self-perceived health","Self-rated general health: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"GEN_02B","Self-perceived mental health","Self-perceived mental health","Categorical","cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2003_p::GENC_02B, cchs2005_p::GENE_02B, cchs2015_2016_p::GEN_015, cchs2017_2018_p::GEN_015, cchs2019_2020_p::GEN_015, cchs2003_m::GENC_02B, cchs2005_m::GENE_02B, cchs2015_2016_m::GEN_015, cchs2017_2018_m::GEN_015, cchs2019_2020_m::GEN_015, [GEN_02B]","Mental health","Health status","N/A",NA,"Self-perceived mental health","Self-rated mental health: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"GEN_07","Self-perceived life stress","Self-perceived life stress","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::GENA_07, cchs2003_p::GENC_07, cchs2005_p::GENE_07, cchs2015_2016_p::GEN_020, cchs2017_2018_p::GEN_020, cchs2019_2020_p::GEN_020, cchs2001_m::GENA_07, cchs2003_m::GENC_07, cchs2005_m::GENE_07, cchs2015_2016_m::GEN_020, cchs2017_2018_m::GEN_020, cchs2019_2020_m::GEN_020, [GEN_07]","Mental health","Health status","N/A",NA,"Self-perceived life stress","Self-perceived life stress: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"GEN_10","Sense of belonging","Sense of belonging in the community","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::GENA_10, cchs2003_p::GENC_10, cchs2005_p::GENE_10, cchs2015_2016_p::GEN_030, cchs2017_2018_p::GEN_030, cchs2019_2020_p::GEN_030, cchs2001_m::GENA_10, cchs2003_m::GENC_10, cchs2005_m::GENE_10, cchs2015_2016_m::GEN_030, cchs2017_2018_m::GEN_030, cchs2019_2020_m::GEN_030, [GEN_10]","Mental health","Health status","N/A",NA,"Sense of belonging in the community","Sense of community belonging: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"CCC_071","Hypertension","Do you have high blood pressure?","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::CCCA_071, cchs2003_p::CCCC_071, cchs2005_p::CCCE_071, cchs2015_2016_p::CCC_065, cchs2017_2018_p::CCC_065, cchs2019_2020_p::CCC_065, cchs2001_m::CCCA_071, cchs2003_m::CCCC_071, cchs2005_m::CCCE_071, cchs2015_2016_m::CCC_065, cchs2017_2018_m::CCC_065, cchs2019_2020_m::CCC_065, [CCC_071]","Chronic condition","Health status","N/A",NA,"Do you have high blood pressure?","Hypertension: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"CCC_091","COPD/Emphysema/Bronchitis","Do you have COPD (eg bronchitis, emphysema)?","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::[CCC_91A, CCC_91B], cchs2003_p::[CCC_91A, CCC_91B], cchs2005_p::[CCC_91A, CCC_91E, CCC_91F], cchs2007_2008_p::[CCC_91A, CCC_91E, CCC_91F], cchs2015_2016_p::CCC_030, cchs2017_2018_p::CCC_030, cchs2019_2020_p::CCC_030, cchs2001_m::[CCC_91A, CCC_91B], cchs2003_m::[CCC_91A, CCC_91B], cchs2005_m::[CCC_91A, CCC_91E, CCC_91F], cchs2007_2008_m::[CCC_91A, CCC_91E, CCC_91F], cchs2015_2016_m::CCC_030, cchs2017_2018_m::CCC_030, cchs2019_2020_m::CCC_030, [CCC_091]","Chronic condition","Health status","N/A",NA,"Do you have COPD (eg bronchitis, emphysema)?","COPD/emphysema (derived): auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"CCC_101","Diabetes","Do you have diabetes?","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::CCCA_101, cchs2003_p::CCCC_101, cchs2005_p::CCCE_101, cchs2015_2016_p::CCC_095, cchs2017_2018_p::CCC_095, cchs2019_2020_p::CCC_095, cchs2001_m::CCCA_101, cchs2003_m::CCCC_101, cchs2005_m::CCCE_101, cchs2015_2016_m::CCC_095, cchs2017_2018_m::CCC_095, cchs2019_2020_m::CCC_095, [CCC_101]","Chronic condition","Health status","N/A",NA,"Do you have diabetes?","Diabetes: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"CCC_121","Heart Disease","Do you have heart disease?","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::CCCA_121, cchs2003_p::CCCC_121, cchs2005_p::CCCE_121, cchs2015_2016_p::CCC_085, cchs2017_2018_p::CCC_085, cchs2019_2020_p::CCC_085, cchs2001_m::CCCA_121, cchs2003_m::CCCC_121, cchs2005_m::CCCE_121, cchs2015_2016_m::CCC_085, cchs2017_2018_m::CCC_085, cchs2019_2020_m::CCC_085,[CCC_121]","Chronic condition","Health status","N/A",NA,"Do you have heart disease?","Heart disease: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"CCC_151","Stroke","Do you suffer from effects of stroke?","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::CCCA_151, cchs2003_p::CCCC_151, cchs2005_p::CCCE_151, cchs2015_2016_p::CCC_090, cchs2017_2018_p::CCC_090, cchs2019_2020_p::CCC_090, cchs2001_m::CCCA_151, cchs2003_m::CCCC_151, cchs2005_m::CCCE_151, cchs2015_2016_m::CCC_090, cchs2017_2018_m::CCC_090, cchs2019_2020_m::CCC_090,[CCC_151]","Chronic condition","Health status","N/A",NA,"Do you suffer from effects of stroke?","Stroke: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"CCC_280","Mood disorder","Do you have a mood disorder?","Categorical","cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2003_p::CCCC_280, cchs2005_p::CCCE_280, cchs2015_2016_p::CCC_195, cchs2017_2018_p::CCC_195, cchs2019_2020_p::CCC_195, cchs2003_m::CCCC_280, cchs2005_m::CCCE_280, cchs2015_2016_m::CCC_195, cchs2017_2018_m::CCC_195, cchs2019_2020_m::CCC_195, [CCC_280]","Chronic condition","Health status","N/A",NA,"Do you have a mood disorder?","Mood disorder: auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"energy_exp","Daily energy expenditure","Daily energy expenditure","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m, cchs2009_2010_m, cchs2009_m, cchs2010_m, cchs2011_2012_m, cchs2012_m, cchs2013_2014_m, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","cchs2001_p::PACADEE, cchs2003_p::PACCDEE, cchs2005_p::PACEDEE, cchs2015_2016_p::[DHHGAGE_cont, PAA_045, PAA_050, PAA_075, PAA_080, PAADVDYS, PAADVVIG, PAYDVTOA, PAYDVADL, PAYDVVIG, PAYDVDYS], cchs2017_2018_p::[DHHGAGE_cont, PAA_045, PAA_050, PAA_075, PAA_080, PAADVDYS, PAADVVIG, PAYDVTOA, PAYDVADL, PAYDVVIG, PAYDVDYS], cchs2019_2020_p::[DHHGAGE_cont, PAA_045, PAA_050, PAA_075, PAA_080, PAADVDYS, PAADVVIG, PAYDVTOA, PAYDVADL, PAYDVVIG, PAYDVDYS], cchs2001_m::PACADEE, cchs2003_m::PACCDEE, cchs2005_m::PACEDEE, cchs2015_2016_m::[DHH_AGE, PAA_045, PAA_050, PAA_075, PAA_080, PAADVDYS, PAADVVIG, PAYDVTOA, PAYDVADL, PAYDVVIG, PAYDVDYS], cchs2017_2018_m::[DHH_AGE, PAA_045, PAA_050, PAA_075, PAA_080, PAADVDYS, PAADVVIG, PAYDVTOA, PAYDVADL, PAYDVVIG, PAYDVDYS], cchs2019_2020_m::[DHH_AGE, PAA_045, PAA_050, PAA_075, PAA_080, PAADVDYS, PAADVVIG, PAYDVTOA, PAYDVADL, PAYDVVIG, PAYDVDYS], [PACDEE]","Exercise","Health behaviour","METS",NA,"Daily energy expenditure","Daily energy expenditure (derived): auxiliary imputation predictor and study-base descriptive.","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"table1, imputation-predictor","both" +"HWTGHTM","Height","Height (metres)/self-reported - (D,G)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p","cchs2001_p::HWTAGHT, cchs2003_p::HWTCGHT, cchs2005_p::HWTEGHTM, cchs2015_2016_p::HWTDGHTM, cchs2017_2018_p::HWTDGHTM, cchs2019_2020_p::HWTDGHTM, [HWTGHTM]","Height","Health status","meters",NA,"Height (metres)/self-reported - (D,G)","Intermediate: feeder for HWTGBMI_der (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","pumf" +"HWTGWTK","Weight","Weight - kilograms (D,G)","Continuous","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2009_2010_p, cchs2010_p, cchs2011_2012_p, cchs2012_p, cchs2013_2014_p, cchs2014_p, cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p","cchs2001_p::HWTAGWTK, cchs2003_p::HWTCGWTK, cchs2005_p::HWTEGWTK, cchs2015_2016_p::HWTDGWTK, cchs2017_2018_p::HWTDGWTK, cchs2019_2020_p::HWTDGWTK, [HWTGWTK]","Weight","Health status","kg",NA,"Weight - kilograms (D,G)","Intermediate: feeder for HWTGBMI_der (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","pumf" +"CCC_91A","Bronchitis","Do you have chronic bronchitis?","Categorical","cchs2001_p, cchs2003_p, cchs2005_p, cchs2007_2008_p, cchs2001_m, cchs2003_m, cchs2005_m, cchs2007_2008_m","cchs2001_p::CCCA_91A, cchs2003_p::CCCC_91A, cchs2005_p::CCCE_91A, cchs2001_m::CCCA_91A, cchs2003_m::CCCC_91A, cchs2005_m::CCCE_91A, [CCC_91A]","Chronic condition","Health status","N/A",NA,"Do you have chronic bronchitis?","Intermediate: feeder for CCC_091 (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"CCC_91B","Emphysema/COPD","Do you have emphysema or COPD?","Categorical","cchs2001_p, cchs2003_p, cchs2001_m, cchs2003_m","cchs2001_p::CCCA_91B, cchs2003_p::CCCC_91B, cchs2001_m::CCCA_91B, cchs2003_m::CCCC_91B","Chronic condition","Health status","N/A",NA,"Do you have emphysema or COPD?","Intermediate: feeder for CCC_091 (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"CCC_91E","Emphysema","Do you have emphysema?","Categorical","cchs2005_p, cchs2007_2008_p, cchs2005_m, cchs2007_2008_m","cchs2005_p::CCCE_91E, cchs2007_2008_p::CCC_91E, cchs2005_m::CCCE_91E, cchs2007_2008_m::CCC_91E","Chronic condition","Health status","N/A",NA,"Do you have emphysema?","Intermediate: feeder for CCC_091 (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"CCC_91F","COPD","Do you have COPD?","Categorical","cchs2005_p, cchs2007_2008_p, cchs2005_m, cchs2007_2008_m","cchs2005_p::CCCE_91F, cchs2007_2008_p::CCC_91F, cchs2005_m::CCCE_91F, cchs2007_2008_m::CCC_91F","Chronic condition","Health status","N/A",NA,"Do you have COPD?","Intermediate: feeder for CCC_091 (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAA_045","Sweat/breathe harder exercises (18+ years old)","Time spent - sweat/breathe hard exercises in a week (18+ years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAA_045]","Exercise","Health behaviour","hours/week",NA,"Time spent - sweat/breathe hard exercises in a week (18+ years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAA_050","Sweat/breathe harder exercises (18+ years old)","Time spent - sweat/breathe hard exercises in a week (18+ years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAA_050]","Exercise","Health behaviour","minutes/week",NA,"Time spent - sweat/breathe hard exercises in a week (18+ years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAA_075","Sweat/breathe harder activities (18+ years old)","Time spent - sweat/breathe hard activities in a week (18+ years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAA_075]","Exercise","Health behaviour","hours/week",NA,"Time spent - sweat/breathe hard activities in a week (18+ years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAA_080","Sweat/breathe harder activities (18+ years old)","Time spent - sweat/breathe hard activities in a week (18+ years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAA_080]","Exercise","Health behaviour","minutes/week",NA,"Time spent - sweat/breathe hard activities in a week (18+ years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAADVDYS","Active days (18+ years old)","Number of active days (18+ years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAADVDYS]","Exercise","Health behaviour","days",NA,"Number of active days (18+ years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAADVVIG","Vigorous activity (18+ years old)","Time spent - vigorous activity (18+ years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAADVVIG]","Exercise","Health behaviour","minutes/week",NA,"Time spent - vigorous activity (18+ years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAYDVTOA","Sweat/breathe hard activities (12-17 years old)","Time spent - sweat/breathe hard activities in a week (12-17 years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAYDVTOA]","Exercise","Health behaviour","minutes/week",NA,"Time spent - sweat/breathe hard activities in a week (12-17 years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAYDVADL","Leisure activities (12-17 years old)","Time spent - leisure activity in a week (12-17 years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAYDVADL]","Exercise","Health behaviour","minutes/week",NA,"Time spent - leisure activity in a week (12-17 years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAYDVVIG","Vigorous activities (12-17 years old)","Time spent - vigorous activity in a week (12-17 years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAYDVVIG]","Exercise","Health behaviour","minutes/week",NA,"Time spent - vigorous activity in a week (12-17 years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" +"PAYDVDYS","Active days (12-17 years old)","Number of active days (12-17 years old)","Continuous","cchs2015_2016_p, cchs2017_2018_p, cchs2019_2020_p, cchs2015_2016_m, cchs2017_2018_m, cchs2019_2020_m","[PAYDVDYS]","Exercise","Health behaviour","minutes/week",NA,"Number of active days (12-17 years old)","Intermediate: feeder for energy_exp (cchsflow derivation chain).","0.1.0","2026-06-11","Added as auxiliary imputation predictor / Table 1 variable (protocol v0.3.1)","active",NA,"intermediate","both" From 6769813551d980ce878f51ce4d101a2c1c8385f0 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Wed, 10 Jun 2026 15:30:08 -0400 Subject: [PATCH 3/6] Protocol v0.3.1: expanded auxiliary predictors; align stage docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Amend section 3.4.1 and Appendix D with the expanded auxiliary covariate set (marital status, alcohol, BMI, self-rated general and mental health, life stress, community belonging, chronic conditions, energy expenditure — all harmonized in cchsflow v3 through 2019-20), serving imputation quality and the study-base description for planned related studies. Income to 2019-20 documented as a pending upstream cchsflow extension. Add Appendix D to the site sidebar. Rewrite the Stage 5 workflow page for the where-matrix design (NA(b) only; structural missingness preserved; analysis_data is now a list of m completed datasets) and align the manuscript methods text and analysis_data usage with the implemented approach. --- _quarto.yml | 1 + docs/protocol/appendix-imputation.qmd | 3 +++ docs/protocol/full-protocol.qmd | 7 +++-- docs/workflow/5-imputation.qmd | 39 ++++++++++++++------------- manuscript/manuscript.qmd | 5 ++-- 5 files changed, 33 insertions(+), 22 deletions(-) diff --git a/_quarto.yml b/_quarto.yml index 6fd92eb..d04783c 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -40,6 +40,7 @@ website: contents: - docs/protocol/full-protocol.qmd - docs/protocol/study-summary.qmd + - docs/protocol/appendix-imputation.qmd - title: "Workflow" contents: diff --git a/docs/protocol/appendix-imputation.qmd b/docs/protocol/appendix-imputation.qmd index 2f3cae5..c0b3c60 100644 --- a/docs/protocol/appendix-imputation.qmd +++ b/docs/protocol/appendix-imputation.qmd @@ -53,6 +53,9 @@ Predictors are selected by the `imputation-predictor` role in `worksheets/cshm-v - **Survey design and demographics:** survey cycle, sampling weight (`WTS_M`), sex, age (`DHHGAGE_cont`), province, education - **Smoking history:** smoking status (`SMKDSTY_original`), `age_first_cigarette`, `age_start_smoking`, `time_quit_smoking_daily`, `cigs_per_day` +- **Auxiliary covariates** (all harmonized in cchsflow v3 through the 2019–20 PUMF): marital status (`DHHGMS`), drinker type and drinks last week (`ALCDTTM`, `ALWDWKY`), body mass index (`HWTGBMI_der`), self-rated general and mental health (`GEN_01`, `GEN_02B`), life stress and community belonging (`GEN_07`, `GEN_10`), chronic conditions (hypertension, COPD/emphysema, diabetes, heart disease, stroke, mood disorder: `CCC_071`, `CCC_091`, `CCC_101`, `CCC_121`, `CCC_151`, `CCC_280`), and daily energy expenditure (`energy_exp`) + +The auxiliary covariates serve two purposes: they strengthen the MAR assumption for item non-response on the smoking variables, and they describe the study base in Tables 1a/1b — groundwork for planned related studies (smoking initiation and cessation prediction models, pseudo-cohort analyses) that will draw on the same harmonized platform. Household income is the one classic non-response auxiliary not yet included: cchsflow harmonizes it only through 2017–18, and extending it to 2019–20 is a pending upstream task. Including the sampling weight and survey cycle as *predictors* makes the imputation design-consistent without requiring a weighted imputation algorithm — a second improvement over the DemPoRTv2 implementation, which omits design variables. Age, sex, and cycle jointly encode the cohort dimension of the APC analysis model, preserving congeniality with the Stage 8 models. diff --git a/docs/protocol/full-protocol.qmd b/docs/protocol/full-protocol.qmd index 90c7750..15e38e6 100644 --- a/docs/protocol/full-protocol.qmd +++ b/docs/protocol/full-protocol.qmd @@ -3,8 +3,11 @@ title: "A Canadian Smoking Histories Model: a study protocol to generate smoking status: "Draft" version-summary: date: "2026-06-11" - version: "0.3.0" + version: "0.3.1" version-history: + - version: "0.3.1" + date: "2026-06-11" + description: "Expanded the imputation-predictor and Table 1 variable set with auxiliary covariates already harmonized to 2019-20 (marital status, alcohol, BMI, self-rated general and mental health, life stress, community belonging, chronic conditions, energy expenditure), supporting both imputation quality and the study-base description for planned related studies. Income to 2019-20 noted as a pending cchsflow extension." - version: "0.3.0" date: "2026-06-11" description: "Missing data and imputation plan: where-matrix MICE restricted to item non-response (m = 5), survey design variables in the imputation model, derived variables recomputed from imputed feeders. Tables 1a/1b specified as unweighted n with survey-weighted statistics in a single table. Added Appendix D (detailed imputation plan and four-phase assessment, adapting the DemPoRTv2 framework). Documented PUMF cycle-level smoking variable gaps (2019-20, 2022) and the 2015-16 SMKG035 recode." @@ -142,7 +145,7 @@ The CSHM uses a two-stage analytical framework. In the first stage, we estimate Analysis-ready datasets will be produced through a standardized pipeline. 1. **Cleaning:** Distribution checks and truncation of extreme values for continuous variables (e.g., smoking intensity). -2. **Imputation:** Item non-response (don't know/refused; tagged NA(b)) on smoking status, initiation age, and cessation timing will be addressed using Multiple Imputation by Chained Equations (MICE) with *m* = 5 imputations. Imputation is restricted to item non-response through an explicit `where` matrix; structural missingness — not-applicable (NA(a), e.g., initiation age for never-smokers) and not-asked-this-cycle (NA(c)) — is preserved and never imputed directly. The imputation model includes the survey design variables (cycle, sampling weight) and sociodemographic predictors (age, sex, province, education) alongside the smoking history variables, preserving congeniality with the APC analysis models. Derived variables (e.g., pack-years) are recomputed from their imputed feeder variables rather than imputed directly. The full specification, including the diagnostic and sensitivity-analysis plan, is given in [Appendix D: Missing data and imputation plan](appendix-imputation.qmd). +2. **Imputation:** Item non-response (don't know/refused; tagged NA(b)) on smoking status, initiation age, and cessation timing will be addressed using Multiple Imputation by Chained Equations (MICE) with *m* = 5 imputations. Imputation is restricted to item non-response through an explicit `where` matrix; structural missingness — not-applicable (NA(a), e.g., initiation age for never-smokers) and not-asked-this-cycle (NA(c)) — is preserved and never imputed directly. The imputation model includes the survey design variables (cycle, sampling weight) and sociodemographic predictors (age, sex, province, education) alongside the smoking history variables, preserving congeniality with the APC analysis models, plus auxiliary covariates (marital status, alcohol use, body mass index, self-rated general and mental health, life stress, community belonging, chronic conditions, and physical activity) that improve imputation quality and describe the study base for planned related studies. Derived variables (e.g., pack-years) are recomputed from their imputed feeder variables rather than imputed directly. The full specification, including the diagnostic and sensitivity-analysis plan, is given in [Appendix D: Missing data and imputation plan](appendix-imputation.qmd). 3. **Descriptive statistics:** Baseline characteristics will be reported pre-imputation (Table 1a), which discloses the amount and type of missing data, and post-imputation (Table 1b), averaged across the *m* completed datasets. Both tables present unweighted n with survey-weighted percentages (and weighted median/IQR for continuous variables) — the unweighted n discloses the information content of each cell while the weighted statistics describe the population; a fully unweighted variant is provided as an appendix table. Tables 1 will include stratification for sex and survey year. Descriptive statistics will be examined for potential discontinuities across CCHS design eras (2001–2005, 2007–2014, 2015–2021, 2022+), which reflect major changes to the survey frame, recruitment, and collection mode [@gagne2017; @backinger2008]. Although smoking variables can be harmonized across cycles, changes to the sampling frame and mode may introduce measurement differences that are not fully correctable through harmonization alone [@chen2020joinpoint]. ### 3.4.2 Age-period-cohort modelling diff --git a/docs/workflow/5-imputation.qmd b/docs/workflow/5-imputation.qmd index 905d91d..b867e49 100644 --- a/docs/workflow/5-imputation.qmd +++ b/docs/workflow/5-imputation.qmd @@ -1,26 +1,26 @@ --- title: "Stage 5 — Multiple imputation" -subtitle: "Handling missing data with MICE" +subtitle: "Handling missing data with where-matrix MICE" --- ## Purpose -The CCHS has substantial missingness in sociodemographic variables, primarily because some variables are only collected in certain cycles (e.g., immigration status is absent from 2019–20 and 2022) or were refused/unknown by some respondents. Complete-case analysis would discard a large fraction of respondents and introduce selection bias. +The CCHS has missingness from two distinct mechanisms: item non-response (a respondent was asked but did not answer — tagged `NA(b)`) and structural missingness (the question did not apply — `NA(a)` — or was not asked in that cycle — `NA(c)`). Complete-case analysis would discard respondents and introduce selection bias; imputing *structural* missingness would invent values that should not exist (for example, a quit age for a never-smoker). -Multiple imputation by chained equations (MICE) fills in missing values by iteratively fitting a model for each incomplete variable given all other variables. The result is `m = 5` complete datasets, which are later combined using Rubin's rules. This stage produces `analysis_data` — the primary input for all downstream modelling. +This stage applies multiple imputation by chained equations (MICE) to **item non-response only**. An explicit `where` matrix marks the `NA(b)` cells as the only cells MICE may fill; `NA(a)` and `NA(c)` survive the round trip untouched. The result is `m` completed datasets. The full specification, including the diagnostic and sensitivity-analysis plan, is the protocol's [Appendix D](../protocol/appendix-imputation.qmd). ## Inputs | Target | Description | |--------|-------------| | `cleaned_data` | Age-restricted, truncated data frame (Stage 3) | -| `variables_sheet` | Variable metadata — `imputation-predictor` role identifies variables for the MICE model | +| `variables_sheet` | Variable metadata — `imputation-predictor` role identifies variables in the MICE model | Configuration parameters: | Parameter | Value | Meaning | |-----------|-------|---------| -| `imputation_m` | 5 | Number of imputed datasets | +| `imputation_m` | 5 | Number of imputed datasets (1 in `dev`/`draft`) | | `imputation_maxit` | 5 | MICE iterations per imputation | ## Code @@ -32,15 +32,16 @@ cfg <- config::get(file = "config.yml") ``` ```{r load-analysis-data} -# analysis_data is a mids object (multiple imputation dataset) +# analysis_data is a list: datasets (m completed data frames), m, +# imputed_cells (audit of filled cells), logged_events (mice diagnostics) analysis_data <- targets::tar_read("analysis_data") ``` ```{r inspect-imputation} -# Summary of imputation -print(analysis_data) +# Which cells were imputed (NA(b) only, by construction) +analysis_data$imputed_cells -# Missingness before imputation +# Missingness before imputation, by type cleaned_data <- targets::tar_read("cleaned_data") variables_sheet <- targets::tar_read("variables_sheet") @@ -48,7 +49,7 @@ source("R/variables-sheet-utils.R") imputation_vars <- select_vars_by_role("imputation-predictor", variables_sheet) imputation_vars <- intersect(imputation_vars, colnames(cleaned_data)) -# Proportion missing per variable +# Proportion missing per variable (all NA types combined) sort(sapply(cleaned_data[, imputation_vars], function(x) mean(is.na(x))), decreasing = TRUE) ``` @@ -56,22 +57,24 @@ sort(sapply(cleaned_data[, imputation_vars], function(x) mean(is.na(x))), decrea | Target | Description | |--------|-------------| -| `analysis_data` | `mids` object (mice): `m = 5` imputed datasets, stored as `data/analysis_data.rds` | +| `analysis_data` | List: `datasets` (the `m` completed data frames), `m`, `imputed_cells`, `logged_events` | -`analysis_data.rds` is the **primary reproducibility artefact** — sharing it allows others to reproduce all downstream analyses without access to the raw CCHS files. +The completed datasets differ from `cleaned_data` only at the `NA(b)` cells. Table 1b (Stage 6) averages descriptive statistics across all `m` datasets; the APC stage (Stages 7–8) is fitted on imputation 1, with a documented upgrade path to per-imputation fits pooled by Rubin's rules (Appendix D). ## Key decisions -**MICE over single imputation.** Single imputation (e.g., mean substitution) underestimates variability and produces overconfident standard errors. MICE preserves uncertainty about missing values by propagating it across `m` imputed datasets and combining results with Rubin's rules. +**`where`-matrix MICE.** `haven::tagged_na()` values are plain `NA` to `mice()`, so without the `where` matrix MICE would impute structural missingness. Only `NA(b)` cells are flagged for imputation; tagged `NA(a)`/`NA(c)` cells and the original factor levels are preserved by construction, because the write-back copies only `where`-matrix cells into the original data. If MICE declines to impute a flagged cell (constant or collinear variable), the stage stops with an error rather than silently degrading an `NA(b)` tag to plain `NA`. -**Imputation predictor selection.** Only variables with `role` containing `"imputation-predictor"` enter the MICE model. This excludes variables derived from others (e.g., `smoking_status` is derived from raw smoking items) and APC-specific construction variables, which would create circular dependencies. +**Design variables as predictors.** The survey cycle and sampling weight (`WTS_M`) enter the imputation model as predictors alongside age, sex, province, and education, making the imputation design-consistent and congenial with the APC analysis models. -**`m = 1` in dev/draft profiles.** The `dev` and `draft` config profiles use `imputation_m: 1` and `imputation_maxit: 1` to speed up iteration. The production run uses `m = 5, maxit = 5` per the protocol. +**Imputation predictor selection.** Variables with the `imputation-predictor` role enter the MICE model. Derived variables (e.g., `pack_years_der`) are not imputed directly — they are recomputed from their imputed feeders. -**Cycle-specific missingness.** Some variables (e.g., immigration status, ethnicity) are structurally absent from certain CCHS cycles. These are `NA(c)` (tagged NA: not asked this cycle) in the harmonized data. MICE treats these as missing and imputes them — this is appropriate because the variable *would* have had a value if collected. Tagged NA codes are stripped to regular `NA` before MICE is run. +**`m = 1` in dev/draft profiles.** The `dev` and `draft` config profiles use `imputation_m: 1` and `imputation_maxit: 1` for fast iteration. Production uses `m = 5, maxit = 5`; adequacy is checked against the fraction of missing information (Appendix D, Phase 2). + +**Structural missingness is an analysis decision, not an imputation default.** `NA(c)` cycle-level absence (e.g., `age_start_smoking` in 2019–20 PUMF) is *not* imputed in the base pipeline. Whether to impute across cycle-level absence — a stronger MAR assumption — is evaluated with the delta-adjustment sensitivity analysis in Appendix D, with the RDC Master files as an external check. ## Related resources -- `R/imputation.R` — `impute_data()` +- `R/imputation.R` — `impute_data()`, `prepare_for_mice()` +- [Protocol Appendix D — Missing data and imputation plan](../protocol/appendix-imputation.qmd) - [mice R package](https://www.rdocumentation.org/packages/mice) -- Protocol §3.3 — missing data strategy diff --git a/manuscript/manuscript.qmd b/manuscript/manuscript.qmd index 2503a22..f1eac9b 100644 --- a/manuscript/manuscript.qmd +++ b/manuscript/manuscript.qmd @@ -36,7 +36,8 @@ fmt_pct <- function(x, digits = 1) paste0(round(x * 100, digits), "%") # Run targets::tar_make() first if results are not available. study_data <- tar_read("study_data") cleaned_data <- tar_read("cleaned_data") -analysis_data <- tar_read("analysis_data") +imputation_result <- tar_read("analysis_data") +analysis_data <- imputation_result$datasets[[1]] # imputation 1; see protocol Appendix D ``` ```{r derived-values, include=FALSE} @@ -105,7 +106,7 @@ Period effects were held constant beyond observed data: men's initiation from `r ## 2.4 Missing data -Missing sociodemographic variables were handled using multiple imputation by chained equations (MICE; `m = `r cfg$imputation_m`, maxit = `r cfg$imputation_maxit`). Variables structurally absent from some cycles (e.g., immigration status) were imputed using available cycle data. +Item non-response (don’t know/refused) was handled using multiple imputation by chained equations (MICE; `m = `r cfg$imputation_m`, maxit = `r cfg$imputation_maxit`), restricted through an explicit `where` matrix so that structural missingness (not-applicable and not-asked-this-cycle values) was preserved rather than imputed. The imputation model included the survey design variables (cycle, sampling weight) and sociodemographic predictors; derived variables were recomputed from their imputed feeders. The full specification and assessment plan are given in the protocol, Appendix D. # 3. Results From 539171dd977bbc2f12d583d40a75955b4062d3e4 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Wed, 10 Jun 2026 17:06:52 -0400 Subject: [PATCH 4/6] Fix PR #6 review findings: predictor matrix, render wiring, guards Imputation (R/imputation.R): - Predictor matrix excludes structural-NA variables as predictors (they may be imputed but cannot predict): mice drops rows with missing predictors from each conditional model, so a structural-NA predictor such as time-since-quit made most NA(b) cells in other variables unpredictable. Synthetic validation: all 700 flagged cells fill, zero untagged NA, all NA(a)/(c) tags preserved. - Chain-mean convergence check scoped to actually-imputed variables and promoted to stop() (the unscoped version warned on every healthy run and could never detect the NaN-free degenerate case). - Dropping a prespecified design predictor (weight, cycle, sex, age) from the imputation model is now an error, not a one-shot warning. - pack_years_der recomputed from imputed feeders via cchsflow::calculate_pack_years after each completed dataset, delivering the protocol's derived-variable claim. Descriptives: - table-1.qmd renders rows by the table1 role (the 15 auxiliaries were computed but silently dropped at render), adds the Health status section, weights the cycle appendix, and adds the fully unweighted appendix variant promised by the protocol (new unweighted targets). - Survey weights validated at the engine boundary (stop on missing or non-positive); weighted_quantile rejects bad weights; zero-row table lookups stop as wiring bugs instead of rendering 'No data'; MI-averaged n rounded for display; MI row-alignment invariant. Docs: Appendix D aligned with the implementation (NA(c) not imputed in the base pipeline with the Phase 4 evaluation path; predictor-matrix design documented; factor-level mechanism wording; checklist updated), stage 5/6 docs and CLAUDE.md role table corrected. Tests: declined-cell guard exercised (either layered stop accepted), structural-predictor exclusion, pack_years recomputation, weighted percent/median integration, MI averaging with exact expectations, vacuous assertion replaced; cchsflow attached in setup.R mirroring tar_option_set. 102 tests pass. --- CLAUDE.md | 6 +- R/create-descriptive-tables.R | 18 +++- R/descriptive-data.R | 10 +- R/get-descriptive-data.R | 19 +++- R/imputation.R | 114 ++++++++++++++++++++--- _docstyle/reference.docx | Bin 9713 -> 9884 bytes _docstyle/reference.docx.hash | 2 +- _targets.R | 8 ++ docs/protocol/appendix-imputation.qmd | 22 +++-- docs/results/table-1.qmd | 44 +++++++-- docs/workflow/5-imputation.qmd | 2 +- tests/testthat/setup.R | 4 + tests/testthat/test-descriptive-tables.R | 59 +++++++++++- tests/testthat/test-imputation.R | 46 +++++++++ 14 files changed, 310 insertions(+), 44 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f19ce29..a70e2a8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -133,10 +133,10 @@ Roles are comma-separated in `cshm-variables.csv`. A variable may carry multiple |------|-------|---------| | `design` | Survey design | Survey infrastructure (SurveyCycle, WTS_M) | | `intermediate` | Harmonization | Raw cchsflow input needed to derive a unified variable; not used directly by pipeline code | -| `predictor` | Model | Covariate in the APC model or descriptive analysis | +| `predictor` | Model | Covariate in the APC model | | `model-stratifier` | Model | Stratifies APC into separate fits (e.g. DHH_SEX) | -| `table1` | Descriptive | Row in Table 1 descriptive statistics | -| `table1-stratifier` | Descriptive | Stratifies Table 1 columns | +| `table1` | Descriptive | Row in Table 1 (drives row selection in `get_cshm_desc_data()`) | +| `table1-stratifier` | Descriptive | Reserved for cycle/extra stratification of descriptive tables (not yet consumed by code) | | `apc-numerator` | APC data prep | Defines the event indicator in Stage 7 | | `apc-denominator` | APC data prep | Constructs the at-risk person-year denominator in Stage 7 | | `imputation-predictor` | Imputation | Included in MICE imputation model | diff --git a/R/create-descriptive-tables.R b/R/create-descriptive-tables.R index daf6bf4..9fab4b6 100644 --- a/R/create-descriptive-tables.R +++ b/R/create-descriptive-tables.R @@ -20,7 +20,11 @@ continuous_predictor_footnote <- paste0( # per protocol v0.3.0 §3.4.1. Without weights they fall back to unweighted stats. format_cat_descriptive_data <- function(descriptive_data_row) { - if (is.na(descriptive_data_row[1, "n"]) || descriptive_data_row[1, "n"] == 0) { + if (nrow(descriptive_data_row) == 0 || is.na(descriptive_data_row[1, "n"])) { + stop("Descriptive table lookup matched no engine row — worksheet/data ", + "category mismatch (wiring bug), not an empty stratum.", call. = FALSE) + } + if (descriptive_data_row[1, "n"] == 0) { return("No data") } pct <- if ("wtd_percent" %in% colnames(descriptive_data_row) && @@ -29,12 +33,17 @@ format_cat_descriptive_data <- function(descriptive_data_row) { } else { descriptive_data_row[1, "percent"] } - formatted_n <- format(descriptive_data_row[1, "n"], big.mark = ",") + # MI-averaged n can be fractional; display as a rounded count + formatted_n <- format(round(descriptive_data_row[1, "n"]), big.mark = ",") paste0(formatted_n, "\n (", round(pct * 100, 1), ")") } format_cont_descriptive_data <- function(descriptive_data_row) { - if (is.na(descriptive_data_row[1, "n"]) || descriptive_data_row[1, "n"] == 0) { + if (nrow(descriptive_data_row) == 0 || is.na(descriptive_data_row[1, "n"])) { + stop("Descriptive table lookup matched no engine row — worksheet/data ", + "mismatch (wiring bug), not an empty stratum.", call. = FALSE) + } + if (descriptive_data_row[1, "n"] == 0) { return("No data") } weighted <- "wtd_median" %in% colnames(descriptive_data_row) && @@ -405,6 +414,7 @@ create_cycle_specific_descriptive_table <- function( cycle_col, cycle_labels, column_stratifier = NULL, + weight_var = NULL, sections_order = NULL, include_na = TRUE ) { @@ -428,7 +438,7 @@ create_cycle_specific_descriptive_table <- function( cycle_desc <- get_descriptive_data( cycle_data, variables_sheet, variable_details_sheet, - variables, stratify_config + variables, stratify_config, weight_var = weight_var ) cycle_tables[[key]] <- .build_descriptive_table_data( cycle_desc, variables_sheet, variable_details_sheet, diff --git a/R/descriptive-data.R b/R/descriptive-data.R index 7c49e97..487560b 100644 --- a/R/descriptive-data.R +++ b/R/descriptive-data.R @@ -9,7 +9,7 @@ #' Calculate descriptive statistics for the CSHM study population #' -#' Computes statistics for all predictor variables stratified by +#' Computes statistics for all table1-role variables stratified by #' model-stratifier. No row stratification is applied in the base table. #' #' @param data Cleaned or imputed study data frame @@ -20,8 +20,8 @@ get_cshm_desc_data <- function(data, variables_sheet, variable_details_sheet, weight_var = NULL) { # Table rows are selected by the table1 role (the documented wiring); - # columns are stratified by model-stratifier (sex). table1-stratifier - # drives the cycle-specific appendix table, not this base table. + # columns are stratified by model-stratifier (sex). The cycle-specific + # appendix table is wired from config (survey_var cycle/sex), not roles. predictor_vars <- select_vars_by_role("table1", variables_sheet) sex_stratifier <- select_vars_by_role("model-stratifier", variables_sheet)[1] stopifnot(!is.na(sex_stratifier)) @@ -71,6 +71,10 @@ get_cshm_desc_data_mi <- function(imputation_result, variables_sheet, if (length(per_imp) == 1) return(per_imp[[1]]) stacked <- dplyr::bind_rows(per_imp, .id = ".imp") + # All imputations must produce identical row sets (worksheet-driven rows; + # factor levels preserved by the write-back) — averaging mismatched groups + # would be silent corruption. + stopifnot(nrow(stacked) == nrow(per_imp[[1]]) * length(per_imp)) mean_cols <- intersect( c("median", "percentile25", "percentile75", "n", "percent", diff --git a/R/get-descriptive-data.R b/R/get-descriptive-data.R index 2f674fa..c660615 100644 --- a/R/get-descriptive-data.R +++ b/R/get-descriptive-data.R @@ -16,6 +16,11 @@ #' @param probs Probabilities in [0, 1] #' @return Numeric vector of weighted quantiles weighted_quantile <- function(x, w, probs) { + if (anyNA(w) || any(w <= 0)) { + stop("weighted_quantile: weights must be positive and non-missing ", + "(found ", sum(is.na(w)), " NA and ", sum(!is.na(w) & w <= 0), + " non-positive of ", length(w), ").", call. = FALSE) + } if (length(x) == 0) return(rep(NA_real_, length(probs))) if (length(x) == 1) return(rep(x, length(probs))) ord <- order(x) @@ -34,8 +39,18 @@ get_descriptive_data <- function( weight_var = NULL ) { use_weights <- !is.null(weight_var) - if (use_weights && !weight_var %in% colnames(data)) { - stop("weight_var '", weight_var, "' not found in data.") + if (use_weights) { + if (!weight_var %in% colnames(data)) { + stop("weight_var '", weight_var, "' not found in data.") + } + w_all <- data[[weight_var]] + if (anyNA(w_all) || any(w_all <= 0, na.rm = TRUE)) { + stop("Survey weight '", weight_var, "' has ", sum(is.na(w_all)), + " missing and ", sum(!is.na(w_all) & w_all <= 0), + " non-positive values — weighted statistics would silently ", + "renormalize over a different population than the displayed n. ", + "Resolve upstream (harmonization/cleaning) first.", call. = FALSE) + } } descriptive_data <- data.frame( variable = c(), diff --git a/R/imputation.R b/R/imputation.R index ab05189..cf30451 100644 --- a/R/imputation.R +++ b/R/imputation.R @@ -73,17 +73,55 @@ impute_data <- function(cleaned_data, variables_sheet, cfg) { maxit <- cfg$imputation_maxit %||% 5 message("Running MICE: m=", m, ", maxit=", maxit) + # Predictor matrix: variables with structural missingness (plain NA in the + # modelling frame at where = FALSE cells, i.e. NA(a)/NA(c) in the source) + # may BE imputed but must not SERVE as predictors. mice excludes rows with + # missing predictors from each conditional model, so a structural-NA + # predictor (e.g. time-since-quit, NA(a) for current and never-smokers) + # would make most NA(b) cells in other variables unpredictable. The + # complete design and auxiliary variables form the predictor core. + has_structural <- vapply(impute_vars, function(v) { + any(is.na(prep$data[[v]]) & !prep$where[, v]) + }, logical(1)) + pred_matrix <- mice::make.predictorMatrix(prep$data) + if (any(has_structural)) { + pred_matrix[, impute_vars[has_structural]] <- 0L + message( + "Excluded as predictors (structural missingness): ", + paste(impute_vars[has_structural], collapse = ", ") + ) + } + mice_result <- mice::mice( prep$data, - m = m, - maxit = maxit, - where = prep$where, - printFlag = FALSE + m = m, + maxit = maxit, + where = prep$where, + predictorMatrix = pred_matrix, + printFlag = FALSE ) - # Surface silently dropped/altered variables (constant, collinear, etc.) + # Surface silently dropped/altered variables (constant, collinear, etc.). + # Dropping a prespecified design predictor (weight, cycle, sex) silently + # voids a protocol property — that is an error, not a warning. logged <- mice_result$loggedEvents if (!is.null(logged) && nrow(logged) > 0) { + design_vars <- intersect( + c("WTS_M", "SurveyCycle", "DHH_SEX", "DHHGAGE_cont"), impute_vars + ) + dropped <- unique(unlist(strsplit(as.character(logged$out), ",\\s*"))) + # mice logs factor predictors as variable+level (e.g. "SurveyCycle10") + design_dropped <- design_vars[vapply(design_vars, function(v) { + any(dropped == v) + }, logical(1))] + if (length(design_dropped) > 0) { + stop( + "MICE dropped prespecified design predictor(s) from the imputation ", + "model: ", paste(design_dropped, collapse = ", "), + " — the design-consistency property of the protocol no longer holds. ", + "Inspect loggedEvents and the predictor matrix.", call. = FALSE + ) + } warning( "MICE logged events (variables dropped or altered in the imputation model):\n", paste(capture.output(print(logged)), collapse = "\n"), @@ -91,12 +129,22 @@ impute_data <- function(cleaned_data, variables_sheet, cfg) { ) } - # Convergence check: mice() does not error on pathological chains, so a - # quick guard on non-finite chain means catches degenerate fits. - if (any(!is.finite(mice_result$chainMean), na.rm = FALSE) && - any(is.nan(mice_result$chainMean))) { - warning("Non-finite MICE chain means detected — inspect convergence ", - "(mice::plot) before using these imputations.", call. = FALSE) + # Convergence check, scoped to the variables that were actually imputed. + # (Variables with zero where-TRUE cells always have NaN chain means, so an + # unscoped check would warn on every healthy run.) + imputed_vars <- imputed_cells$variable[imputed_cells$n_imputed > 0] + cm <- mice_result$chainMean[ + intersect(imputed_vars, dimnames(mice_result$chainMean)[[1]]), , , + drop = FALSE + ] + if (length(cm) > 0 && any(!is.finite(cm))) { + bad <- dimnames(cm)[[1]][apply(!is.finite(cm), 1, any)] + stop( + "Non-finite MICE chain means for imputed variable(s): ", + paste(bad, collapse = ", "), + " — degenerate fit; inspect mice::plot() before proceeding.", + call. = FALSE + ) } # Write back: for each imputation, copy ONLY where-matrix cells into the @@ -129,7 +177,7 @@ impute_data <- function(cleaned_data, variables_sheet, cfg) { out[[var]][cells] <- imputed_vals } } - out + recompute_derived(out) }) message("Imputation complete: ", m, " datasets.") @@ -139,6 +187,48 @@ impute_data <- function(cleaned_data, variables_sheet, cfg) { ) } +#' Recompute derived variables from imputed feeders +#' +#' Derived variables are not imputed directly (protocol Appendix D): their +#' feeders are imputed and the derived values are recomputed with the same +#' cchsflow function used at harmonization, guaranteeing internal consistency +#' by construction. Currently covers pack_years_der, the one derived variable +#' in the Table 1 set whose feeders are imputation targets. +#' +#' TODO(worksheet-driven): generalize by walking the DerivedVar chain in the +#' variable-details worksheet instead of naming feeders here. +#' +#' @param data A completed (imputed) data frame +#' @return Data frame with derived variables recomputed +recompute_derived <- function(data) { + if (!"pack_years_der" %in% colnames(data)) return(data) + + as_num <- function(x) { + if (is.factor(x)) suppressWarnings(as.numeric(as.character(x))) else x + } + feeders <- c("SMKDSTY_original", "DHHGAGE_cont", "age_start_smoking", + "cigs_per_day", "time_quit_smoking", "SMK_05B", "SMK_05C", + "age_first_cigarette", "smoked_100_lifetime") + if (!all(feeders %in% colnames(data))) { + warning("pack_years_der not recomputed — missing feeders: ", + paste(setdiff(feeders, colnames(data)), collapse = ", "), + call. = FALSE) + return(data) + } + data$pack_years_der <- cchsflow::calculate_pack_years( + smoking_status = as_num(data$SMKDSTY_original), + age = as_num(data$DHHGAGE_cont), + age_start_smoking = as_num(data$age_start_smoking), + cigs_per_day = as_num(data$cigs_per_day), + time_quit_smoking = as_num(data$time_quit_smoking), + cigs_occasional = as_num(data$SMK_05B), + days_per_month = as_num(data$SMK_05C), + age_first_cigarette = as_num(data$age_first_cigarette), + smoked_100_lifetime = as_num(data$smoked_100_lifetime) + ) + data +} + #' Prepare the MICE modelling frame and where matrix #' #' Builds a copy of `vars` in which every tagged NA is plain NA (mice cannot diff --git a/_docstyle/reference.docx b/_docstyle/reference.docx index 49d00a679f274810b13c47efecce13f2d77cb20e..9445ac49d7597cdae6e896cbec9e904f1799901d 100644 GIT binary patch delta 3217 zcmY+Gc{tQv|HsFYA}xSYvb3q}yB_KO{qE;E*LAM5%%S+39d=e1y-WeKr1XJh9FadL8kWZgO| zAmS`+drDGdL9nv%uy^8Y*=BzmI6Za*;AY;c!cUu6lQ(0IAw;dX{K3Z}f}|WUx41hP zc7D4LJ+AMp7FK;#Up8zv@77woaV=3KQd;DZjRiX5uP)F-R%5nW*mXKhqIM1nt>`NZ(JvS~FQUaDA22BG4_9X|POzrrn5 zhlQ057iZx%P_ha5-uT1nS;{-O#~-qWRXe!m zDlaesz7n)OCi3d5VjAh1wK zEc$&(dg(hMv9%H~-zLety?&Ow3(d^Ef)4pMSR>z#zJ^jsO3 zgS2}u5*FQumCBs}psL7K!8N@_)(_Tcy1F_)`uO0x-ixjcXRVi)m4}LYIFaAdM=NPB zM`QDvPXUmlc;M}>7S~+#_iGyzUV$aYqYL~H?{?bx=wgMfMYh1j7Z<@14K60?pryL) zc-Ij*qjmcS95uNjv&bjsTpmWn6*@G@ifT0U{p~b8Op2BdG|331#une!HloPyrHsm6zvXEcGo*eOOYzyZO{=%3)ViUXm1 zISx|pPg;3dPSL5SJE#5fs?rox82*Ft$1$Fh?~9KgkLUiV5ZVP8E8F*s*1f3ezWU$* zCfPR@dhfdoJn-uT$H`i0LV~DOV96DgH8mZ zNGymAYUB;vWfj&78|JG3v$Bs=c_|*>zW(bmCK;oenFeXcGcjXw0cJhf)|uOY(fBpc zhnpL~R%aC^YUY5_HlzMijG9e%L-z)Yg_^F{Xbv@tK4g|96umQJM!zN0AZee(0iT1Y zp1LScix8Uh)>0};6|$^=FPb5I5iM@5jwkv-rhrU74+v*R9e+AW()p&(E4Xw2X44ns zyZPxiGj*&9%8%7;6z49}Bg=XE+o9B#=wVxf4qMbG4BE6txR0ZWr2-jpK4c+y@)Qd+ zCN1DtiPA6I>#p;O)NM(x^Bx`C{1tX^J*jpyR%^( z222;?O&FVNS&?tHa1NhQie2B#R(6;zxK6y0qufFa8H*u8NAyF>{ zyasT-$r!r~?`VYF14Ubvj$9l~EF62hL>LXsPc?R5(R(R=Kb<#pTAL^fkoW>UUA10p z*OIu7Nn_)%q3rx8C)ojL*G#|uPf;n4nWea8IE=ED&e9OE`$aeN9OVfh@p$-+- zJbZYdGB;$TQZvpg(ZDaX>`0TbB}dmB)D=%Y)@n z$T`8_J)ZL`D8z}qrmr=7GH%YvnU%SX zQOM@azkQC(gk_bA(EJ<6CbLzul=1;Ei60#(F8E{mBiS1nEc{6 zs->1c{VfdBzw4ZPI_({U6*cG8o+YX5(VZlKngQis<~TDK78Uqp#jwi$`D)4irG}GJ zbFt}U!M3+dD;hJN4DM#WNGCHsdj2r^XKAh$p`4sXX2g>PB6uE?Z};#iIb8fzd8w@8 z``M#DFz@ha3cG-t2BcV-g zmoA913-WQquA~%PI!AkQ*Q;lTKe_80Mr4dsbjgf{XZzI>DyH&utMWp}LB#>HO(_+k zXoY2~8NRoKMXBRKO=JH=CRtYIi;dRs*_OhLwPrqXB7j#P6Yg?;=AWHWY~&=+57j0X zw3kN*`IbMgpSKs&8eJa$n066wIN~QuS)Q`5Tfw85O-yMTQwiSIn59k8eQICW<4j)a zhAEf(^Z+TJQGSNgqtn|+g%^IBzvQ){3R;{dPR^o{1zum!#|m_AXrAkCMLnfi#L@|n zngk0#zwcJq5vO)7B^AU7>ny_SIW=xOxf~7)Dak>XRa}yID?A*;p(&Nx@bwO~(D} zxFjy@GM}%EC4n$5)ye7DcNlBiTAG~B4K4@nXkunAee!TQFtYZAKQ`^kiU4$IH%ZM< zWH6i>c&Fu^3_NBov$T*W2~;)UHA2)}X8)`7R6=4BQXv!n-9_w%tsVP~xSGmcRYWPo zn~SLXI(;G)26-%#pp62$G^2Bi68d+nuUqt3X{h$G+flXo7_pUXCos>HsS}B32J-+1 zitsrFvD^ynFWz?t?j97YEAQ|(&@wiPSf@_CzqY_0U$fHkJt+dMjH`nsRj5qpKN_s9 z9urj~>j@3aq8!KAJ3Q};5$ZIeietsW{bKcPl6gO`>^i&Klu(ip^EZ}Y;i76+zo9!6 zIG%Uq(uSCC4t4sYfx{b4T>!O5qWX$AI41R>v*DbB37DP&O)^VvyQnAO{JO5>=dxT_ zWn6xQ0Zl&D<j7;MFk8uM~`)8PZUAv z<@R^fc_msma7uDXjz&4P*eLwf%|*4+`Y;FZw5qUF@fV~vT__y I>EG^u0GTN2g#Z8m delta 3049 zcmY+GWmFW<)`o`;i2+7X$r(aYO1dNjDG3ola_CM)V1NO{0hAt)8akzv4uPRdYUmaO zNdajPkV|~7@7}Z4S?@Y~zbE!Se;$*YE4l$HJ^%nY>?0MhoO@jJs{WLfSljryFA&;qLVbS5Mo>EzE9ffawfu12Z0oFkvy+G?g zFNBh*gB3&^jF%R@Z2FamNfbfG!oKa2lzkj?V48DUZtAwYgsF5d*gnl3$TLh}Z{JSN zi?=i7!uh1tyj=0MVi|X?B&Dk^vRe9dt5*6YDU^`Bw4sZY zzs|Zf9OG-M5)iu3OY8Ds0RRHn0KosWy1WQ4@VeJXeuCcX7V{HOms1lqVO{rKhJqTo zO;mfGK$B4;8%glXJBjCRcH9IOualR?@;XxNn{Pvt1^jc+Ax(g@S4Wl0P;g-Xy8v0k zN>Uz9C4A%R072l>Zi~W{zGpMS6Ot`_3n0mT$OPb%A^v>WdC<^~?_-pT>GT=^1Q_kPj zjz5R5#%n74&K6qp&_eeZfqxSJyb6?RYad~zBgH|?G-<(KghYd(nsKN{m^YeRUJS(O zGPctmpcQVkGJEc7i=rC~?d zU5@2!1;^dQI!FjlS;BwGo~c;y9K@^V_%pnV^+_gj1keFtW+JRKXaxq){1e=h+?e&< zys`~9Ym4Oz4wpO);7AWWQ_ho6Yc>tDtT814(dCVA>*xGdY$6{A&`GA1ZDBFEh z!v{ib>SUDTN_K}@N-WT0dpzgf@WvdIfwWKPKvYRZOYYM!dwaN+of6_h_cFH=(Suv&`x}HmYF_tlu|NwpPN2i?n?n4YirDZd!C|qaxaHKzr#Nce##%+kdv&l zaDl3zBG7&sq<^~D`aFiyz-@zi*Y@XM&6r(W1gMXd*Uq}~px8PDOg zNlrGN>gsQ?s9feVFf(yJqvf&xw265Xry;~OUEhHaMf=@dP}{T5tE zmm*wJqDB1o&cRRehsk%J!C!bsA^8Q?)+$ipp`a99nxNj~-h!=qK`P6!>O<#k?e4Pb zo(9z)+f{c-H>>4ZdpLr7G~MbV*)iKL)3a|Blsm&>`gbz38zR~DEgxTyv5N_2E&cM; z8ah!;SBMv$#THnx>OwK{TJ%TU78nI&+2bo7c?P%|@5?s4C_X9(hVStSF08yuY@q4< zHs{ngm()PeKc^}P)EA2Nu-KKK4=5H~(F9{W{13CsW0a0`3k?au>e?A&`B zehbmot{L&@tqp$z-#Pa&PI3T#zcY4-G}+I1|D^F;DrnC4lM#*Xjbkr`@AQ(;X=X)g z4$_KP)epqq;<<%jcJM>GhcE^$f5LjuK_pEu3w zD8?TasS)we&IitTZWNw1Od1GeOUIe}LL4%^7$t{gRcy;sFh#fRH%1ibht{J!2EjZ^ z2-aAGw~hyn=;f~F+?CYEeI!Yp71v2lT4PUfp28viAvMS0Q#hX3Vf)C*C`E&S@7z|} zn`+Mhby!V|T0E2h(#aOgC1z{cx}egn|7?PA*y`mm5lg{i#R=+g zs4)WVhAtyR2MzmOBfvS-eE0c_iSO?+wm5LFD#%3Zn0HH33l@!%xICis8b7B~cgVa8 z%?zYWnI&;ufS0!OuE~0-rW*C`kKNHzX7L8A?MsUH*zK^hgh_-U%r>J;q$^=j{Gp7r2@-$x{@jR6S6C@4@76z*W(;3b(1RP+sdlalLO@IR9Wz7bAnP zs9fLwEa;@`{6KqJh%u+(P%qOvu@`q@Cz*@lv z>|@Ryd&C4bMJ0cx=WSANM;J~lRi&kXGo*aX{5D14*zKp@&^Mh@OYz-?w>ow7oYEBR z45v54;fZsMVP5WuMUH_>net@4Gs%xeno|jE$z~TL^*k#uZd4z<1bBwOxZ5UbPfI<> z7y8bsU$NODf0ECy;-k|D;8?Hccv*#F2!oR_bta`ptGs?THEXAc5EJ?BE9p3qq+^Fp zADXGj6D5m(qZNjUBXD;l(|12c5TaU|X=!tP;YVU6w1>bt4NRfWUBS6Pvt*j1?1`tr z*;S!sR(G0~K8vpu*Umz9-@rf>2kbUz9^yfB@l9>+-nI`RPb?(b+8<;@$&5c6n^8)5 zwa>RU!sq3lfHK+wtLYGz@;|hAm9~2?xI)l(1zs6l%}u%Zvm;Ig?^L*OZOSOeetfE| z8cvbb5fNLD7{J#x!cNtBY(4$Hf3-WZ(?~j%=l2q&#?j!Efnz6T)9BR$rw?1yVbVxD zHCf*hudKVcby$j%Zxs)CdkUt=@TgA6Yb!2jWrCClL%Q1Fu4_u7hX`M*E=2m8#Nr_mgqH+tctvu^DTv5*62z=@t;Wof z*P3#vSGYXGEOou6W0APlm;a{;EI089mYc)qM}DLsO<1Ur{H#pZ8+BN346&?|WLLNS z|I?9kti0D2AR8?{k8LQ|LV?cmEY(l00IUpbeWjg`cNgo3G&2sR+TIru8` EAM$3QCIA2c diff --git a/_docstyle/reference.docx.hash b/_docstyle/reference.docx.hash index 7347d0c..26a7f44 100644 --- a/_docstyle/reference.docx.hash +++ b/_docstyle/reference.docx.hash @@ -1 +1 @@ -df16fcecdd10278402b00fd6628083a56c8eb6d3d63d2ea55398b08e95ea7787 +35f6b93bf4138dfd8b3564314afa95f1958b243e288ab042f47f34977e8ac004 diff --git a/_targets.R b/_targets.R index 1f5e41c..bdbbee0 100644 --- a/_targets.R +++ b/_targets.R @@ -66,6 +66,14 @@ list( weight_var = survey_var(cfg, "weight")) ), + # Fully unweighted variants (protocol §3.4.1 appendix tables) + tar_target(table_1a_unweighted_data, + get_cshm_desc_data(cleaned_data, variables_sheet, variable_details_sheet) + ), + tar_target(table_1b_unweighted_data, + get_cshm_desc_data_mi(analysis_data, variables_sheet, variable_details_sheet) + ), + # Stage 7: Prepare APC datasets (numerator + denominator combined, by sex) # Single target; denominator construction is the expensive step. # Keeping as one target lets {targets} cache the full APC data independently diff --git a/docs/protocol/appendix-imputation.qmd b/docs/protocol/appendix-imputation.qmd index c0b3c60..75251c9 100644 --- a/docs/protocol/appendix-imputation.qmd +++ b/docs/protocol/appendix-imputation.qmd @@ -9,7 +9,7 @@ This appendix specifies the missing-data strategy for the Canadian Smoking Histo **Method:** Multiple imputation by chained equations (MICE) -**Number of imputations:** *m* = 5, with *maxit* = 5 iterations (production configuration; *m* = 1 in the `dev` profile for fast iteration). The adequacy of *m* will be assessed by examining the fraction of missing information (FMI) for key APC model coefficients; if FMI exceeds 0.5 for any coefficient, *m* will be increased. +**Number of imputations:** *m* = 5, with *maxit* = 5 iterations (production configuration; *m* = 1 and *maxit* = 1 in the `dev` and `draft` profiles for fast iteration). The adequacy of *m* will be assessed by examining the fraction of missing information (FMI) for key APC model coefficients; if FMI exceeds 0.5 for any coefficient, *m* will be increased. **Key dependencies:** mice, haven, naniar, VIM @@ -21,7 +21,7 @@ The CSHM pipeline preserves the reason for missingness throughout, using `haven: |-----|---------|-----------| | NA(a) | Not applicable (e.g., initiation age for never-smokers) | Structural — never imputed | | NA(b) | Don't know / refused / not stated | Item non-response — imputed under MAR | -| NA(c) | Not asked in this survey cycle | Cycle-level absence — imputed under a stronger MAR assumption, with sensitivity analysis | +| NA(c) | Not asked in this survey cycle | Cycle-level absence — not imputed in the base pipeline; imputing across cycles is evaluated in Phase 4 (delta-adjustment, RDC check) | Three sources of missingness are specific to this study: @@ -43,9 +43,9 @@ Three sources of missingness are specific to this study: ### Cells eligible for imputation: the `where` matrix -MICE will be called with an explicit `where` matrix in which a cell is `TRUE` only if its value is tagged NA(b) (or NA(c), for variables imputed across cycle-level absence). This differs from the DemPoRTv2 implementation, which converts all tagged NAs to plain `NA` before imputation: because `haven::tagged_na()` values *are* `NA` to `mice()`, that approach silently imputes structural missingness — for example, inventing quit times for never-smokers. The `where` matrix makes the universe logic automatic: a never-smoker's NA(a) on `age_first_cigarette` is never a candidate cell, while an ever-smoker's NA(b) on the same variable is. +MICE will be called with an explicit `where` matrix in which a cell is `TRUE` only if its value is tagged NA(b). (Extending the matrix to NA(c) cells for prespecified variables is the mechanism by which cycle-level imputation would be enabled, if the Phase 4 evaluation supports it.) This differs from the DemPoRTv2 implementation, which converts all tagged NAs to plain `NA` before imputation: because `haven::tagged_na()` values *are* `NA` to `mice()`, that approach silently imputes structural missingness — for example, inventing quit times for never-smokers. The `where` matrix makes the universe logic automatic: a never-smoker's NA(a) on `age_first_cigarette` is never a candidate cell, while an ever-smoker's NA(b) on the same variable is. -For categorical variables, the `"NA(a)"` and `"NA(c)"` factor levels are removed from the modelling frame before imputation (and restored afterward) so that polytomous regression cannot assign a structural-missing label as an imputed value. +For categorical variables, all `"NA(x)"` factor levels are removed from the modelling frame so that polytomous regression cannot assign a structural-missing label as an imputed value; the original levels are preserved in the output because only `where`-matrix cells are written back into the original data. ### Variables in the imputation model @@ -57,6 +57,8 @@ Predictors are selected by the `imputation-predictor` role in `worksheets/cshm-v The auxiliary covariates serve two purposes: they strengthen the MAR assumption for item non-response on the smoking variables, and they describe the study base in Tables 1a/1b — groundwork for planned related studies (smoking initiation and cessation prediction models, pseudo-cohort analyses) that will draw on the same harmonized platform. Household income is the one classic non-response auxiliary not yet included: cchsflow harmonizes it only through 2017–18, and extending it to 2019–20 is a pending upstream task. +Variables that carry structural missingness (NA(a)/NA(c)) may *be imputed* but do not *serve as predictors*: mice excludes rows with missing predictor values from each conditional model, so a structural-NA predictor (for example, time since quitting, which is not applicable for current and never-smokers) would make most item non-response in other variables unpredictable. The predictor matrix therefore restricts prediction to the complete design, demographic, and auxiliary variables. A refinement using universe-nested smoking predictors is a documented follow-up. + Including the sampling weight and survey cycle as *predictors* makes the imputation design-consistent without requiring a weighted imputation algorithm — a second improvement over the DemPoRTv2 implementation, which omits design variables. Age, sex, and cycle jointly encode the cohort dimension of the APC analysis model, preserving congeniality with the Stage 8 models. ### Derived variables @@ -65,7 +67,7 @@ Derived smoking variables (`pack_years_der`, and any variable with a cchsflow de ### Imputed data storage and downstream use -All *m* completed datasets are retained (long format keyed by imputation number). +All *m* completed datasets are retained (a list of completed data frames). - **Table 1b** statistics are computed within each completed dataset and averaged across imputations. - **APC models (Stage 8)** are initially fitted on the first completed dataset, with a documented upgrade path to fitting per imputation and pooling coefficients by Rubin's rules. FMI from the pooled fits (Phase 2.3) determines whether the pooled analysis is required for the primary results. @@ -143,11 +145,11 @@ Fit the APC models on (a) cycles where all key variables are observed (2001–20 ## Implementation checklist -- [ ] Assign `imputation-predictor` roles in `worksheets/cshm-variables.csv` (prespecified set above) -- [ ] Rewrite `impute_data()` with the `where` matrix, factor-level protection, tag restoration on write-back, and `loggedEvents` surfacing (cshm-dev issue #3) -- [ ] Extend the descriptive engine with a `weight_var` argument (weighted %, weighted median/IQR); Tables 1a/1b present unweighted n with weighted statistics -- [ ] Recompute derived variables from imputed feeders via the cchsflow derivation chain -- [ ] Retain all *m* completed datasets; average Table 1b across imputations +- [x] Assign `imputation-predictor` roles in `worksheets/cshm-variables.csv` (prespecified set above) +- [x] Rewrite `impute_data()` with the `where` matrix, factor-level protection, tag restoration on write-back, `loggedEvents` surfacing, and the structural-missingness predictor matrix (cshm-dev issue #3) +- [x] Extend the descriptive engine with a `weight_var` argument (weighted %, weighted median/IQR); Tables 1a/1b present unweighted n with weighted statistics +- [x] Recompute derived variables from imputed feeders via the cchsflow derivation chain (pack_years_der; worksheet-driven generalization pending) +- [x] Retain all *m* completed datasets; average Table 1b across imputations - [ ] Add an imputation-diagnostics target and Quarto report (trace plots, density plots, FMI table) - [ ] Golden-record and tagged-NA preservation tests (cshm-dev issue #4) - [ ] Delta-adjustment sensitivity analysis for cycle-absent variables; RDC cross-check at the Master-file stage diff --git a/docs/results/table-1.qmd b/docs/results/table-1.qmd index 6005d62..ddf95c0 100644 --- a/docs/results/table-1.qmd +++ b/docs/results/table-1.qmd @@ -31,15 +31,16 @@ variables_sheet <- targets::tar_read("variables_sheet") variable_details_sheet <- targets::tar_read("variable_details_sheet") table_1a_data <- targets::tar_read("table_1a_data") -predictor_vars <- select_vars_by_role("predictor", variables_sheet) +# Table rows: the table1 role (includes auxiliary covariates that are not APC predictors) +table1_vars <- select_vars_by_role("table1", variables_sheet) create_descriptive_table( descriptive_data = table_1a_data, variables_sheet = variables_sheet, variable_details_sheet = variable_details_sheet, - variables = predictor_vars, + variables = table1_vars, column_stratifier = survey_var(cfg, "sex"), - sections_order = c("Sociodemographics", "Health behaviour") + sections_order = c("Sociodemographics", "Health status", "Health behaviour") ) ``` @@ -52,9 +53,9 @@ create_descriptive_table( descriptive_data = table_1b_data, variables_sheet = variables_sheet, variable_details_sheet = variable_details_sheet, - variables = predictor_vars, + variables = table1_vars, column_stratifier = survey_var(cfg, "sex"), - sections_order = c("Sociodemographics", "Health behaviour") + sections_order = c("Sociodemographics", "Health status", "Health behaviour") ) ``` @@ -70,10 +71,39 @@ create_cycle_specific_descriptive_table( study_data = cleaned_data, variables_sheet = variables_sheet, variable_details_sheet = variable_details_sheet, - variables = predictor_vars, + variables = table1_vars, cycle_col = survey_var(cfg, "cycle"), cycle_labels = cfg$cycle_labels, column_stratifier = survey_var(cfg, "sex"), - sections_order = c("Sociodemographics", "Health behaviour") + weight_var = survey_var(cfg, "weight"), + sections_order = c("Sociodemographics", "Health status", "Health behaviour") +) +``` + +# Appendix — Fully unweighted Tables 1a and 1b + +Unweighted percentages and quantiles for comparison with the survey-weighted +presentation above (protocol §3.4.1). + +```{r table-1-unweighted} +table_1a_unweighted <- targets::tar_read("table_1a_unweighted_data") +table_1b_unweighted <- targets::tar_read("table_1b_unweighted_data") + +create_descriptive_table( + descriptive_data = table_1a_unweighted, + variables_sheet = variables_sheet, + variable_details_sheet = variable_details_sheet, + variables = table1_vars, + column_stratifier = survey_var(cfg, "sex"), + sections_order = c("Sociodemographics", "Health status", "Health behaviour") +) + +create_descriptive_table( + descriptive_data = table_1b_unweighted, + variables_sheet = variables_sheet, + variable_details_sheet = variable_details_sheet, + variables = table1_vars, + column_stratifier = survey_var(cfg, "sex"), + sections_order = c("Sociodemographics", "Health status", "Health behaviour") ) ``` diff --git a/docs/workflow/5-imputation.qmd b/docs/workflow/5-imputation.qmd index b867e49..2e84c0a 100644 --- a/docs/workflow/5-imputation.qmd +++ b/docs/workflow/5-imputation.qmd @@ -21,7 +21,7 @@ Configuration parameters: | Parameter | Value | Meaning | |-----------|-------|---------| | `imputation_m` | 5 | Number of imputed datasets (1 in `dev`/`draft`) | -| `imputation_maxit` | 5 | MICE iterations per imputation | +| `imputation_maxit` | 5 | MICE iterations per imputation (1 in `dev`/`draft`) | ## Code diff --git a/tests/testthat/setup.R b/tests/testthat/setup.R index 8c6c0ce..fad3ec3 100644 --- a/tests/testthat/setup.R +++ b/tests/testthat/setup.R @@ -1,3 +1,7 @@ +# cchsflow must be attached (its derivation functions resolve unqualified +# dplyr/rlang helpers via Depends) — mirrors tar_option_set in _targets.R +suppressPackageStartupMessages(library(cchsflow)) + # setup.R — loaded automatically by testthat before all test files # Sources all R functions from the project root, making them available in tests. diff --git a/tests/testthat/test-descriptive-tables.R b/tests/testthat/test-descriptive-tables.R index 01dc7d4..9d99aaa 100644 --- a/tests/testthat/test-descriptive-tables.R +++ b/tests/testthat/test-descriptive-tables.R @@ -255,6 +255,63 @@ test_that("get_cshm_desc_data returns rows only for predictor variables", { returned_vars <- unique(result$variable) expect_true("DHH_AGE" %in% returned_vars) expect_true("SMKDSTY" %in% returned_vars) - expect_false("WTS_M" %in% returned_vars) + # DHH_SEX is the column stratifier, never a row variable + expect_false("DHH_SEX" %in% returned_vars) expect_false("SurveyCycle" %in% returned_vars) }) + +# ---- Weighted statistics and MI averaging (protocol v0.3.1) ----------------- + +test_that("weighted percent and weighted median use the survey weights", { + vs <- make_variables_sheet() + vds <- make_variable_details_sheet() + d <- data.frame( + DHH_SEX = factor(rep("1", 4), levels = c("1", "2")), + DHH_AGE = c(10, 20, 30, 40), + SMKDSTY = factor(c("1", "1", "2", "2"), levels = c("1", "2")), + WTS_M = c(1, 1, 1, 7) + ) + out <- get_descriptive_data( + d, vs, vds, c("DHH_AGE", "SMKDSTY"), + stratify_config = list(all = list("DHH_SEX")), + weight_var = "WTS_M" + ) + cat2 <- out[out$variable == "SMKDSTY" & out$cat == "2" & out$groupByValue_1 == "1", ] + expect_equal(cat2$percent, 0.5) + expect_equal(cat2$wtd_percent, 0.8) # (1 + 7) / 10 + cont <- out[out$variable == "DHH_AGE" & is.na(out$cat) & out$groupByValue_1 == "1", ] + expect_equal(cont$median, 25) # unweighted type-7 + expect_gt(cont$wtd_median, 30) # pulled toward upweighted 40 + expect_error( + get_descriptive_data(d, vs, vds, "DHH_AGE", + list(all = list("DHH_SEX")), weight_var = "nope"), + "not found" + ) +}) + +test_that("get_cshm_desc_data_mi averages estimates across imputations", { + vs <- make_variables_sheet() + vds <- make_variable_details_sheet() + base <- data.frame( + DHH_SEX = factor(rep("1", 10), levels = c("1", "2")), + DHH_AGE = as.numeric(1:10) + ) + d1 <- base; d1$SMKDSTY <- factor(c(rep("1", 4), rep("2", 6)), levels = c("1", "2")) + d2 <- base; d2$SMKDSTY <- factor(c(rep("1", 6), rep("2", 4)), levels = c("1", "2")) + d2$DHH_AGE <- as.numeric(3:12) + + out <- get_cshm_desc_data_mi(list(datasets = list(d1, d2), m = 2L), vs, vds) + daily <- out[out$variable == "SMKDSTY" & out$cat == "1" & out$groupByValue_1 == "1", ] + expect_equal(daily$percent, 0.5) # mean of 0.4 and 0.6 + expect_equal(daily$n, 5) # mean of 4 and 6 + age <- out[out$variable == "DHH_AGE" & is.na(out$cat) & out$groupByValue_1 == "1", ] + expect_equal(age$min, 1) # min of minima + expect_equal(age$max, 12) # max of maxima + expect_equal(age$median, 6.5) # mean of 5.5 and 7.5 + + # m = 1 reduces to the single-dataset call + expect_identical( + get_cshm_desc_data_mi(list(datasets = list(d1), m = 1L), vs, vds), + get_cshm_desc_data(d1, vs, vds) + ) +}) diff --git a/tests/testthat/test-imputation.R b/tests/testthat/test-imputation.R index 7793188..484ee4e 100644 --- a/tests/testthat/test-imputation.R +++ b/tests/testthat/test-imputation.R @@ -129,3 +129,49 @@ test_that("weighted_quantile interpolates and matches type-7 median under equal expect_equal(weighted_quantile(numeric(0), numeric(0), 0.5), NA_real_) expect_equal(weighted_quantile(7, 3, c(0.25, 0.75)), c(7, 7)) }) + +test_that("impute_data stops when MICE declines to impute a flagged cell", { + d <- make_imputation_test_data() + d$flat <- 1 # constant: mice sets method "" + d$flat[5] <- haven::tagged_na("b") + vs <- rbind(make_imputation_variables_sheet(), + data.frame(variable = "flat", variableType = "Continuous", + role = "imputation-predictor", source = "both")) + cfg <- list(imputation_m = 1, imputation_maxit = 1) + # Either guard may fire first: the declined-cell write-back stop or the + # scoped chain-mean degeneracy stop — both protect the same contract. + expect_error(suppressWarnings(impute_data(d, vs, cfg)), "unimputed|degenerate fit") +}) + +test_that("structural-NA variables are excluded as predictors, not as targets", { + d <- make_imputation_test_data() + vs <- make_imputation_variables_sheet() + cfg <- list(imputation_m = 1, imputation_maxit = 1) + msgs <- capture_messages(suppressWarnings(impute_data(d, vs, cfg))) + excl <- msgs[grepl("Excluded as predictors", msgs)] + expect_length(excl, 1) + # years_quit (NA(a)/(c)) and status (NA(a) level) carry structural missingness + expect_match(excl, "years_quit") + expect_match(excl, "status") + expect_false(grepl("\\bage\\b", excl)) # complete variables stay predictors +}) + +test_that("pack_years_der is recomputed from imputed feeders", { + skip_if_not(all(c("calculate_pack_years") %in% getNamespaceExports("cchsflow"))) + n <- 12 + d <- data.frame( + SMKDSTY_original = factor(rep("6", n), levels = c("1", "4", "6")), + DHHGAGE_cont = rep(50, n), + age_start_smoking = rep(NA_real_, n), + cigs_per_day = rep(NA_real_, n), + time_quit_smoking = rep(NA_real_, n), + SMK_05B = rep(NA_real_, n), + SMK_05C = rep(NA_real_, n), + age_first_cigarette = rep(NA_real_, n), + smoked_100_lifetime = rep(2, n), + pack_years_der = rep(99, n) # stale value to be overwritten + ) + out <- recompute_derived(d) + # never-smokers: pack-years recomputed to 0, replacing the stale 99 + expect_true(all(out$pack_years_der == 0)) +}) From 0c0b5abdbdb4e3e10b8afe107fee27661d261689 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Wed, 10 Jun 2026 17:16:43 -0400 Subject: [PATCH 5/6] Protocol v0.3.2: record the NA(c) and predictor-matrix specification changes The PR #6 review fixes changed prespecified methods in Appendix D after the v0.3.1 stamp; record them in the version history rather than edit silently: NA(c) is not imputed in the base pipeline (extending the where matrix is the documented mechanism, gated on Phase 4), and structural-missingness variables do not serve as imputation predictors. Correct the 0.3.0/0.3.1 entry dates (2026-06-10, not 06-11). --- docs/protocol/full-protocol.qmd | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/protocol/full-protocol.qmd b/docs/protocol/full-protocol.qmd index 15e38e6..cf40759 100644 --- a/docs/protocol/full-protocol.qmd +++ b/docs/protocol/full-protocol.qmd @@ -2,14 +2,17 @@ title: "A Canadian Smoking Histories Model: a study protocol to generate smoking cohorts from 1940 and project to 2050" status: "Draft" version-summary: - date: "2026-06-11" - version: "0.3.1" + date: "2026-06-10" + version: "0.3.2" version-history: + - version: "0.3.2" + date: "2026-06-10" + description: "Imputation specification refined after implementation review (Appendix D): NA(c) cycle-level absence is not imputed in the base pipeline (the taxonomy previously implied it was) — extending the where matrix to NA(c) is the documented mechanism, gated on the Phase 4 delta-adjustment evaluation with the RDC Master files as the external check. Added the predictor-matrix specification: variables with structural missingness may be imputed but do not serve as predictors (mice excludes rows with missing predictors, so structural-NA predictors made item non-response in other variables unpredictable). Derived-variable recomputation (pack-years) implemented as specified." - version: "0.3.1" - date: "2026-06-11" + date: "2026-06-10" description: "Expanded the imputation-predictor and Table 1 variable set with auxiliary covariates already harmonized to 2019-20 (marital status, alcohol, BMI, self-rated general and mental health, life stress, community belonging, chronic conditions, energy expenditure), supporting both imputation quality and the study-base description for planned related studies. Income to 2019-20 noted as a pending cchsflow extension." - version: "0.3.0" - date: "2026-06-11" + date: "2026-06-10" description: "Missing data and imputation plan: where-matrix MICE restricted to item non-response (m = 5), survey design variables in the imputation model, derived variables recomputed from imputed feeders. Tables 1a/1b specified as unweighted n with survey-weighted statistics in a single table. Added Appendix D (detailed imputation plan and four-phase assessment, adapting the DemPoRTv2 framework). Documented PUMF cycle-level smoking variable gaps (2019-20, 2022) and the 2015-16 SMKG035 recode." - version: "0.2.0" date: "2026-03-09" From 82f2a71c6fe628f86786e71ed3b840ce07ddfab0 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Wed, 10 Jun 2026 17:22:45 -0400 Subject: [PATCH 6/6] Enforce protocol versioning: CI check + CLAUDE.md convention Any PR touching docs/protocol/ must bump version-summary.version in full-protocol.qmd and add a dated version-history entry. The GitHub Actions check compares the version against the PR base and fails on a silent edit; CLAUDE.md documents the convention for agent sessions. Same gate-as-pipeline-citizen pattern as the coverage validator. --- .github/workflows/protocol-version.yml | 48 ++++++++++++++++++++++++++ CLAUDE.md | 4 +++ 2 files changed, 52 insertions(+) create mode 100644 .github/workflows/protocol-version.yml diff --git a/.github/workflows/protocol-version.yml b/.github/workflows/protocol-version.yml new file mode 100644 index 0000000..f8fb2ec --- /dev/null +++ b/.github/workflows/protocol-version.yml @@ -0,0 +1,48 @@ +name: Protocol version check + +# The study protocol is a prespecified governance document: any change to +# docs/protocol/ must be recorded in the version history, not edited silently. +# This check fails a PR that touches protocol files without bumping +# version-summary.version in full-protocol.qmd. + +on: + pull_request: + paths: + - "docs/protocol/**" + +jobs: + version-bumped: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Require a version bump when protocol files change + env: + BASE_REF: ${{ github.base_ref }} + run: | + base="origin/${BASE_REF}" + git fetch origin "${BASE_REF}" --depth=1 + + changed=$(git diff --name-only "$base"...HEAD -- docs/protocol/) + echo "Protocol files changed:" + echo "$changed" + + get_version() { + git show "$1:docs/protocol/full-protocol.qmd" 2>/dev/null \ + | awk '/^version-summary:/{f=1; next} f && /version:/{gsub(/[" ]/, "", $2); print $2; exit}' + } + base_version=$(get_version "$base") + head_version=$(get_version HEAD) + echo "Base version: ${base_version:-} | Head version: ${head_version:-}" + + if [ -z "$head_version" ]; then + echo "::error::Could not parse version-summary.version from full-protocol.qmd" + exit 1 + fi + if [ "$base_version" = "$head_version" ]; then + echo "::error::docs/protocol/ changed but version-summary.version is still ${head_version}. Bump the version and add a version-history entry describing the change." + exit 1 + fi + echo "Version bumped: ${base_version:-} -> ${head_version}" diff --git a/CLAUDE.md b/CLAUDE.md index a70e2a8..702b842 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -148,6 +148,10 @@ Role vocabulary (single source of truth): [schemas/cshm-variables.yaml](schemas/ `haven::tagged_na()` throughout: **NA(a)** = not applicable · **NA(b)** = don't know/refused · **NA(c)** = not asked this cycle +## Protocol versioning + +The study protocol is prespecified: any change under `docs/protocol/` (including Appendix D) must bump `version-summary.version` in [docs/protocol/full-protocol.qmd](docs/protocol/full-protocol.qmd) and add a dated `version-history` entry describing the change — in the same commit. Enforced on PRs by `.github/workflows/protocol-version.yml`. + ## Code style - Follow tidyverse design principles; snake_case for all function and variable names