`compute_demand` tests tenure status against a value the field never takes, excluding all renters from the housing market

[nuochan]

Summary

In HouseholdPropertyFunction.compute_demand, the renting mask at macromodel/agents/households/func/property.py:261 tests the tenure status field against 0:

ind_renting = np.array(household_residence_tenure_status == 0)

The field never holds 0. The mask is therefore empty on every call, and renting households are excluded from the buy-versus-rent decision entirely. The owning mask on line 266 has a narrower version of the same problem.

The encoding

Tenure Status of the Main Residence carries HFCS HB0300 codes, imported at macro_data/readers/population_data/hfcs_reader.py:94. The codebase's own treatment of those codes is consistent everywhere except the two lines above — 3 is renting, {1, 2, 4} is owning, and -1 marks social housing:

Site	Test
macro_data/processing/synthetic_population/hfcs_household_tools.py:113-114	== 3 renting, .isin([1, 2, 4]) owning
macro_data/processing/synthetic_matching/matching_households_with_houses.py:319, 403, 466	== 3 renting, .isin([1, 2, 4]) owning
macromodel/agents/households/households.py:932, 947, 1331	== 3 renting, np.isin(..., [1, 2, 4]) owning
macromodel/country/country.py:1365	== 3 to select renters for the rental income tax base

The only writes to the field in the entire tree are:

Site	Value
macro_data/processing/synthetic_matching/matching_households_with_houses.py:370	-1 (renter moved into social housing)
macromodel/markets/housing_market/housing_market.py:468	3 (household rents a property)
macromodel/markets/housing_market/housing_market.py:494	1 (household buys a property)

So the reachable value set is {-1, 1, 2, 3, 4}. 0 is not in it, and no dataset is needed to establish that — it follows from the assignment sites alone.

The imported values agree. Across the 2010, 2014, 2017 and 2021 HFCS waves — all ten implicates of each — HB0300 never takes the value 0:

Wave	HB0300 values present	Zeros	NaN
2010	1, 2, 3, 4, 6	0	0
2014	1, 2, 3, 4	0	5
2017	1, 2, 3, 4	0	10
2021	1, 2, 3, 4	0	0

Two incidental observations from that sweep, both independent of the defect above but relevant to how a fix should be written. The 2010 wave carries an additional code 6, which falls into neither == 3 nor isin([1, 2, 4]). A small number of NaN values appear in the 2014 and 2017 waves, and NaN fails every comparison. Households in either category are silently dropped from all three masks rather than raising, so the suggested fix would benefit from an explicit branch — or an assertion that the observed code set is covered — instead of leaving unmatched households to fall through unnoticed.

Effects

ind_renting feeds ind_dec at property.py:273, which gates the whole decision:

ind_dec = ind_in_social_housing | ind_renting_not_staying | ind_owning_not_staying

1. Renting households never participate in the housing market. They cannot buy, and they cannot move to another rental. Renting is an absorbing state: once a household holds code 3, nothing in the model can change its tenure again.

2. Owning households with codes 2 and 4 are likewise frozen. Line 266 tests == 1 rather than isin([1, 2, 4]), so those owners never enter ind_owning_not_staying, never list their main residence for sale via households.py:783, and never move.

3. Market participation decays monotonically over a run. Social-housing households (-1) are matched correctly and do participate. When one of them wins a rental, housing_market.py:468 writes code 3, moving it permanently into the frozen set. There is no path back to -1, so the participating population drains toward the code-1 owners alone as a simulation proceeds.

4. Both sides of the market are biased, not just one. max_rent_willing_to_pay is populated only for households in ind_dec and flows into rental clearing at macromodel/markets/housing_market/func/clearing.py:131, so rental demand omits every sitting renter. households_hoping_to_move is ind_owning_not_staying, so sale supply omits every code-2/4 owner.

5. probability_stay_in_rented_property has no effect. At property.py:264, inside the ind_renting_not_staying block spanning lines 262-265, it is applied to the empty renting mask. The parameter can be set to any value in [0, 1] without changing model output, which means it also cannot be calibrated.

Reproduction

Effects 1, 2, 4 and 5 are observed rather than inferred; effect 3 follows from the write sites listed above, since 3 and 1 are the only values written anywhere under macromodel/ at runtime. macromodel/agents/households/func/property.py imports only numpy, pandas and scipy, so compute_demand can be exercised directly with no dataset. Both stay-probabilities are set to 0.0 below, meaning every renter and every owner wants to move — so any household still excluded is excluded structurally rather than by chance, and the noise variance is set to 0.0 to make the result deterministic.

Feeding one household of each tenure code:

Code	Meaning	Admitted to housing market	After the suggested fix
-1	social housing	yes	yes
1	owning outright	yes	yes
2	part owning	no	yes
3	renting	no	yes
4	free use	no	yes
	total	2 / 5	5 / 5
	flagged as hoping to move (sale supply)	1	3

max_rent_willing_to_pay comes back as NaN for codes 2, 3 and 4, so sitting renters submit no rental bid at all.

On the parameter: sweeping probability_stay_in_rented_property across 0.0, 0.25, 0.5, 0.75, 1.0 leaves every output array bit-identical. Applying the suggested fix and repeating the sweep makes the same parameter change the result, which confirms the parameter is genuinely inert under the current encoding rather than merely unused in this configuration.

These participation counts are exact and deterministic. We have not gone further and quantified how far prices, rents or tenure shares move once the excluded households re-enter the market, which would need a controlled comparison with seed variation.

Suggested fix

ind_renting = np.array(household_residence_tenure_status == 3)
ind_owning = np.isin(household_residence_tenure_status, [1, 2, 4])

This matches the convention used at every other site listed above.

Worth noting that the fix is behaviour-changing rather than cosmetic: it admits a substantial group of households into the housing market for the first time, so house prices, rents and tenure composition will all move relative to any existing baseline. Anyone with calibrated housing parameters will likely need to revisit them, and probability_stay_in_rented_property becomes live.

These lines predate the HFCS-code migration

We looked into whether the exclusion might be deliberate, and the history suggests it is not. The tenure field originally used a binary encoding in which 0 meant renting and 1 meant owning. Under that scheme lines 261 and 266 were correct, and git blame dates them to 28bbc942 (2024-05-17), while the binary scheme was still in force.

Commit d11bc5f (2025-05-15) migrated the model to preserve the raw HFCS codes instead, and rewrote the comparisons at every other call site:

File	Before	After
macro_data/processing/synthetic_population/hfcs_household_tools.py	== 0 / == 1	== 3 / .isin([1, 2, 4])
macromodel/agents/households/households.py	== 0 / == 1	== 3 / np.isin(..., [1, 2, 4])
macromodel/country/country.py	== 0	== 3
macro_data/processing/synthetic_matching/matching_households_with_houses.py	== 1	.isin([1, 2, 4])
macromodel/agents/households/func/property.py	== 0 / == 1	not modified

That commit does not touch func/property.py. The two lines are the only remaining consumers of the retired binary encoding, which is why they now compare against a value the field can no longer hold.

The same commit introduced the comment at hfcs_household_tools.py:111-112 — "This maintains codes used in HFCS. 1, 2 and 4 are owning, part owning or free use. 3 is renting" — recording the new contract explicitly. The docstring above it at line 93-95 still describes the retired binary mapping, and may be worth updating alongside any fix.

[t-youngman]

Thanks, this is a really good spot and could be why the housing market hasn't been firing. As you've noticed originally the model used different tenure codes to the originals in HFCS which got quite confusing, so this probably originates from there. Do you want to submit a PR making the fix so you get credit for it? I'm happy to review and do some test runs

[nuochan]

@t-youngman Happy to open a PR for this one — thank you for the offer.

One thing worth flagging first, since you mentioned test runs. These two masks gate model behaviour rather than just test output: while they are wrong, no renting household ever enters the buy-versus-rent decision and two of the three ownership codes are invisible to it, so most of the housing path never executes — including the mortgage demand it feeds into the credit market and the property wealth that enters household balance sheets. That is consistent with your reading that this could be why the housing market has not been firing.

The corollary is that repairing it makes several further defects in that path start to matter. They are unobservable rather than absent today, so a run afterwards may look worse before it looks right. Rather than restate them here, they are written up with traces and suggested fixes on the uvic-sesit fork's tracker: the sale-listing gate is uvic-sesit#15, three property columns overwritten at construction uvic-sesit#17, the rental vacancy detector uvic-sesit#18, the matching objective's affordability guard uvic-sesit#25, and a listing being consumed before settlement uvic-sesit#26. The rent-setting path has more of the same kind in uvic-sesit#22, uvic-sesit#23, uvic-sesit#24 and uvic-sesit#28, which only become reachable once rentals complete.

Those were found against that fork, but the files they cite are byte-identical to this repository at 76ac000 — I checked — so they read directly against main here. Happy to re-file any of them on this tracker if that is more useful than a link.