Add update_stats query for HOT and same-page update analysis

README.md

@@ -114,6 +114,36 @@ RubyPgExtras.diagnose
 
 Keep reading to learn about methods that `diagnose` uses under the hood.
 
+### `new_page_updates`
+
+This is a `diagnose` check, not a standalone query method. On PostgreSQL 16 and newer, it uses the [`update_stats`](#update_stats) breakdown to flag tables where a significant share of updates placed the new row version on another heap page instead of staying on the original page. Those tables are worth reviewing for page-space pressure, row growth, and whether a lower table `fillfactor` would help.
+
+By default, a table is reported when it has at least 10,000 cumulative updates and 20% or more of its updates are new-page updates. The report includes each table's new-page ratio, current `fillfactor`, and the percentage of same-page updates that were HOT. A low HOT-among-same-page value suggests indexed-column changes are also preventing HOT, so lowering `fillfactor` alone may not be enough.
+
+You can override the default thresholds with environment variables:
+
+```ruby
+ENV["PG_EXTRAS_NEW_PAGE_UPDATES_MIN_SAMPLE"] = "5000"
+ENV["PG_EXTRAS_NEW_PAGE_UPDATES_MAX_PERCENT"] = "15"
+```
+
+The underlying counters are cumulative, so compare their values over time rather than treating a single snapshot as definitive.
+
+### `low_hot_same_page`
+
+This is a `diagnose` check, not a standalone query method. On PostgreSQL 16 and newer, it uses the [`update_stats`](#update_stats) breakdown to flag tables where updates that stayed on the original heap page were almost never HOT. That usually means those updates modified indexed columns, so lowering `fillfactor` alone will not help.
+
+By default, a table is reported when it has at least 10,000 cumulative updates and fewer than 10% of its same-page updates were HOT. The report includes each table's HOT-among-same-page ratio, same-page and new-page ratios, and current `fillfactor`. Review which columns your application updates and which indexes cover them; removing or adjusting indexes on frequently updated columns (or avoiding updating those columns) can restore HOT updates.
+
+You can override the default thresholds with environment variables:
+
+```ruby
+ENV["PG_EXTRAS_LOW_HOT_SAME_PAGE_MIN_SAMPLE"] = "5000"
+ENV["PG_EXTRAS_LOW_HOT_SAME_PAGE_MIN_PERCENT"] = "5"
+```
+
+The underlying counters are cumulative, so compare their values over time rather than treating a single snapshot as definitive.
+
 ## Available methods
 
 ### `missing_fk_indexes`
@@ -725,6 +755,21 @@ RubyPgExtras.vacuum_io_stats
 
 This command surfaces cumulative I/O statistics for autovacuum-related VACUUM activity, based on the `pg_stat_io` view introduced in PostgreSQL 16 ([pg_stat_io documentation](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-IO-VIEW)). It shows how many blocks autovacuum workers have read and written, how many buffer evictions and ring-buffer reuses occurred, and when the statistics were last reset; this is useful for determining whether autovacuum is responsible for I/O spikes, as described in the pganalyze article on `pg_stat_io` ([Tracking cumulative I/O activity by autovacuum and manual VACUUMs](https://pganalyze.com/blog/pg-stat-io#tracking-cumulative-io-activity-by-autovacuum-and-manual-vacuums)). On PostgreSQL versions below 16 this method returns a single informational row indicating that the feature is unavailable.
 
+### `update_stats`
+
+```ruby
+
+RubyPgExtras.update_stats
+
+ table  | fillfactor | estimated_heap_bytes_per_live_row | total_updates | hot_updates | hot_pct | same_page_non_hot_updates | same_page_non_hot_pct | new_page_updates | new_page_pct | same_page_pct | hot_given_same_page_pct
+--------+------------+-----------------------------------+---------------+-------------+---------+---------------------------+-----------------------+------------------+--------------+---------------+-------------------------
+ users  |        100 |                               256 |       1250000 |      980000 |   78.40 |                     45000 |                  3.60 |           225000 |        18.00 |         82.00 |                   95.61
+ orders |         80 |                               128 |        450000 |      410000 |   91.11 |                     12000 |                  2.67 |            28000 |         6.22 |         93.78 |                   97.16
+ (truncated results for brevity)
+```
+
+This command breaks down table updates into HOT, same-page non-HOT, and new-page updates using `pg_stat_user_tables` columns including `n_tup_newpage_upd` ([pg_stat_all_tables documentation](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ALL-TABLES-VIEW)). HOT updates require that changed columns are not indexed and that the new row version fits on the same page ([HOT updates in PostgreSQL for better performance](https://www.cybertec-postgresql.com/en/hot-updates-in-postgresql-for-better-performance/), [Heap-Only Tuples](https://www.postgresql.org/docs/current/storage-hot.html)). High `same_page_non_hot_pct` usually points to updates of indexed columns, while high `new_page_pct` often means pages are too full and lowering `fillfactor` (then rewriting the table with `VACUUM FULL` or `CLUSTER`) may help. `estimated_heap_bytes_per_live_row` divides main-fork heap size by `pg_class.reltuples` when that estimate is positive; it reflects physical storage per estimated live row (including page overhead, fillfactor free space, and bloat) rather than logical tuple width, and is NULL until the table has been analyzed or vacuumed. Larger values often call for a lower `fillfactor`. These counters are cumulative and can be reset with PostgreSQL statistics-reset functions. On PostgreSQL versions below 16, where `n_tup_newpage_upd` is unavailable, the method returns a reduced breakdown of total, HOT, and non-HOT updates.
+
 ### `kill_all`
 
 ```ruby

lib/ruby-pg-extras.rb

@@ -28,7 +28,7 @@ module RubyPgExtras
     records_rank seq_scans table_index_scans table_indexes_size
     table_size total_index_size total_table_size
     unused_indexes duplicate_indexes vacuum_stats vacuum_progress vacuum_io_stats
-    analyze_progress
+    analyze_progress update_stats
     kill_all kill_pid
     pg_stat_statements_reset buffercache_stats
     buffercache_usage ssl_used connections
@@ -58,6 +58,8 @@ module RubyPgExtras
     vacuum_io_stats: {},
     vacuum_io_stats_legacy: {},
     analyze_progress: {},
+    update_stats: { schema: DEFAULT_SCHEMA },
+    update_stats_legacy: { schema: DEFAULT_SCHEMA },
     buffercache_stats: { limit: 10 },
     buffercache_usage: { limit: 20 },
     unused_indexes: { max_scans: 50, schema: DEFAULT_SCHEMA },
@@ -119,6 +121,15 @@ def self.run_query_base(query_name:, conn:, exec_method:, in_format:, args: {})
       end
     end
 
+    # The detailed update breakdown relies on n_tup_newpage_upd, available from PostgreSQL 16.
+    # Older versions fall back to the HOT/non-HOT breakdown in update_stats_legacy.
+    if query_name == :update_stats
+      server_version_num = conn.send(exec_method, "SHOW server_version_num").to_a[0].values[0].to_i
+      if server_version_num < 160000
+        query_name = :update_stats_legacy
+      end
+    end
+
     REQUIRED_ARGS.fetch(query_name) { [] }.each do |arg_name|
       if args[arg_name].nil?
         raise ArgumentError, "'#{arg_name}' is required"

lib/ruby_pg_extras/diagnose_data.rb

@@ -10,6 +10,10 @@ class DiagnoseData
     PG_EXTRAS_NULL_MIN_NULL_FRAC_PERCENT = 50 # 50%
     PG_EXTRAS_BLOAT_MIN_VALUE = 10
     PG_EXTRAS_OUTLIERS_MIN_EXEC_RATIO = 33 # 33%
+    PG_EXTRAS_NEW_PAGE_UPDATES_MAX_PERCENT = 20 # 20%
+    PG_EXTRAS_NEW_PAGE_UPDATES_MIN_SAMPLE = 10_000
+    PG_EXTRAS_LOW_HOT_SAME_PAGE_MIN_PERCENT = 10 # 10%
+    PG_EXTRAS_LOW_HOT_SAME_PAGE_MIN_SAMPLE = 10_000
 
     def self.call
       new.call
@@ -26,6 +30,8 @@ def call
         :unused_indexes,
         :null_indexes,
         :bloat,
+        :new_page_updates,
+        :low_hot_same_page,
         :duplicate_indexes,
       ].yield_self do |checks|
         extensions_data = query_module.extensions(in_format: :hash)
@@ -292,6 +298,120 @@ def duplicate_indexes
       end
     end
 
+    def new_page_updates
+      max_percent = ENV.fetch(
+        "PG_EXTRAS_NEW_PAGE_UPDATES_MAX_PERCENT",
+        PG_EXTRAS_NEW_PAGE_UPDATES_MAX_PERCENT,
+      ).to_f
+      min_sample = ENV.fetch(
+        "PG_EXTRAS_NEW_PAGE_UPDATES_MIN_SAMPLE",
+        PG_EXTRAS_NEW_PAGE_UPDATES_MIN_SAMPLE,
+      ).to_i
+
+      tables = query_module.update_stats(in_format: :hash)
+
+      if tables.any? && !tables.first.key?("new_page_pct")
+        return {
+          ok: true,
+          message: "New-page update analysis requires PostgreSQL 16 or newer.",
+        }
+      end
+
+      tables = tables.select do |table|
+        table.fetch("total_updates").to_i >= min_sample &&
+          table.fetch("new_page_pct").to_f >= max_percent
+      end
+
+      if tables.empty?
+        {
+          ok: true,
+          message: "No tables with a high new-page update ratio detected.",
+        }
+      else
+        table_details = tables.map do |table|
+          <<~DETAIL.strip
+            '#{table.fetch("table")}':
+              new-page updates: #{table.fetch("new_page_pct")}% (#{table.fetch("new_page_updates")} of #{table.fetch("total_updates")})
+              HOT among same-page updates: #{table.fetch("hot_given_same_page_pct")}%
+              fillfactor: #{table.fetch("fillfactor")}
+          DETAIL
+        end.join("\n\n")
+
+        {
+          ok: false,
+          message: <<~MESSAGE.strip,
+            High new-page update ratios detected:
+
+            #{table_details}
+
+            A high new-page ratio means many successor tuple versions were placed on another heap page and therefore could not be HOT. This commonly indicates insufficient reusable space on the original page. `n_tup_newpage_upd` records that placement directly; it does not identify the underlying reason or whether the update would otherwise have been HOT-eligible. Investigate page-space pressure, row growth, long-lived transactions, large update batches, and whether a lower table fillfactor is appropriate.
+
+            The HOT-among-same-page percentage provides additional context: a low value suggests indexed-column changes are preventing HOT on updates that did stay on the same page, so changing fillfactor alone may not resolve the issue.
+
+            These counters are cumulative; compare their deltas before and after a change.
+          MESSAGE
+        }
+      end
+    end
+
+    def low_hot_same_page
+      min_percent = ENV.fetch(
+        "PG_EXTRAS_LOW_HOT_SAME_PAGE_MIN_PERCENT",
+        PG_EXTRAS_LOW_HOT_SAME_PAGE_MIN_PERCENT,
+      ).to_f
+      min_sample = ENV.fetch(
+        "PG_EXTRAS_LOW_HOT_SAME_PAGE_MIN_SAMPLE",
+        PG_EXTRAS_LOW_HOT_SAME_PAGE_MIN_SAMPLE,
+      ).to_i
+
+      tables = query_module.update_stats(in_format: :hash)
+
+      if tables.any? && !tables.first.key?("hot_given_same_page_pct")
+        return {
+          ok: true,
+          message: "HOT-among-same-page update analysis requires PostgreSQL 16 or newer.",
+        }
+      end
+
+      tables = tables.select do |table|
+        hot_given_same_page_pct = table["hot_given_same_page_pct"]
+        next false if hot_given_same_page_pct.nil?
+
+        table.fetch("total_updates").to_i >= min_sample &&
+          hot_given_same_page_pct.to_f < min_percent
+      end
+
+      if tables.empty?
+        {
+          ok: true,
+          message: "No tables with a low HOT-among-same-page update ratio detected.",
+        }
+      else
+        table_details = tables.map do |table|
+          <<~DETAIL.strip
+            '#{table.fetch("table")}':
+              HOT among same-page updates: #{table.fetch("hot_given_same_page_pct")}%
+              same-page updates: #{table.fetch("same_page_pct")}%
+              new-page updates: #{table.fetch("new_page_pct")}%
+              fillfactor: #{table.fetch("fillfactor")}
+          DETAIL
+        end.join("\n\n")
+
+        {
+          ok: false,
+          message: <<~MESSAGE.strip,
+            Low HOT-among-same-page update ratios detected:
+
+            #{table_details}
+
+            A low HOT-among-same-page ratio means updates that stayed on the original heap page still could not be HOT. That usually means those updates modified indexed columns. Review which columns your application updates and which indexes cover them; removing or adjusting indexes on frequently updated columns (or avoiding updating those columns) can restore HOT updates and reduce index and vacuum overhead.
+
+            These counters are cumulative; compare their deltas before and after a change.
+          MESSAGE
+        }
+      end
+    end
+
     def outliers
       queries = query_module.outliers(in_format: :hash).select do |q|
         q.fetch("prop_exec_time").gsub("%", "").to_f >= PG_EXTRAS_OUTLIERS_MIN_EXEC_RATIO

lib/ruby_pg_extras/queries/update_stats.sql

@@ -0,0 +1,71 @@
+/* HOT, same-page non-HOT, and new-page update statistics (PostgreSQL 16+) */
+
+WITH table_stats AS (
+  SELECT
+    s.relid,
+    s.schemaname,
+    s.relname,
+    s.n_tup_upd,
+    s.n_tup_hot_upd,
+    s.n_tup_newpage_upd,
+    c.reltuples,
+    COALESCE(
+      (
+        SELECT option_value::integer
+        FROM pg_options_to_table(c.reloptions)
+        WHERE option_name = 'fillfactor'
+      ),
+      100
+    ) AS fillfactor
+  FROM pg_stat_user_tables s
+  INNER JOIN pg_class c ON c.oid = s.relid
+  WHERE s.schemaname = '%{schema}'
+)
+SELECT
+  relname AS table,
+  fillfactor,
+  CASE
+    WHEN reltuples > 0 THEN
+      ROUND(
+        pg_relation_size(relid)::numeric / reltuples
+      )::bigint
+  END AS estimated_heap_bytes_per_live_row,
+  n_tup_upd AS total_updates,
+  n_tup_hot_upd AS hot_updates,
+  ROUND(
+    100.0 * n_tup_hot_upd
+    / NULLIF(n_tup_upd, 0),
+    2
+  ) AS hot_pct,
+  n_tup_upd
+    - n_tup_hot_upd
+    - n_tup_newpage_upd
+    AS same_page_non_hot_updates,
+  ROUND(
+    100.0 * (
+      n_tup_upd
+      - n_tup_hot_upd
+      - n_tup_newpage_upd
+    )
+    / NULLIF(n_tup_upd, 0),
+    2
+  ) AS same_page_non_hot_pct,
+  n_tup_newpage_upd AS new_page_updates,
+  ROUND(
+    100.0 * n_tup_newpage_upd
+    / NULLIF(n_tup_upd, 0),
+    2
+  ) AS new_page_pct,
+  ROUND(
+    100.0 * (n_tup_upd - n_tup_newpage_upd)
+    / NULLIF(n_tup_upd, 0),
+    2
+  ) AS same_page_pct,
+  ROUND(
+    100.0 * n_tup_hot_upd
+    / NULLIF(n_tup_upd - n_tup_newpage_upd, 0),
+    2
+  ) AS hot_given_same_page_pct
+FROM table_stats
+WHERE n_tup_upd > 0
+ORDER BY n_tup_upd DESC;

lib/ruby_pg_extras/queries/update_stats_legacy.sql

@@ -0,0 +1,46 @@
+/* HOT and non-HOT update statistics (PostgreSQL 15 and older) */
+
+WITH table_stats AS (
+  SELECT
+    s.relid,
+    s.relname,
+    s.n_tup_upd,
+    s.n_tup_hot_upd,
+    c.reltuples,
+    COALESCE(
+      (
+        SELECT option_value::integer
+        FROM pg_options_to_table(c.reloptions)
+        WHERE option_name = 'fillfactor'
+      ),
+      100
+    ) AS fillfactor
+  FROM pg_stat_user_tables s
+  INNER JOIN pg_class c ON c.oid = s.relid
+  WHERE s.schemaname = '%{schema}'
+)
+SELECT
+  relname AS table,
+  fillfactor,
+  CASE
+    WHEN reltuples > 0 THEN
+      ROUND(
+        pg_relation_size(relid)::numeric / reltuples
+      )::bigint
+  END AS estimated_heap_bytes_per_live_row,
+  n_tup_upd AS total_updates,
+  n_tup_hot_upd AS hot_updates,
+  ROUND(
+    100.0 * n_tup_hot_upd
+    / NULLIF(n_tup_upd, 0),
+    2
+  ) AS hot_pct,
+  n_tup_upd - n_tup_hot_upd AS non_hot_updates,
+  ROUND(
+    100.0 * (n_tup_upd - n_tup_hot_upd)
+    / NULLIF(n_tup_upd, 0),
+    2
+  ) AS non_hot_pct
+FROM table_stats
+WHERE n_tup_upd > 0
+ORDER BY n_tup_upd DESC;