Zoho CRM SQL templates

README.md

@@ -14,6 +14,7 @@ Each integration ships both dialects:
 | [`shopify/`](shopify/) | sales over time, product sales over time, SKU cost per day, sales by product |
 | [`amazon-seller-central/`](amazon-seller-central/) | sales over time, product sales over time, settlement ledger, ASIN profitability, traffic and conversion, inventory health |
 | [`amazon-vendor-central/`](amazon-vendor-central/) | sales over time, sales by ASIN, inventory health, forecast vs actuals |
+| [`zoho-crm/`](zoho-crm/) | deal pipeline, deal flow by month, rep activity, account 360 |
 
 The rest of the library is at [weld.app/templates](https://weld.app/templates).
 

zoho-crm/README.md

@@ -0,0 +1,138 @@
+# Zoho CRM templates
+
+Pipeline, activity and account reporting from Weld's Zoho CRM connector. BigQuery,
+org reporting currency.
+
+| Model | Answers | Grain |
+|---|---|---|
+| [`weld/core/deal_pipeline.sql`](weld/core/deal_pipeline.sql) · [dbt](dbt/models/core/core_zoho_crm__deal_pipeline.sql) | What is open, won and lost, and whose it is | deal |
+| [`weld/core/deal_flow_by_month.sql`](weld/core/deal_flow_by_month.sql) · [dbt](dbt/models/core/core_zoho_crm__deal_flow_by_month.sql) | Created / won / lost and win rate over time | month × owner |
+| [`weld/core/rep_activity.sql`](weld/core/rep_activity.sql) · [dbt](dbt/models/core/core_zoho_crm__rep_activity.sql) | How much each rep is doing | day × owner |
+| [`weld/core/account_360.sql`](weld/core/account_360.sql) · [dbt](dbt/models/core/core_zoho_crm__account_360.sql) | People, pipeline and last touch per account | account |
+
+## Read this first — what this connector cannot do
+
+The Zoho CRM connector syncs 12 modules as flat tables. Three limits change what is
+worth building, and all three are properties of the connector, not of these models:
+
+**1. Activities have no parent record.** Zoho's Calls, Events and Tasks modules each
+carry `What_Id` / `Who_Id` — the deal, account, contact or lead the activity belongs
+to. Neither field is synced. `owner_id` is the only foreign key on all three
+streams, so activity can be counted **per rep and never per deal**. There is no
+honest "activities per won deal" from this connector alone.
+
+The exception is **Notes**, whose `Parent_Id` *is* synced. That single column is why
+`account_360` can report a last-touch date at all.
+
+**2. There is no won/lost flag.** Zoho ships no `is_won` or `is_closed` boolean, no
+probability and no forecast category. `deal_pipeline` derives the outcome from the
+stage *string*. Zoho's own defaults are covered — including `Closed-Lost to
+Competition`, which is why the match is `LIKE '%lost%'` and not an equality list —
+but a custom stage called `Contract Signed` or `Churned` will read as open pipeline
+until you add it. Run
+[`assert_deal_stages_are_classified`](dbt/tests/assert_deal_stages_are_classified.sql);
+this is the one edit almost every org has to make.
+
+**3. There is no history.** Weld's Zoho CRM connector does not support history
+tables, and the deal stream carries no stage-change audit. Nothing here can tell
+you what the pipeline looked like last month, how long a deal sat in Negotiation,
+or when a deal actually closed — `closing_date` is the *expected* close date and
+Zoho does not clear it on close. **If stage velocity or a pipeline trend matters,
+materialise `deal_pipeline` on a daily schedule and keep the runs.** That is the
+only route to history, and it only starts working from the day you set it up.
+
+Two smaller ones worth knowing:
+
+- **`call_duration` is a string.** Zoho's API reference documents it as `hh:mm`; the
+  CRM UI shows `mm:ss` for short calls, and the stream carries no
+  `Call_Duration_in_seconds` column to settle it. `staging/call.sql` parses the
+  documented `hh:mm`. Reconcile against one call of known length before reporting on
+  call time.
+- **Deletes are not captured.** Zoho's records API does not report deletions, so a
+  deleted record stops receiving updates and stays in your warehouse. Run a ReSync
+  on the affected table when the destination has to match Zoho exactly.
+
+## Layers
+
+```
+raw.zoho_crm.*          ELT output, untouched
+  ↓
+weld/staging/*.sql      9 thin wrappers: cast, rename, blanks to NULL
+  ↓
+weld/core/*.sql         the reports - all the business logic lives here
+  ↓
+weld/analytics/*.sql    BI-facing contracts, deliberately thin
+```
+
+**Staging** casts, renames and normalises, and nothing else. It also unpicks Zoho's
+lookup flattening: a lookup field arrives as an `_id` / `_name` / `_email` triple,
+so `Account_Name` on a deal becomes `account_name_id` — which is the account's id,
+despite the name. Staging renames it to `account_id` and drops the denormalised
+label, so a renamed rep or account does not leave stale copies of its old name
+scattered across every module.
+
+**Core** holds the stage classification, the date spines and the joins. This is the
+layer worth reviewing. `deal_pipeline` is deal-grain and the other three read it.
+
+**Analytics** exists for indirection, not transformation. Bind dashboards and
+reverse-ETL syncs to `analytics.zoho_crm.deal_pipeline` so core stays free to be
+renamed or re-grained. All four are `SELECT *` — that is the point, not an oversight.
+
+With [GitHub Sync](https://weld.app/docs/transformations/github-sync) a push deploys
+all of them and Weld resolves the dependency order. Weld references map to folder
+paths, so `{{staging.zoho_crm.deal}}` expects the model in a `staging > zoho_crm`
+folder — mirror that structure in your synced repo, or adjust the refs.
+
+## Required tables
+
+| Model | Raw tables |
+|---|---|
+| deal pipeline | `deal`, `account`, `user` |
+| deal flow by month | the same |
+| rep activity | `call`, `event`, `task`, `user` |
+| account 360 | `account`, `contact`, `deal`, `user`, `note` |
+
+**Sync the `user` stream even if you do not think you need it.** It is the only
+dimension the other modules can join to, and without it every report groups by a
+NULL rep name. [`assert_owner_ids_resolve`](dbt/tests/assert_owner_ids_resolve.sql)
+is there to catch exactly that.
+
+Three synced modules are deliberately not modelled, because nothing in the schema
+joins to them: **campaign** (no deal or lead key, and no budget or response
+metrics), and **product** and **price_book** (no line items on deals, and no key
+between the two).
+
+## What deal_pipeline returns
+
+Grain is one row per deal. `stage_status` is `Open`, `Won` or `Lost`, with
+`is_open` / `is_won` / `is_lost` alongside it, and `amount` split into
+`pipeline_amount`, `won_amount` and `lost_amount` so BI can sum a column instead of
+repeating the CASE.
+
+`age_days` is time-to-close for a decided deal and time-in-pipeline for an open
+one. `is_overdue` flags open deals whose expected close date has already passed —
+the cheapest pipeline-hygiene number available, and usually the first thing a sales
+lead asks for. Account (`account_name`, `industry`) and owner (`owner_name`,
+`owner_email`, `owner_role`, `owner_is_active`) are joined on. A deal owned by a
+deactivated rep is unmanaged pipeline; `owner_is_active` is how you find it.
+
+## Notes
+
+- **`win_rate` excludes open deals from the denominator.** Counting them as
+  not-yet-won drags every current month down and makes the trend look like a
+  collapse. Both a count-based and a value-based rate are returned.
+- **`deal_flow_by_month` books wins on `closing_date`**, because no real close
+  timestamp is synced. `modified_time` would be worse: it moves every time anyone
+  edits the record, so a note added in August would move a June win.
+- **Both time-series models are built on a spine**, so a rep with a quiet month
+  still returns a row of zeros instead of vanishing from the series and letting a
+  BI line chart interpolate straight over the gap.
+- **Multi-org ready.** Every staging model emits a `zoho_org` label and it is part
+  of every join key. To add an org, `UNION ALL` a second block in each staging model
+  with a different label — and change nothing else.
+- **Treat every model as unverified until you have reconciled it.** Open pipeline
+  against Zoho's own Deals view is the check that matters; if it disagrees, the
+  stage classification is where to look first.
+
+[Zoho CRM connector docs](https://weld.app/docs/applications/zoho-crm) ·
+[Full walkthrough](https://weld.app/blog/zoho-crm-connector-sql-reports)

zoho-crm/dbt/README.md

@@ -0,0 +1,56 @@
+# Zoho CRM dbt models
+
+Drop-in model files. Not a runnable project — no `dbt_project.yml`, not run in CI.
+
+```
+models/staging/sources.yml                        one source, one Zoho org
+models/staging/stg_zoho_crm__*.sql                9 thin wrappers over the raw tables
+models/core/core_zoho_crm__deal_pipeline.sql      every deal, classified - the business logic
+models/core/core_zoho_crm__deal_flow_by_month.sql created / won / lost per month per rep
+models/core/core_zoho_crm__rep_activity.sql       calls, meetings, tasks per rep per day
+models/core/core_zoho_crm__account_360.sql        account grain: people, pipeline, notes
+models/analytics/analytics__zoho_crm_*.sql        thin BI-facing contracts
+tests/assert_deal_stages_are_classified.sql       closed stages hiding in open pipeline
+tests/assert_owner_ids_resolve.sql                deals whose owner is not in `user`
+tests/assert_call_duration_parses.sql             duration strings the regex rejected
+tests/assert_no_duplicate_deals.sql               one row per deal
+```
+
+Four layers, same as the Weld side: raw → staging → core → analytics. The analytics
+models are `SELECT *` by design — dashboards bind to them so core stays free to
+change.
+
+**These are the same models as [`../weld/`](../weld/), differing only in ref
+syntax.** They are generated from the Weld versions, so the two dialects cannot
+silently disagree about how a deal is classified as won.
+
+## You supply
+
+**Your schema.** Change `schema:` in `sources.yml` to wherever your loader lands
+the Zoho tables.
+
+**Your closed stages.** Zoho CRM has no `is_won` or `is_closed` field, so
+`core_zoho_crm__deal_pipeline` derives won and lost from the stage *string*
+(`LIKE '%won%'` / `'%lost%'`). That covers Zoho's defaults, including
+`Closed-Lost to Competition`. It does not cover a custom stage named
+`Contract Signed`, `Churned` or `Dead`. Run
+`tests/assert_deal_stages_are_classified.sql` and add whatever it surfaces to the
+`CASE` in that model — this is the one edit almost every org has to make.
+
+No vars to set, no packages required — the tests are plain `SELECT`s, so
+`dbt_utils` is not needed.
+
+## Notes
+
+- **BigQuery.** `GENERATE_DATE_ARRAY`, `COUNTIF`, `SAFE_DIVIDE`, `TIMESTAMP_DIFF`
+  and `REGEXP_CONTAINS` all need swapping for other engines.
+- **`core_zoho_crm__deal_pipeline` is materialized as a table.** The three other
+  core models plus its own analytics contract all read it; as a view the warehouse
+  re-scans staging four times.
+- **Plain SQL, no macros.** Multi-org works by `UNION ALL` in the staging models:
+  add a source block per org, union the blocks with different `zoho_org` labels,
+  and change nothing else. `zoho_org` is already part of every join key.
+- **Not run end to end.** Reconcile open pipeline against Zoho's own Deals view
+  before relying on the output.
+
+[Full walkthrough](https://weld.app/blog/zoho-crm-connector-sql-reports)

zoho-crm/dbt/models/analytics/analytics__zoho_crm_account_360.sql

@@ -0,0 +1,11 @@
+-- analytics__zoho_crm_account_360
+-- BI-facing contract over the core model. See analytics/deal_pipeline.sql for why
+-- this layer exists even when it is a passthrough.
+--
+-- This is the natural source for a reverse-ETL sync back into Zoho: writing
+-- open_pipeline_amount or is_stale_with_open_pipeline onto the Account record puts
+-- the warehouse's view in front of the reps who need it.
+
+{{ config(materialized='view') }}
+
+SELECT * FROM {{ ref('core_zoho_crm__account_360') }}

zoho-crm/dbt/models/analytics/analytics__zoho_crm_deal_flow_by_month.sql

@@ -0,0 +1,7 @@
+-- analytics__zoho_crm_deal_flow_by_month
+-- BI-facing contract over the core model. See analytics/deal_pipeline.sql for why
+-- this layer exists even when it is a passthrough.
+
+{{ config(materialized='view') }}
+
+SELECT * FROM {{ ref('core_zoho_crm__deal_flow_by_month') }}

zoho-crm/dbt/models/analytics/analytics__zoho_crm_deal_pipeline.sql

@@ -0,0 +1,12 @@
+-- analytics__zoho_crm_deal_pipeline
+-- BI-facing contract over the core model. Dashboards, scheduled reports and
+-- reverse-ETL syncs bind HERE, never to core, so core stays free to be renamed,
+-- re-grained or split without breaking anything downstream.
+--
+-- A passthrough is the correct content for this layer. Put BI-specific shaping
+-- (renames for a semantic layer, row filters for a workspace) in this file rather
+-- than in core.
+
+{{ config(materialized='view') }}
+
+SELECT * FROM {{ ref('core_zoho_crm__deal_pipeline') }}

zoho-crm/dbt/models/analytics/analytics__zoho_crm_rep_activity.sql

@@ -0,0 +1,11 @@
+-- analytics__zoho_crm_rep_activity
+-- BI-facing contract over the core model. See analytics/deal_pipeline.sql for why
+-- this layer exists even when it is a passthrough.
+--
+-- Reminder for whoever binds a dashboard to this: activity here is per rep only.
+-- There is no deal or account key on Zoho's call, task and event streams in this
+-- connector, so do not label a tile "activity per opportunity".
+
+{{ config(materialized='view') }}
+
+SELECT * FROM {{ ref('core_zoho_crm__rep_activity') }}

zoho-crm/dbt/models/core/core_zoho_crm__account_360.sql

@@ -0,0 +1,132 @@
+-- zoho_crm_account_360 - one row per account, with its people, pipeline and notes.
+-- Grain: account. Currency: org reporting currency.
+-- Depends on: stg_zoho_crm__account, .contact, .user, .note,
+--             core_zoho_crm__deal_pipeline
+--
+-- The account-level view the rest of the schema can actually support: who works
+-- there, what is open, what has been won, and when anybody last wrote anything
+-- down. This is the model to reverse-ETL back into Zoho, or to join to product
+-- usage and billing data for a real customer view.
+--
+-- NOT IN HERE: calls, meetings and tasks. They carry no account key in this
+-- connector - see core_zoho_crm__rep_activity for why. last_note_at is the closest
+-- honest proxy for account engagement the connector allows, and it only reflects
+-- what reps bothered to write down.
+
+WITH contacts AS (
+    SELECT
+        zoho_org,
+        account_id,
+        COUNT(*)                                  AS contacts,
+        COUNTIF(email IS NOT NULL)                AS contacts_with_email,
+        MAX(created_time)                         AS last_contact_added_at
+    FROM {{ ref('stg_zoho_crm__contact') }}
+    WHERE account_id IS NOT NULL
+    GROUP BY 1, 2
+),
+
+deals AS (
+    SELECT
+        zoho_org,
+        account_id,
+        COUNT(*)                     AS deals_total,
+        COUNTIF(is_open)             AS deals_open,
+        COUNTIF(is_won)              AS deals_won,
+        COUNTIF(is_lost)             AS deals_lost,
+        COUNTIF(is_overdue)          AS deals_overdue,
+        SUM(pipeline_amount)         AS open_pipeline_amount,
+        SUM(won_amount)              AS won_amount,
+        SUM(lost_amount)             AS lost_amount,
+        MIN(CASE WHEN is_open THEN closing_date END) AS next_expected_close,
+        MAX(created_time)            AS last_deal_created_at
+    FROM {{ ref('core_zoho_crm__deal_pipeline') }}
+    WHERE account_id IS NOT NULL
+    GROUP BY 1, 2
+),
+
+-- Notes reach an account three ways: written on the account itself, on one of its
+-- deals, or on one of its contacts. Zoho ids are globally unique across modules,
+-- so one join per route resolves the parent without needing $se_module.
+note_targets AS (
+    SELECT a.zoho_org, a.account_id, a.account_id AS target_id
+    FROM {{ ref('stg_zoho_crm__account') }} a
+    UNION ALL
+    SELECT d.zoho_org, d.account_id, d.deal_id
+    FROM {{ ref('core_zoho_crm__deal_pipeline') }} d
+    WHERE d.account_id IS NOT NULL
+    UNION ALL
+    SELECT c.zoho_org, c.account_id, c.contact_id
+    FROM {{ ref('stg_zoho_crm__contact') }} c
+    WHERE c.account_id IS NOT NULL
+),
+
+notes AS (
+    SELECT
+        t.zoho_org,
+        t.account_id,
+        COUNT(*)               AS notes,
+        MAX(n.created_time)    AS last_note_at
+    FROM {{ ref('stg_zoho_crm__note') }} n
+    JOIN note_targets t
+      ON t.zoho_org  = n.zoho_org
+     AND t.target_id = n.parent_id
+    GROUP BY 1, 2
+)
+
+SELECT
+    a.zoho_org,
+    a.account_id,
+    a.account_name,
+    a.industry,
+    a.website,
+    a.phone,
+    a.created_time AS account_created_time,
+
+    a.owner_id,
+    u.full_name AS owner_name,
+    u.email     AS owner_email,
+    u.is_active AS owner_is_active,
+
+    COALESCE(c.contacts, 0)            AS contacts,
+    COALESCE(c.contacts_with_email, 0) AS contacts_with_email,
+
+    COALESCE(d.deals_total, 0)         AS deals_total,
+    COALESCE(d.deals_open, 0)          AS deals_open,
+    COALESCE(d.deals_won, 0)           AS deals_won,
+    COALESCE(d.deals_lost, 0)          AS deals_lost,
+    COALESCE(d.deals_overdue, 0)       AS deals_overdue,
+    COALESCE(d.open_pipeline_amount, 0) AS open_pipeline_amount,
+    COALESCE(d.won_amount, 0)          AS won_amount,
+    COALESCE(d.lost_amount, 0)         AS lost_amount,
+    d.next_expected_close,
+
+    COALESCE(n.notes, 0)               AS notes,
+    n.last_note_at,
+
+    -- Latest of anything datable on the account. Note the absence of calls and
+    -- meetings: this is "last recorded touch", not "last contact".
+    GREATEST(
+        COALESCE(n.last_note_at,             TIMESTAMP '1970-01-01'),
+        COALESCE(d.last_deal_created_at,     TIMESTAMP '1970-01-01'),
+        COALESCE(c.last_contact_added_at,    TIMESTAMP '1970-01-01'),
+        a.created_time
+    ) AS last_recorded_activity_at,
+
+    -- An account with open pipeline and nothing written on it in a quarter is the
+    -- report this model exists to produce.
+    -- Wrapped for the same reason as is_overdue above: if there is no note and no
+    -- account created_time, the comparison is NULL rather than FALSE, and the flag
+    -- stops being a clean boolean. Nothing to go on means not stale.
+    COALESCE(
+        COALESCE(d.deals_open, 0) > 0
+          AND COALESCE(n.last_note_at, a.created_time)
+                < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY),
+        FALSE
+    ) AS is_stale_with_open_pipeline
+FROM {{ ref('stg_zoho_crm__account') }} a
+LEFT JOIN contacts c ON c.zoho_org = a.zoho_org AND c.account_id = a.account_id
+LEFT JOIN deals    d ON d.zoho_org = a.zoho_org AND d.account_id = a.account_id
+LEFT JOIN notes    n ON n.zoho_org = a.zoho_org AND n.account_id = a.account_id
+LEFT JOIN {{ ref('stg_zoho_crm__user') }} u
+       ON u.zoho_org = a.zoho_org AND u.user_id = a.owner_id
+ORDER BY open_pipeline_amount DESC