Leads & CRM — Lead Ingestion (webhook channels)

How external lead sources auto-create Taikan leads, stamped with the hook (the ad / form / campaign that attracted them). This is one shared pipeline with per-channel adapters:

This page is the canonical reference for the pipeline. Planning history lives in spike-social-whatsapp-lead-ingestion.md and whatsapp-lead-ingestion.md.

What it is

A coach connects an external account (a Facebook Page, a WhatsApp number) once. From then on, Meta delivers a webhook every time a prospect submits a lead form or messages the number from an ad. Taikan verifies the webhook, routes it to the owning org, and creates an organization_lead — automatically, with the ad as its hook — that lands in the same Leads list as a minisite or manual lead.

The design goal (see docs/decisions/) is API-first and channel-extensible: the data model and queue are channel-agnostic, so a new webhook source is a new adapter, not a new schema. (Email lead capture is being built on a different track — as a channel of the conversations inbox, not a lead-ingestion webhook adapter; see Related.)

Architecture — the shared spine

All channels share one module, apps/api/src/lead-ingestion/, and one set of infrastructure:

• integration_connections — the per-org connection row. external_account_id (page_id | phone_number_id) is what inbound webhooks resolve on to route an event to the owning org. Holds the encrypted access token + a config blob.
• lead-ingestion BullMQ queue — every channel enqueues onto it. The processor (lead-ingestion.processor.ts, an ObservableWorkerHost) dispatches by job.name to the channel’s processor service.
• Webhook HMAC — meta-signature.util.ts (verifyMetaSignature) validates the X-Hub-Signature-256 header with the same META_APP_SECRET for both Meta Lead Ads and WhatsApp (one Taikan Meta app serves both products).
• Credential encryption — tokens are encrypted at rest with the same key/service as payment provider credentials (PAYMENT_CREDENTIALS_ENCRYPTION_KEY).
• Event tracking — EventTrackingService records pipeline observability (*_lead_enqueued, *_message_dropped, *_connection_finalized, …) keyed on orgId.

The hook / attribution model

Two tables capture “where did this lead come from”:

• lead_campaigns — the org-scoped hook bucket surfaced in the UI (a named group: an ad, a form, a campaign). Resolved/created on the fly from the inbound payload, keyed uniquely on (organization_id, channel, external_ref). organization_leads.campaign_id points at it.
• lead_attribution — the 1:1 raw attribution for a single lead: the external id (leadgen_id | ctwa_clid), the Meta ad/adset/campaign ids + names, the CTWA creative (source_url/headline/body), consent, and the full raw payload for forensics.

See data-model.md for the column-level schema.

Gating

Ingestion is gated by an active integration_connections row, not a feature flag — no connection, no webhook routing, no lead. The connect endpoints require owner/admin membership. (Contrast the staff CRM endpoints, gated by the lead_management platform feature — see behavior.md.)

Channel: Meta Lead Ads (meta_lead_ads) — shipped

Connect (owner/admin): Facebook Login for Business OAuth.

1. GET /organizations/:orgId/integrations/meta/connect-url → redirect the coach to Facebook to authorize the Taikan app against their Pages.
2. GET /integrations/meta/callback — OAuth callback. Exchanges the code for a long-lived token, lists the coach’s managed Pages, and stores a pending connection.
3. Frontend page-picker → POST /organizations/:orgId/integrations/meta/pending/:pendingId/finalize with the chosen Page ids → subscribes each Page to the leadgen webhook field and writes one active connection per Page (external_account_id = page_id).
4. DELETE /organizations/:orgId/integrations/meta/:connectionId — unsubscribe + revoke.

Webhook: GET /webhooks/meta/leadgen (hub verify), POST /webhooks/meta/leadgen (HMAC). A leadgen change routes by page_id → active connection → enqueue. MetaLeadProcessorService fetches the full lead from the Graph API by leadgen_id, maps the form fields (meta-field-mapping.ts), resolves the hook bucket from the form/ad, creates the lead (source = facebook | instagram), and writes attribution.

Web: apps/web/src/components/settings/meta-lead-ads-connect.tsx (the Integrations-tab card + page-picker dialog).

Channel: WhatsApp Cloud (whatsapp_cloud) — in progress

Connect (owner/admin): WhatsApp Embedded Signup (Facebook JS SDK popup).

1. GET /organizations/:orgId/integrations/whatsapp/config → { appId, configId, graphApiVersion } for the popup.
2. Frontend launches Embedded Signup; the popup returns an OAuth code + the chosen waba_id / phone_number_id.
3. POST /organizations/:orgId/integrations/whatsapp/finalize with {code, wabaId, phoneNumberId} → exchange code → subscribe the WABA to webhooks → register the number → write one active connection (external_account_id = phone_number_id).
4. DELETE /organizations/:orgId/integrations/whatsapp/:connectionId — unsubscribe + revoke.

Webhook: GET /webhooks/whatsapp (hub verify, uses WHATSAPP_VERIFY_TOKEN), POST /webhooks/whatsapp (HMAC). A messages change routes by phone_number_id → active connection. v1 rule: only messages carrying a CTWA referral (i.e. from a Click-to-WhatsApp ad) create a lead — organic inbound is intentionally ignored so ordinary member chats don’t become leads. Qualifying messages enqueue with job name whatsapp. WhatsAppProcessorService is idempotent on ctwa_clid (falling back to wamid), resolves the hook bucket from referral.source_id (named from the ad headline), creates the lead (source = whatsapp, phone = wa_id, name = profile name), and writes attribution from the referral.

Web: apps/web/src/components/settings/whatsapp-connect.tsx + fb-embedded-signup.ts (the popup launcher).

See whatsapp-lead-ingestion.md for the locked decisions (direct Meta Cloud API, per-org numbers, inbound-only) and Meta review/verification gates.

Code map

Shared LeadUpsertService (planned): the campaign-resolve + lead-upsert helpers are currently duplicated across MetaLeadProcessorService and WhatsAppProcessorService. Extract when a 3rd webhook channel lands.

Routes

Environment

One central Taikan Meta app serves every org and both products. See .env.example for the full list; the ingestion-specific vars:

(Meta also uses a login config id + OAuth redirect uri for the Lead Ads flow — see meta-config.service.ts and .env.example.)

For the full Meta App Dashboard setup (webhooks, Login config, Embedded Signup, App Review checklist), see the runbook meta-whatsapp-integrations.md.

Related: conversations inbox & pipelines

Two adjacent efforts (branch feat/lead-pipelines-inbox, WIP) build around the lead this pipeline creates — keep the axes distinct:

• Conversations inbox (apps/api/src/conversations/) — a channel-agnostic message thread per contact (manual / email / SMS / WhatsApp / IG / FB). Email lead capture lands here, not in lead-ingestion: inbound email threads back via Cloudflare Email Routing → /webhooks/email/inbound (runbook runbooks/inbox-email-inbound.md, ships with that branch). The inbox defines a ChannelAdapter interface that WhatsApp (this pipeline) is expected to plug into for two-way messaging — acquisition (here) and conversation (there) converge on the same contact.
• Configurable lead pipelines (apps/api/src/lead-pipelines/) — replaces the fixed five-state board with org-defined stages, each mapped to a lead_status category (adds organization_leads.pipeline_id / stage_id).

The three “provenance vs. position vs. history” axes, so they don’t blur: lead_campaigns = acquisition hook (where the lead came from) · lead_pipeline_stages = funnel position (where it is now) · conversations = the message history.

Current status

• Meta Lead Ads — shipped (PR #137). Connect (OAuth + page-picker), webhook, field mapping, hook + attribution.
• WhatsApp Cloud — inbound pipeline + connect flow built (PR #138); blocked on Meta Business Verification + App Review (whatsapp_business_management, whatsapp_business_messaging) before a real number can go live. Testable now against Meta’s free WhatsApp test number.
• Email — in progress on a different track: a channel of the conversations inbox (branch feat/lead-pipelines-inbox), not a lead-ingestion webhook adapter. See Related.

Known gaps

• No shared LeadUpsertService yet (per-channel duplication — see code map note).
• WhatsApp v1 ignores organic inbound (CTWA-only). Fast-follow option: broaden to “any new non-member inbound”.
• Outbound nurture (replies / templates) is out of scope here — it belongs to automations (whatsapp.channel, FIT-140).
• No UI yet to rename / recolor / archive auto-created lead_campaigns buckets (the columns exist: name, color, is_archived).