Spike: Social + WhatsApp Lead Ingestion (“the hook”)

Status: Spike / exploration (not a decision yet) Date: 2026-06-17 Author: Saar (with Claude) Trigger: First alpha customer saw a lead pipeline in competitor more-than.co.il  — Instagram leads auto-appearing in the platform, each tagged with the hook that interested them (course / campaign / topic), and wants the same in Taikan.

1. TL;DR / recommendation

The customer’s ask decomposes into three things:

1. Leads auto-created from Facebook/Instagram and WhatsApp.
2. Each lead carries the hook that attracted them (a specific course, campaign, topic) — not just the channel.
3. Same-hook leads group into a list for funnel follow-up.

The good news: Taikan already ships a leads/CRM domain, the Spotter agent already has lead tools, and we already have the webhook pattern, per-org encrypted credential storage, and a BullMQ queue. This is an extension of existing surfaces, not a new product.

What’s actually missing: (a) a hook/campaign attribution layer, (b) ingestion endpoints for Meta Lead Ads + WhatsApp, (c) per-org connect flow for Meta assets. The competitor’s “magic” is a UX wrapper around one standard Meta primitive — Meta hands us the hook on a plate.

Recommended sequencing (build the cheap, defensible parts first; defer the bureaucracy):

The single most important non-engineering fact: to ingest other businesses’ Meta leads natively you need Meta App Review + Business Verification + (for WhatsApp) Tech Provider verification. Meta’s review SLA is days, but first-time end-to-end is realistically weeks with iteration. Start that paperwork the moment we commit to P2. Until then, P0 + P1 deliver the demo with zero Meta gatekeeping.

2. What the customer actually saw (demystified)

more-than.co.il is a Hebrew white-label of GoHighLevel (HighLevel) — confirmed technically: app.more-than.co.il is a CNAME to whitelabel.ludicrous.cloud (GHL’s white-label host), the login serves LeadConnector (GHL) assets, and the site pitches GHL’s “Snapshots” and “SaaS Mode” reseller features. There is no proprietary magic — the customer is impressed by good packaging of a standard Meta integration.

The mechanism behind “Instagram leads auto-appear with their hook”:

Meta campaign (Leads objective, IG placement, instant form)
   └─> user submits the in-app lead form
        └─> Meta fires the `leadgen` webhook (IDs only: leadgen_id, form_id, ad_id, page_id)
             └─> CRM calls Graph API GET /{leadgen_id} → field_data + form/ad/campaign names
                  └─> create contact, tag by form name (= the "hook"), drop into a pipeline
                       └─> Smart List / segment filtered by that tag = "leads grouped by hook"

So the “hook” is literally the lead-form name (and/or ad/campaign name), which the Graph API returns alongside the lead’s answers. Grouping by hook is a GROUP BY/saved filter. This is squarely inside Taikan’s existing leads domain.

Important expectation-setting: what the customer saw is the Lead Ads instant-form path — it requires the org to run Lead Ads campaigns. “A DM that says ‘lead’ on organic Instagram” is a different path (IG Messaging API / comment automation / our screenshot-forward idea). We should be explicit about which we’re matching.

3. Current state in Taikan (what exists vs the gap)

Already shipped (verified in code)

The gap (what the customer’s ask needs)

1. No hook/campaign attribution. leads.source captures the channel (instagram) but not which campaign/course/topic. There is nothing to group by.
2. No external-source IDs. No leadgen_id / form_id / ad_id / ctwa_clid / wamid → can’t dedup webhook retries, can’t re-fetch from Meta, can’t attribute.
3. No raw-payload / consent capture. Meta deletes leads after 90 days; we must persist immediately and store the consent record (GDPR / Israel Amendment 13, in force 2025-08-14).
4. No ingestion endpoints for Meta leadgen or WhatsApp inbound.
5. No per-org Meta/WhatsApp connect flow (OAuth / embedded signup → store tokens).
6. No “list” concept beyond ad-hoc filters — need a first-class hook/campaign bucket.
7. No vision tool on Spotter to parse a screenshot/DM into a lead.

4. The “hook” — proposed data model

The hook is the heart of the request. Design goal: every lead, regardless of channel, arrives stamped with an immutable channel + a human-readable hook chip, and hooks are first-class buckets you can filter/group by. Steal GoHighLevel’s hard-won lesson: don’t bury attribution in a free-text UTM field nobody can filter — promote it to a real column/entity from day one.

New tables

lead_campaigns               -- the "hook" / list (org-scoped, first-class bucket)
  id, organization_id, name, channel (leadSource), color,
  external_ref (form_id | ad_id | ig ref | ctwa source_id),   -- for auto-routing
  is_archived, created_at, updated_at
  UNIQUE(organization_id, channel, external_ref)

lead_attribution             -- 1:1 with a lead; channel-specific evidence + consent
  id, lead_id,
  external_lead_id,          -- leadgen_id | wamid | ctwa_clid  (dedup key)
  form_id, form_name,
  ad_id, ad_name, adset_name, campaign_id, campaign_name,
  ref_payload,               -- ig.me/m.me ?ref= value
  source_url, headline, body,-- CTWA referral creative (the literal hook text)
  consent jsonb,             -- privacy-policy link, consent checkboxes, lawful basis
  raw jsonb,                 -- full provider payload (audit / replay)
  created_at
  INDEX(external_lead_id)

Change to existing

organization_leads
  + campaign_id  uuid NULL REFERENCES lead_campaigns(id)   -- which hook/list

Why this shape

• lead_campaigns IS the list. “Aggregate same-hook leads under the same list” = organization_leads WHERE campaign_id = ?. The bucket can be auto-created from an incoming form/ad name (via a mapping rule) or manually defined by the coach.
• external_ref enables auto-routing. form_id 12345 → campaign “12-week shred”. When a lead comes in for an unmapped ref, auto-create a campaign named after the form/ad (so nothing is lost) and let the coach rename/merge later.
• external_lead_id extends dedup. Webhooks retry; key inserts on external_lead_id to stay idempotent, on top of the existing email+phone dedup.
• raw + consent satisfy the 90-day-retention and privacy obligations and let us replay/repair without re-fetching from Meta.
• Channel stays an enum, hook is an entity. Channel is closed-set (good for Taikan’s API-first “explicit state” principle); hooks are open-ended and org-defined.

This keeps all existing flows (dedup, auto-task, convert) intact — we only add fields and a join.

5. Integration landscape (condensed technical reference)

Full payloads + citations in the appendix. The crux is attribution is solved on every channel — Meta gives us the hook on the first event — and the only real cost is review/verification gates.

5.1 Meta Lead Ads (Facebook + Instagram)

• Subscribe the app to the leadgen field on the page object. The webhook carries IDs only (leadgen_id, form_id, ad_id, adgroup_id (=adset), page_id, created_time) — never the answers.
• Fetch the lead: GET /{leadgen_id}?fields=field_data,ad_name,adset_name,campaign_name,form_id,created_time. field_data[].values is always an array; custom + hidden form fields appear here keyed by question key. (Custom disclaimer/consent checkboxes need custom_disclaimer_responses separately.)
• Hook resolution: request ad_name/adset_name/campaign_name directly on the lead node (no join); form_id → name via GET /{form_id}?fields=name,questions.
• 90-day retention — persist on receipt; backfill missed webhooks via GET /{form_id}/leads?filtering=[time_created>...].
• Permissions (all need App Review → Advanced Access to serve other orgs): leads_retrieval, pages_manage_metadata, pages_show_list, pages_read_engagement, plus ads_read/business_management for campaign-level names.
• Four gates, all required: (1) App Review per permission (screencast mandatory), (2) Business Verification (prereq for #1), (3) Tech Provider verification, (4) the page owner must grant our app Leads Access in Business Suite — app permission ≠ page lead access, this is the #1 real-world tripwire.
• Tokens: Facebook Login for Business with a saved config → Business Integration System User token (server-to-server). Page access token subscribes each page to the webhook.
• One central app serves all tenants — never one app per customer. Each org connects via the login flow.

5.2 WhatsApp Business Platform

• Cloud API only (On-Premise sunset 2025-10-23). Meta-hosted, free infra; pay per message.
• Inbound webhook: object: whatsapp_business_account → messages[] with from, type, text.body, etc.
• The WhatsApp “hook” = the CTWA referral object on the first inbound message from a click-to-WhatsApp ad: source_id (ad id), source_type, headline, body (the ad copy = literal hook), ctwa_clid (click id). Capture ctwa_clid before any auto-reply runs. Close the loop later via the Conversions API for Business Messaging.
• WhatsApp Flows = in-chat multi-screen forms; submission arrives as interactive.nfm_reply.response_json (stringified JSON) — ideal for structured qualification.
• Pricing shifted 2025-07-01 to per-message. Service/free-form messages inside the 24h customer-service window are free; a 72h free-entry window opens on a CTWA click. Marketing/Auth templates always cost. Design takeaway: keep nurture as utility templates inside the window and lean on the CTWA free window → most inbound-lead messaging is effectively free.
• BSP choice: 360dialog (EU/GDPR, zero per-message markup, unlimited WABAs via one API) is the best economic fit; Twilio if we want richer SDKs. Avoid Green API — unofficial, ToS-violating, ban-risk, and it does not deliver the official CTWA attribution (disqualifying for the hook).
• Onboarding friction: Business Verification, phone-number verification (no VoIP/toll-free), display-name approval, template approval, Tech Provider enrollment (mandatory for ISVs).

5.3 Brokers (skip App Review for an MVP)

Zapier / Make / LeadsBridge hold their own Meta App Review and act as the connected app — the org OAuths + grants page Leads Access, and neither the org nor Taikan does App Review. Taikan only exposes an inbound webhook. Best for P1 validation. Trade-offs: per-lead cost, latency (instant vs 15-min poll), and lead PII transiting an extra sub-processor (note in the DPA). n8n does NOT help here — self-hosting still requires us to create the Meta app and get Advanced Access, and Meta allows one webhook per app, so you end up writing the same router you’d write in NestJS anyway.

6. Build vs Buy vs Broker

7. Proposed architecture

7.1 Ingestion flow (native, P2/P3)

Meta `leadgen` / WhatsApp inbound webhook
  → POST /webhooks/meta/leadgen  |  POST /webhooks/whatsapp     (@Public, @Throttle)
  → verify: GET challenge (verify_token) + X-Hub-Signature-256 HMAC
            (reuse WebhookEnforcementService pattern + PostHog flag, fail-open during rollout)
  → resolve org: page_id / phone_number_id → integration_connections.external_account_id
  → enqueue BullMQ job  (webhook has IDs only; the Graph fetch must be async + retryable)
       → LeadIngestionProcessor:
           - Meta: GET /{leadgen_id} → field_data + ad/form/campaign names
           - WhatsApp: read referral{} off the first message
           - map → upsert lead (dedup on external_lead_id, then email+phone)
           - resolve/auto-create lead_campaigns by external_ref  (= the hook/list)
           - write lead_attribution (+ consent + raw)
           - existing: auto-task `contact_lead`, push notify owner
           - optional: Spotter drafts first reply (speed-to-lead)

7.2 Per-org connection

New integration_connections table (generalizes paymentProviderConfigs):

id, organization_id, provider (meta_lead_ads | whatsapp_cloud | whatsapp_unofficial),
status, external_account_id (page_id | waba_id | phone_number_id),
encrypted_credentials (tokens), config jsonb, created_at, updated_at

Connect flow: Facebook Login for Business (Meta) / Embedded Signup (WhatsApp) → store token → subscribe page to leadgen → done. Token refresh handled in a @Cron (System User tokens; 60-day expiry unless Marketing-API standard access).

7.3 Spotter extensions

• leads.parse_capture (new vision tool): input = image/text (a DM screenshot, a forwarded card, pasted text) → returns a structured draft lead (name/phone/email/hook guess) → routes through the existing leads.create quick-action with a human-confirm step. No Meta dependency.
• The existing leads.create already supports arbitrary source — extend its schema with campaignId/hook.
• Mobile: a share-sheet target + paste box so a coach can share an Instagram DM straight into Spotter.

7.4 Feature flag

Per project policy, gate behind a PostHog flag (per-org, organization group), default off, fail toward current behavior: e.g. social-lead-ingestion. Each channel can have its own sub-flag (meta-lead-ads, whatsapp-concierge).

8. Creative ideas (ranked by ROI / friction)

1. Spotter “paste/drop a screenshot or DM → lead” (P0, ship first). Pure vision-LLM via the existing agent; no Meta review. A coach drops an Instagram DM screenshot, Spotter pre-fills a lead card with a guessed hook, coach confirms in one tap. Highest ROI, lowest risk, validates the whole thesis. Always human-in-the-loop.
2. Taikan WhatsApp concierge number. A shared Taikan number coaches forward a lead/DM/screenshot/voice note to; Spotter parses it (Whisper for voice — Israeli coaches send many) and inserts the lead, replying a confirmation. Nearly free: the coach’s inbound message opens a 24h service window so the reply costs nothing. Route by sender → org; disambiguate if a coach belongs to several orgs.
3. Per-campaign deep links (no ads-platform dependency). Generate, per hook, an ig.me/m/<handle>?ref=<hook> and a click-to-WhatsApp link with a unique pre-filled message. The coach drops it in their bio/story/ad; when the lead lands, we read ref/the referral and stamp the hook server-side. Gets “leads tagged by hook” without Lead Ads or App Review — great P1 win. (Kommo/ManyChat’s exact trick.)
4. Speed-to-lead auto-draft. On any new lead, Spotter instantly drafts the first reply for the coach to approve. Responding within 5 min ≈ 21× more likely to qualify (MIT/Oldroyd) — the marketing wedge.
5. Hook auto-derivation + merge. Unmapped form/ad → auto-create a campaign named after it (never drop attribution); coach renames/merges later. Smart-list view auto-maintained per hook.
6. Convert → compliant receipt. On conversion, fire Sumit/Morning to issue an Israeli receipt — closes the loop and removes the #1 reason to keep a second tool open.

9. Risks & open questions

Risks

• Meta App Review + Business Verification + Tech Provider = weeks of calendar time, not code. Schedule at P2 kickoff. P0/P1 deliver the demo meanwhile.
• Page Leads Access must be granted by each org owner separately from OAuth — design the connect UI to verify it (has-lead-access endpoint) and nag if missing.
• Privacy: org = controller, Taikan = processor, Meta = processor. Need lawful basis, retention policy, DPA chain, consent capture. Israel Amendment 13 (in force 2025-08-14) + GDPR.
• Unofficial WhatsApp = ban risk + no CTWA attribution; only behind an explicit per-org flag with disclosure.
• WhatsApp template/category approval friction for any outbound nurture (keep to utility-in-window where possible).

Decisions for Saar

1. Match scope: replicate exactly the Lead-Ads instant-form path the customer saw, or lead with the screenshot/forward path (faster, no Meta gate)? (Recommend: both — screenshot now, Lead Ads next.)
2. Broker first? OK to run the alpha customer through Make.com for P1 to validate before App Review? (Recommend: yes.)
3. WhatsApp posture: official BSP (360dialog, green-tick, paid) vs unofficial (own number, cheap, ban-risk) vs both behind a flag? (Recommend: BSP for concierge, both later.)
4. Concierge number: one shared Taikan number with org-routing, or per-org numbers? (Recommend: one shared to start.)
5. Tier/packaging: is this inside lead_management or a new higher tier?

10. Appendix — payload references

Meta leadgen webhook (IDs only):

{"object":"page","entry":[{"id":"<PAGE_ID>","time":1438292065,
 "changes":[{"field":"leadgen","value":{"leadgen_id":"...","page_id":"...",
   "form_id":"...","adgroup_id":"<ADSET>","ad_id":"...","created_time":1440120384}}]}]}

Lead fetch GET /{leadgen_id}?fields=field_data,ad_name,campaign_name,form_id:

{"id":"...","created_time":"2015-02-28T08:49:14+0000","ad_id":"...","form_id":"...",
 "field_data":[{"name":"full_name","values":["Joe"]},{"name":"email","values":["j@x.com"]},
               {"name":"course","values":["12-week shred"]}]}

WhatsApp CTWA referral (first inbound message — the hook):

{"messages":[{"referral":{"source_url":"<FB_AD_URL>","source_id":"<AD_ID>",
   "source_type":"ad","headline":"<AD_TITLE>","body":"<AD_COPY>","ctwa_clid":"<CLICK_ID>"},
   "from":"<WA_ID>","type":"text","text":{"body":"Hi, interested!"}}]}

Sources (high-confidence, Meta official + multi-source verified): Meta Webhooks for Lead Ads; Lead Gen Data reference; “About expired leads” (90-day); Access Levels / leads_retrieval reference; Facebook Login for Business; WhatsApp Cloud API webhooks + payload examples; CTWA referral (Sinch/360dialog/Meta); WhatsApp pricing July 2025 (CleverTap/YCloud/Meta); On-Premises sunset; competitor teardown (GHL white-label CNAME whitelabel.ludicrous.cloud); Kommo/ManyChat ref attribution; respond.io Lifecycle (stage-on-conversation); 360dialog pricing. Full URLs in the research threads behind this spike.

Confidence notes: webhook shapes, 90-day retention, the App-Review/Business-Verification/Tech-Provider gate stack, July-2025 per-message pricing, and the CTWA attribution mechanism are high-confidence. WhatsApp Israel per-message dollar figures and exact form sub-fields should be re-verified against Meta’s live cards before P3 build.