# meSEO Deep Audit Inventory **Date:** 2026-06-10 **Commit:** 07e37ef (HEAD, main) **Author:** Claude Code -- audit-only session **Purpose:** Onboarding reference for new dev team. Grounds all claims in actual file paths, commit hashes, and code. Where something cannot be confirmed from code, it is marked UNCONFIRMED. --- ## Table of Contents 1. [Session and Phase History](#1-session-and-phase-history) 2. [Sidebar Pages](#2-sidebar-pages) 3. [Core Platform Architecture](#3-core-platform-architecture) 4. [Audit Flows](#4-audit-flows) 5. [AI Strategist](#5-ai-strategist) 6. [JSX Dashboards](#6-jsx-dashboards) 7. [Guardrails](#7-guardrails) 8. [Customer-Facing Surfaces](#8-customer-facing-surfaces) 9. [Schema and Crons](#9-schema-and-crons) 10. [Risk Register](#10-risk-register) 11. [Patent Map](#11-patent-map) --- ## 1. Session and Phase History ### Pre-git-history phases (reconstructed from code evidence) The visible git history begins at 2026-05-25 (phase 30B). The code itself, however, contains evidence of a complete multi-phase build predating that commit. The following phases are reconstructed from what exists in the codebase today. #### Phase 1-3: Foundation **Evidence:** `prisma/schema.prisma` multi-tenant models; `src/lib/auth/`, `src/middleware.ts`, Clerk provider in `src/app/layout.tsx`. - Multi-tenant architecture: `Organization`, `Brand`, `User`, `OrgMembership`, `BrandMembership` - Clerk authentication: userId sync, organization context, `BrandMembership` role enforcement - Brand model: `domain`, `industry`, `locations[]`, `primaryServices[]`, `secondaryServices[]`, `plan`, `billingCycle`, `planExpiresAt`, `healthScore`, `isDemo` - Organization + subscription management - Onboarding flow: `src/app/onboarding/page.tsx` - Settings: `src/app/settings/page.tsx` (tabs: profile, integrations, notifications, white-label) - Brands Hub: multi-brand switcher, `src/app/brands-hub/page.tsx` - Pricing page: `src/app/pricing/page.tsx` #### Phase 4-6: Data Layer + Integrations **Evidence:** `src/lib/services/ga4-sync.ts`, `src/app/api/gsc/backfill/route.ts`, `src/lib/integrations/nango.ts`, `prisma/schema.prisma` BrandIntegration model. **Nango OAuth provider** (`src/lib/integrations/nango.ts`): - Universal OAuth for GSC, GA4, GBP, Bing - `getValidAccessToken(brandId, integrationId)` -- refreshes if expiring - `getCredentials(brandId, integrationId)` -- returns stored credentials - Credentials stored encrypted in `BrandIntegration.credentialsEnc` (AES-256-GCM) - Token expiry tracked in `BrandIntegration.tokenExpiresAt` **GSC backfill** (`src/app/api/gsc/backfill/route.ts`): - `POST /api/gsc/backfill` -- backfills 16 months of historical data - Dimensions: `["date", "page"]` and `["date", "query"]` - Pagination: 25,000 rows per batch; handles multi-batch responses - `GscPage` upsert: `brandId, pageUrl, clicks, impressions, ctr, position, date` - `GscQuery` upsert: `brandId, query, clicks, impressions, ctr, position, date` - Pre-deletes date range to prevent duplicate rows - Verifies connection via totals call before backfilling **GA4 sync** (`src/lib/services/ga4-sync.ts`): - `storeGa4Data(brandId, rows, startDate, endDate)` -- upserts daily snapshots - Safeguard: detects inflated key events (>3x sessions) and skips conversion writes - `MetricSnapshot` upsert: `{ brandId, date, source: "ga4", sessions, users, pageviews, conversions, bounceRate }` - `getGa4Summary(brandId, days)` -- summary query for dashboard/AI avatar **MetricSnapshot + ChannelMetric** (schema): - `MetricSnapshot`: daily per-brand per-source (ga4|gsc|site_tag|daily|manual|seed) - `ChannelMetric`: daily per-channel breakdown (organic|ai_search|social|gbp|direct) **Integrations page** (`src/app/integrations/page.tsx`): - Connect/disconnect: GSC, GA4, GBP, Bing - Status indicators per integration - API: `/api/integrations/connect`, `/api/integrations/disconnect` #### Phase 7-9: Core Audit Runner **Evidence:** `src/lib/audit/audit-runner.ts` (2038 lines), 26 dimension aggregator files under `src/lib/audit/aggregators/`, 7 synthesizer files under `src/lib/audit/synthesizers/`, `src/lib/audit/types.ts` (900+ lines). **audit-runner.ts** -- main orchestration: - 7-phase pipeline: (1) brand context, (2) data fetch, (3) external APIs (DataForSEO + Firecrawl), (4) dimension aggregation, (5) LLM synthesis, (6) output assembly, (7) DB write - Cost tracking constants: `COST_PER_CRAWL_PAGE = 0.0001`, `COST_PER_SONNET_CALL = 0.05` - Cost ceilings: $4.00 first-party, $2.00 third-party -- hard abort if exceeded - Degraded mode: triggers when `pages.length === 0`, uses Site Tag telemetry + GSC data - Prompt caching: `cache_control: { type: "ephemeral" }` on narrative synthesis input - External calls: DataForSEO (`/on_page/task_post`, `/backlinks/summary/live`, `/serp/google/organic/live/advanced`), Firecrawl (5-10 key pages, markdown + metadata), Anthropic **AuditInput type** (`src/lib/audit/types.ts`): 40+ optional input fields including: brandId, domain, businessName, industry, locations[], primaryServices[], targetAudience, pages[], gscData, ga4Data, siteTagData, competitors[], competitorGaps[], backlinks, aiVisibility, trackedKeywords[], conversionData, adData, signalData, userEngagementData, existingAudit, analysisConfig **SeoSection type** (`src/lib/audit/types.ts`): 40+ optional output fields including: overallScore, headline, summary, keyFindings[], priorityActions[], competitiveLandscape, organicSearch, technicalHealth, conversionGap, eeat, aiSearchVerification, aiEnginePersonality, voiceSearch, pageSpeed, backlinks, multiTouch, revenueAttribution, executiveBrief, strategicRoadmap, implementationAppendix, industryContext, dimensionScores **DimensionScores**: technicalSeo, contentMetadata, aiVisibility, conversionOptimization, userEngagement **26 dimension aggregators** (in `src/lib/audit/aggregators/`): Each returns a typed object fed into the LLM context. Input = slices of AuditInput. Output = dimension-specific analysis. | Aggregator | Domain | |------------|--------| | `backlinks.ts` | Domain authority, link quality, toxicity scoring | | `brand-authority.ts` | Trust signals, E-E-A-T surface area | | `competitor-deep-dive.ts` | Per-competitor detailed breakdown | | `competitor-landscape.ts` | Market positioning, threat levels | | `competitor-visual.ts` | Side-by-side metric comparison | | `content-metadata.ts` | Title/meta/H1 coverage and quality | | `conversion-heatmap.ts` | Conversion-weighted session data | | `conversion-opportunity.ts` | Revenue gap vs. competitors | | `ecommerce.ts` | Product listing, checkout funnel signals | | `eeat.ts` | Expertise-Authority-Trust signals | | `executive-brief.ts` | Score + business impact narrative | | `industry-benchmark.ts` | Brand vs. vertical median KPIs | | `key-findings.ts` | Top 5-7 prioritized findings assembly | | `live-visual-annotations.ts` | Real-time behavioral overlay signals | | `lost-opportunity.ts` | Revenue impact of ranking gaps | | `multi-touch-ai.ts` | Multi-touch attribution across channels | | `organic-search.ts` | Click/impression/position from GSC | | `page-speed.ts` | Core Web Vitals (LCP/CLS/INP/TTFB/FCP) | | `page-strategic-pass.ts` | Per-page content/technical recommendations | | `predictive-conversion.ts` | ML-based conversion probability estimates | | `priority-actions.ts` | Ranked action item assembly | | `revenue-attribution.ts` | Per-channel revenue attribution | | `schema-opportunity.ts` | Schema markup gap analysis | | `signal-center.ts` | Rank/traffic/competitor signal aggregation | | `snippet-capture.ts` | Snippet-by-snippet capture tactics | | `strategic-roadmap.ts` | 90-day implementation roadmap | **7 synthesizer files** (in `src/lib/audit/synthesizers/`): Each receives aggregated dimension data and returns narrative + structured output. | Synthesizer | Model | Timeout | Notes | |------------|-------|---------|-------| | `seo-synthesizer.ts` | Sonnet 4.6 (narrative) + Haiku 4.5 (structured) | 280s + 45s in `Promise.all` | Primary: 14K tokens. 14 industry lenses. Vendor confidentiality enforced (service names stripped). | | `real-user-technical.ts` | -- | -- | Surfaces non-SEO issues from RealUserMetric | | `ecommerce.ts` | -- | -- | E-commerce product/checkout funnel lens | | `multilingual.ts` | -- | -- | hreflang and regional content gaps | | `compliance.ts` | -- | -- | HIPAA/GDPR/legal compliance signals | | `b2b.ts` | -- | -- | B2B SEO lens | | `industry-classifier.ts` | -- | -- | Derives industry string[] from domain/content/integrations (commit `3fc9729`) | **Industry lenses** (14 categories): healthcare-medical, b2b-saas, ecommerce, local-services, professional-services, finance, legal, real-estate, nonprofit, hospitality, higher-ed, b2b, dentist, other. `industryContext` persisted to `seoSection` JSONB (commit `ce67953`). #### Phase 10-12: AI + Public Audit **Evidence:** `src/app/api/public-audit/route.ts`, `src/lib/services/ai-visibility-checker.ts` (250+ lines), `src/app/ai-strategist/page.tsx`, `src/lib/ai-strategist/artifacts.ts`. **Public audit flow** (`src/app/api/public-audit/route.ts`): - No auth required - Lead capture: firstName, lastName, email, companyName, phoneNumber, websiteUrl - Rate limit: 3 req/IP/hour - OTP email verification -- returning leads (matched by email) bypass OTP - Same synthesizer pipeline as first-party audit - `LockedFirstPartyTeaser.tsx` shows upgrade CTA to anonymous users - `Audit.auditToken` stores the shareable audit URL **AI visibility checker** (`src/lib/services/ai-visibility-checker.ts`): - Uses Anthropic API with `web_search_20250305` tool - 7 AI sources checked: ChatGPT, Perplexity, Gemini, Claude, Copilot, Phind, You.com - `checkAiVisibility(brandId, query)` -- returns `AiCheckResult { mentioned, position, context, sentiment, sourceUrls }` - Results stored in `AiVisibilityCheck` + aggregate to `AiVisibilitySnapshot` - Weekly cron at `0 0 * * 1` **AI Strategist** (`src/app/ai-strategist/page.tsx`, 1342 lines): - Conversational interface - Context: brand profile + GSC + Site Tag + AnalysisJob.result + paid media + LLMO context - Artifacts: `` tags rendered in iframe sandbox via `ArtifactRenderer` - Models: claude-sonnet-4-6, 12K max output tokens, 120s maxDuration - Every call logged to `AiInteraction` (userId, brandId, feature, guardrailFlags[]) #### Phase 13-15: Content + Execution + Competitive **Evidence:** `src/app/keyword-research/page.tsx`, `src/app/rank-tracker/page.tsx`, `src/app/api/cron/rank-check/route.ts`, `src/app/backlinks/page.tsx`, `src/app/competitors/page.tsx`, `src/app/api/content-briefs/generate/route.ts`, `src/app/execution-hub/page.tsx`. **Keyword research** (`src/app/keyword-research/page.tsx`): - Deep-link: `?keyword=X` from Topic Map / Execution Hub - AI analysis output (Claude): summary, topic clusters, quick wins, gap analysis, question keywords, avoid keywords, content plan - Each cluster: `name, keywords[], contentStrategy, priority, estimatedTrafficPotential` - Search history: `GET /api/keyword-research/analyze?brandId=...` - maxDuration: analyze 120s, search 60s **Rank tracker** (`src/app/rank-tracker/page.tsx`): - `GET /api/cron/rank-check` (daily, 300s maxDuration) - DataForSEO `/serp` live checks per `TrackedKeyword` - Cost cap: soft $5 USD/day, hard 100 keywords/brand - Summary cards: total tracked, avg position, top-3/10/20/not-ranking counts - Distribution chart with click-to-filter, movers panel (improved/declined) - `TrackedKeyword`: keyword, location, device, targetUrl, tags[], isActive - `KeywordRanking`: per-day position + previousPosition + checkedAt **Backlinks** (`src/app/backlinks/page.tsx`): - `Backlink`: sourceDomain, sourceUrl, targetUrl, anchorText, linkType, domainAuthority, relevanceScore, toxicityScore, qualityTier, toxicityReasons[], status (active|lost|disavowed) - `BacklinkSnapshot`: historical totals + avgDA + toxicCount - Features: quality distribution chart, toxic panel, gap analysis (competitor backlinks we lack), trend chart (monthly new/lost), ref domain list with DA ranking - DataForSEO backlinks API: daily `backlink-sync` cron **Competitors** (`src/app/competitors/page.tsx`): - `Competitor`: domain, name, type, isActive, lastCrawledAt, detectedCms - `CompetitorSnapshot`: date, pageCount, newPages, movement, rankChange, aiMentions, trafficEstimate - Features: discovery + manual add, sitemap monitoring, ranking comparison, AI strategic analysis (Claude), change timeline, content gaps - `CompetitorDiscoveryJob`: async discovery with progress tracking **Content briefs** (`src/app/api/content-briefs/generate/route.ts`): - `POST /api/content-briefs/generate` - Types: `"seo"` (keyword-optimized editorial, H1/H2 structure, FAQ, meta) and `"llmo"` (Question-Answer-Explanation-Insight cadence for AI citation) - Model: `callLlm()` from `@/lib/ai-avatar/llm-client` - Humanization toggle, tone options (professional|conversational|persuasive) - `ContentBrief`: type, status (draft|review|approved|published), primaryKeyword, sections (JSON), humanizationMode, contentScore **Execution Hub** (`src/app/execution-hub/page.tsx`, 73KB): - `Task`: title, category (seo|llmo|gbp|schema|content|technical|web-dev), status (todo|in_progress|completed), priority, source, sourceId, pipelineStage, briefId, assignedTo, dueDate - Views: Calendar, Pipeline board - Tasks linked to audit/signal sources via sourceId #### Pre-git-history continuation: supporting systems **Email reports** (`src/lib/services/email-report-generator.ts`): - Sections (configurable per schedule): GSC, GA4, Site Tag, Technical Audit, Rank Tracker, Conversions, Page Lifecycle, UX Signals - Frequency: daily|weekly|biweekly|monthly - AI enrichment: Claude analysis of snapshot data -> summary, wins, concerns, recommendation (JSON) - Timezone support from `User.timezone` (fallback: America/New_York) - `EmailReportSchedule`: brandId, frequency, sections[], time, lastSentAt - `SentEmailReport`: scheduleId, recipientEmail, sentAt, contentHash **Alert system** (`src/lib/services/signals.ts`): 14 signal generators, all returning `Signal[]`: - `generateTechnicalAuditSignals`, `generatePerformanceAuditSignals`, `generateDeadPageSignals` - `generateConversionSignals`, `generateExecutionSignals`, `generateAiVisibilitySignals` - `generateSiteTagSignals`, `generatePageLifecycleSignals`, `generateRankTrackingSignals` - `generateCompetitorSignals`, `generateDetectedSignals`, `generateAiRecommendationSignals` - `generateDataFreshnessSignals`, `generateAiRecommendationSignals` Signal shape: ```typescript { id, title, description, category: "issue"|"win"|"opportunity"|"revenue"|"ai"|"execution", priority: "critical"|"high"|"medium"|"low", status: "new"|"viewed"|"actioned", timestamp, relatedPage?, relatedChannel?, source, issueIds?, suggestedAction?, competitor?, revenueImpact?: "high"|"medium"|"low" } ``` Dismissal TTL: 14 days (synthetic signals re-surface after this period). Revenue impact derivation: rules-based on category + priority + content keywords. **CRM integration** (`src/lib/services/crm.ts`): - Supported CRMs: HubSpot, Salesforce, HighLevel, manual - `CrmDeal`: stages (lead|qualified|proposal|negotiation|closed_won|closed_lost), status (open|won|lost), value, closeDate, contactName, contactEmail, externalId - `getDeals(brandId, range?, filters?)` -- with status/stage/source filtering - `getPipelineSummary(brandId, range?)` -- pipeline value, win rate, by-source breakdown - `upsertDeal()` -- create/update via externalId (sync with external CRM) - `CrmDeal` links to `ConversionEvent` via `conversion` relation - `RevenueAttribution` tracks conversion -> deal attribution **Page lifecycle** (`src/lib/services/page-lifecycle.ts`, `src/app/page-launch-tracker/page.tsx`): - `resolveSiteFromTag(siteId, pageUrl)` -- validates TrackedSite + domain allowlist - `toPathKey(url)` -- normalizes to canonical `/path` (strips protocol/host/query/hash) - `buildFullUrl(brandDomain, path)` -- reconstructs absolute URLs - `PageRecord`: url, currentStatus, firstSeenAt, lastSeenAt, redirect chains - `PageStatusChange`: status transitions with timestamps - `RedirectRecord`: source, target, statusCode - `PageDailyMetric`: GSC-joined per-URL daily clicks/impressions - Page launch detection: joins Site Tag first-seen dates with GSC baselines; tracks post-launch deltas **Admin overview** (`src/app/api/admin/overview/route.ts`): - Aggregates: total orgs/brands/users, active brands (7-day), integration counts (GSC/GA4/GBP/Site Tags) - AI usage: today/week/month + costUsd - Event coverage: 13 categories (Core, Forms, Phone, Booking, Video, E-commerce, UX Signals, Performance, Social, Content, Competitor, Chat, Scheduling) - Data sources: `TrackedEvent` count by eventType, `SiteEvent`, `RealUserMetric` - In-memory 60s TTL cache **Dashboard** (`src/app/dashboard/page.tsx`): - Module-scope Map (60s TTL) + localStorage (12h TTL) for cache - 3-phase loading: Phase -1 = localStorage fallback (instant), Phase 0 = server snapshot (`/api/dashboard/summary`), Phase 1 = live fetch with 20s AbortController timeout - 9 customizable sections: KPI Cards, Paid Media, Technical Issues, Execution Hub, Recommendations, AI Visibility, Competitors, Search Performance (lazy-loaded), Quick Actions - Channel toggle: GA4 split All Traffic vs. Organic Search - API: `/api/dashboard`, `/api/dashboard/summary`, `/api/dashboard/search-performance`, `/api/conversion-intelligence/attribution` **public/t.js** (client tracking tag, 162.8 KB minified): - 74 tracked event types across 13 categories (full list in Section 8) - Session stitching: `meseo_sid` + `meseo_sts` cookies; `meseo_xsid`/`meseo_xsts` for cross-subdomain - Client session ID: random string, persisted in cookies + memory - SPA support: `page_view` on route changes (history API + URL polling) - Event batching: buffered, flushed on timeout or batch-full - Collection endpoint: `POST /api/collect` **Collect route** (`src/app/api/collect/route.ts`): - Batch path: `{ site_id, visitor_id?, events: [...] }` (up to 200 events) - Single-event path (legacy): full validation + rate check - Domain allowlist enforcement (TrackedSite.allowedDomains) - `createMany` for batch inserts; single `create` for legacy path - Side effects (when `SITE_TAG_SIDE_EFFECTS_ENABLED=true`): platform detection, integration detection, external conversion initiation - Request size limit: 64 KB --- ### Git log overview 106 commits on main from 2026-05-25 to 2026-06-10. Commit-message phases (19, 19A.x, 19.5x, 20A) do not map 1:1 to roadmap numbering (13-27). Reconciliation is below. ### Phase groups (git history) #### Phase 16 (docs/validation) -- 2026-05-26 - `cadef8e` docs(phase-16): add validation and auth architecture reference - `d9417b6` docs(phase-16): replace em dash substitutes - Files: `docs/architecture/phase-16-validation.md`, `docs/architecture/CONVENTIONS.md` - Shipped: Documentation only. No code changes. #### Phase 17 (Voice Search Eligibility) -- 2026-05-26 - `705d2b3` feat(phase-17): voice search eligibility audit backend - `14c639f` feat(phase-17): voice search eligibility dashboard component - `4711328` feat(phase-17): add Voice Search Eligibility section to PDF audit report - Shipped: Full voice search eligibility pipeline -- schema, synthesizer output, dashboard component, PDF section. #### Phase 19 / 19A.x (Technical Audit + Site Tag Performance) -- 2026-05-27 to 2026-05-28 25+ commits. Sub-milestones: | Commit | Date | What shipped | |--------|------|--------------| | `545a6c7` | 05-27 | Phase 19A: technical audit page scaffold (mock data) | | `88b8dde` | 05-27 | Wire technical audit data layer lib | | `e976da2` | 05-27 | Wire audit trigger buttons | | `91c41ce` | 05-27 | Wire technical audit page to real seoSection data | | `e2b1a64` | 05-27 | Hide site tag banner when installed | | `715e470` | 05-27 | Wire technical audit to background refresh + notifications | | `3d1c64e` | 05-27 | Add SiteTagDailyRollup cron pre-aggregation | | `27553b7` | 05-27 | Remove analytics block from site-tag polling route | | `f909229` | 05-27 | t.js error dedupe + rate limit + counter-based capture | | `7593eb8` | 05-27 | Redis caching layer for analytics (5-min TTL) | | `959a662` | 05-28 | Phase 19A.20: loading states, tier label clarification | | `5488ffd` | 05-28 | Phase 19.5: multilingual, compliance, ecommerce, B2B synthesizers | | `08420cc` | 05-28 | Phase 19A.21: graceful stats degradation + audit page cache | | `3b7228f` | 05-28 | Fix 19A.22: wire SiteTagDailyRollup fast path into getTrafficSummary | | `0572928` | 05-28 | Fix 19A.23: register form_filled, defensive tier guard | | `51135cf` | 05-28 | Phase 19A.25: expand SiteTagDailyRollup with conversion/event columns | | `1db278e` | 05-28 | Fix 19A.26: use real siteTagInstalled on audit page | | `003e7d6` | 05-28 | Fix 19A.26.1: raise first-party audit maxDuration to 800s | | `8b1b5a8` | 05-28 | Fix 19A.26.2: fix third site-tag banner false-positive | | `d5e5396` | 05-28 | Fix 19A.27: base maturity banner on rollup active-days | | `321fdb2` | 05-28 | Phase 19A.28: universal siteTagInstalled in brand context | | `4dfb003` | 05-28 | Fix 19A.28 followup: suppress banner during brand switch | | `dd72106` | 05-28 | Fix 19A.28 switch: close banner flash during brand switch | | `8c761f9` | 05-28 | Run crawl-independent audit sections in degraded mode | | `1f1fe87` | 05-28 | Raise DFS polling window 240s -> 300s | | `4f96111` | 05-28 | Fix 19A.29: rollup-ize confidence route + Redis cache + hard 30s timeout | #### Phase 20A.x (White-Label) -- 2026-05-28 | Commit | What shipped | |--------|--------------| | `6bf61ee` | Add WhiteLabelConfig + CustomDomain + DomainStatus schema | | `151f654` | White-label settings tab + API route | | `c961a65` | Active-org isolation in white-label API + getEventTrends rollup fast path | #### Market Position Block 1 and 2 -- 2026-05-28 | Commit | What shipped | |--------|--------------| | `cdb15a9` | feat(market-position-block1): 6 new Prisma models (ProbeRun, VisibilityObservation, CompetitorPanel, QueryCorpusEntry, CalibrationRun, MarketPositionSnapshot) + 2 enums | | `c735d74` | feat(market-position-block2): canonicalization + corpus seeding | | `c018c80` | fix(market-position-block2): conversion types, N-scan perf, stop-words | #### Phase 30B (Platform Detection + Plugin Version) -- 2026-05-25 to 2026-05-29 | Commit | What shipped | |--------|--------------| | `1bad836` | Detect + store site platform from page_view events | | `b2abf55` | Platform confidence as strings, tighten signals, fix regexes | | `55fad07` | Plugin version extraction via version() on ANALYTICS_PLUGINS | | `8f5df21` | Gate bounded-async side effects behind SITE_TAG_SIDE_EFFECTS_ENABLED | #### Phase 32 + 33 (Dedup + Conversion Tier Fixes) -- 2026-05-25 | Commit | What shipped | |--------|--------------| | `f22100e` | Fix visitorId mismatch Path A vs Path B | | `8abc2a0` | Propagate synthetic flag from TrackedEvent to SiteConversion | | `b4a44b6` | Promote visitor_id to top-level collect body field | | `2946d61` | Fix conversion tier dropdown revert from polling | | `6282295` | Wire canonicalConversionType into BrandConversionConfig override | #### Session: Ecommerce + Shopify -- 2026-05-29 to 2026-06-02 | Commit | What shipped | |--------|--------------| | `1c25751` | orderId column, ingest extraction, API route, UI tab | | `6c4e8e0` | Shopify Web Pixel purchase ingest + lineItems column | | `9eece0a` | Extend event buffer flush 10s -> 30s, flush-on-full safety valve | | `4601e07` | Consolidate plugin_detected to single plugins[] event per page | | `dad4b60` | Exclude ingest beacon paths from middleware matcher | | `7b6e608` | 1%-sampled drop-reason logging on /event | | `f677a5f` | Per-session dedup for plugin_detected, phone_number_visible, call_tracking_detected | #### Session: Site Audit Phase 1 -- 2026-06-04 | Commit | What shipped | |--------|--------------| | `791d921` | SiteAuditRun, SiteAuditIssue, SiteAuditPage schema + renderTier on Brand | | `62610b6` | Crawler service, API routes, cron cleanup | | `d207a05` | Default crawl_delay 200ms, expose zero as opt-in | | `b0881e5` | Replace BrandMembership filter with verifyBrandAccess() on all user-facing routes | | `b62fdae` | Use waitUntil to keep Lambda alive after response | | `3ce9c07` | Enforce 60s minimum before accepting DFS "finished" | | `f4b4582` | Resolve canonical host before DFS submit; guard 0-page runs | | `9ad1005` | Cheap pre-reject in /event, sweep stuck runs | #### Session: Connection-Pool + Cron Fixes -- 2026-06-05 | Commit | What shipped | |--------|--------------| | `accf550` | Remove recomputeSessionScore from ingest hot path (stops 25P03 leaks) | | `5990d96` | Attach submissionId token to form_submit | | `2d1c1b9` | Unified phone_call dedup across both write paths | | `c376913` | Restore precompute-dashboard to every 30 min, add budget guard | | `6a69290` | Schedule site-audit-cleanup cron, per-tick cap + debounce | | `636ac66` | Stand up Inngest infrastructure (client, serve route, ping function) | #### Session: Site Tag 504 Fixes -- 2026-06-08 (this session) | Commit | What shipped | |--------|--------------| | `543f271` | Route 28d analytics onto SiteTagDailyRollup; bound all multi-day raw scans | | `d6dfed4` | Bound activePagesCount to today to stay under withDeadline | | `6c8cdae` | Add ?debugTiming=1 instrumentation to GET /api/site-tag | | `07e37ef` | Bound all 9 getReconciliationSummary queries to 48h window | ### Phase numbering reconciliation - Phases actually committed: 16, 17, 19/19A, 19.5, 20A, 30B, 32, 33 - Phases reconstructed from code (pre-git): 1-15 (see above) - Phases not in git or code: 18, 21-29 (not built or labelled differently) - Market Position is labelled "block1/block2" - Site Audit is labelled "Phase 1" internally with no roadmap number --- ## 2. Sidebar Pages | Page | Route | File | Wiring | |------|-------|------|--------| | Dashboard | `/dashboard` | `src/app/dashboard/page.tsx` | Full | | Audit | `/audit` | `src/app/audit/page.tsx` | Full | | Technical Audit | `/technical-audit` | `src/app/technical-audit/page.tsx` | Full | | AI Strategist | `/ai-strategist` | `src/app/ai-strategist/page.tsx` | Full | | Site Tag | `/site-tag` | `src/app/site-tag/page.tsx` | Full | | Site Tag Analytics | `/site-tag-analytics` | `src/app/site-tag-analytics/page.tsx` | Full | | Conversion Intelligence | `/conversion-intelligence` | `src/app/conversion-intelligence/page.tsx` | Full | | Rank Tracker | `/rank-tracker` | `src/app/rank-tracker/page.tsx` | Full | | Keyword Research | `/keyword-research` | `src/app/keyword-research/page.tsx` | Full | | Backlinks | `/backlinks` | `src/app/backlinks/page.tsx` | Full | | Site Performance Audit | `/site-performance-audit` | `src/app/site-performance-audit/page.tsx` | Full | | Competitors | `/competitors` | `src/app/competitors/page.tsx` | Full | | Industry Intel | `/industry-intel` | `src/app/industry-intel/page.tsx` | Full | | AI Visibility | `/ai-visibility` | `src/app/ai-visibility/page.tsx` | Full | | Signals | `/signals` | `src/app/signals/page.tsx` | Full | | Execution Hub | `/execution-hub` | `src/app/execution-hub/page.tsx` | Full | | Page Launch Tracker | `/page-launch-tracker` | `src/app/page-launch-tracker/page.tsx` | Full | | Content Brief | `/content-brief` | `src/app/content-brief/page.tsx` | Full | | LLMO Content Brief | `/llmo-content-brief` | `src/app/llmo-content-brief/page.tsx` | Full | | Schema Generator | `/schema-generator` | `src/app/schema-generator/page.tsx` | Partial (UI-driven, not auto-populated) | | Dead Pages | `/dead-pages` | `src/app/dead-pages/page.tsx` | Full | | Broken Links | `/broken-links` | `src/app/broken-links/page.tsx` | Full | | Report Studio | `/report-studio` | `src/app/report-studio/page.tsx` | Full | | Content Audit | `/content-audit` | `src/app/content-audit/page.tsx` | Full | | Recommendations | `/recommendations` | `src/app/recommendations/page.tsx` | Full | | Writing Assistant | `/writing-assistant` | `src/app/writing-assistant/page.tsx` | Full | | Email Reports | `/email-reports` | `src/app/email-reports/page.tsx` | Full | | Notifications | `/notifications` | `src/app/notifications/page.tsx` | Full | | Integrations | `/integrations` | `src/app/integrations/page.tsx` | Full | | Settings | `/settings` | `src/app/settings/page.tsx` | Full | | Brands Hub | `/brands-hub` | `src/app/brands-hub/page.tsx` | Full | | Client Portal Setup | `/client-portal-setup` | `src/app/client-portal-setup/page.tsx` | Full | | Local GEO | `/local-geo` | `src/app/local-geo/page.tsx` | **Stubbed -- "Coming Soon"** | | Paid Media | `/paid-media` | `src/app/paid-media/page.tsx` | Full | | Onboarding | `/onboarding` | `src/app/onboarding/page.tsx` | Full | | Pricing | `/pricing` | `src/app/pricing/page.tsx` | Full | | Admin | `/admin` | `src/app/admin/page.tsx` | Full (admin-only) | ### Key page details #### Dashboard (`/dashboard`) - **Data sources:** `MetricSnapshot`, `SiteTagDailyRollup` (fast path), `ConversionEvent`, `CompetitorSnapshot`, `ChannelMetric` - **API calls:** `/api/dashboard/summary`, `/api/dashboard/search-performance`, `/api/conversion-intelligence/attribution` - **Cache strategy:** 60s TTL module-scope Map + 12h TTL localStorage - **Range controls:** `1d | 7d | 28d | 90d | month | custom` #### Site Tag (`/site-tag`) - **Data sources:** `TrackedSite`, `TrackedEvent` (recent 15), `SiteTagDailyRollup` (fast path), `SourceEvent`, `CanonicalEvent` - **API:** `GET /api/site-tag?brandId=xxx` (polls every 30s) - **Tabs:** Setup/Snippet, Domain Status, Custom Events, Plugins Detected, Reconciliation - **Wiring notes:** 07e37ef bounded reconciliation to 48h; 543f271 uses rollup fast path for today counts; d6dfed4 bounded activePagesCount to today; 6c8cdae added `?debugTiming=1` block-level timing #### Site Tag Analytics (`/site-tag-analytics`) - **Data sources:** `SiteTagDailyRollup` (primary), `TrackedEvent` (live fallback), `HeatmapEvent`, `RealUserMetric`, `SiteConversion` - **API:** `GET /api/site-tag/analytics?brandId=xxx&range=30d` - **Tabs:** Traffic, Conversions, UX Signals, Heatmap, Ecommerce (conditional), AI Attribution, Video - **Deadline:** 20s `HANDLER_DEADLINE_MS`; returns `partial: true` on timeout #### AI Visibility (`/ai-visibility`) - **Data sources:** `AiVisibilitySnapshot`, `AiVisibilityCheck`, `AiVisibilityQuery` - **API:** `GET /api/ai-visibility?summary=true&range=30d`, `/api/ai-visibility/queries`, `/check` - **Feature gate:** `ai_visibility` flag -- Professional+ plan - **Cron:** Weekly `0 0 * * 1` via `/api/cron/ai-visibility` (maxDuration 300s) #### Execution Hub (`/execution-hub`) - **Data sources:** `Task` (source, sourceId link to audit/signal/dead-page) - **API:** Task CRUD via `/api/tasks` - **Categories:** seo|llmo|gbp|schema|content|technical|web-dev - **Views:** Calendar, Pipeline board #### Page Launch Tracker (`/page-launch-tracker`) - **Data sources:** `PageRecord`, `PageLaunch`, `PageLaunchSnapshot`, GSC click/impression join - **API:** `/api/page-launches` - **Features:** lifecycle from first detection (sitemap/GSC/tag), inbound redirect tracking, daily delta chart #### Local GEO (`/local-geo`) - **Status: COMING SOON -- intentionally stubbed** - `GbpLocation`, `GbpReview`, `GbpMetric` models and daily `gbp-sync` cron are active - Page deliberately disabled to avoid empty-state before end-to-end wiring **Gaps and unknowns:** `/topic-map` wiring status UNCONFIRMED (POC visualization). `issue-detail` wiring UNCONFIRMED. --- ## 3. Core Platform Architecture ### Multi-tenant model ``` Organization (1) -> Brand (N) -> TrackedSite (1) Organization (1) -> OrgMembership (N) -> User Brand (1) -> BrandMembership (N) -> User ``` Auth: Clerk JWT. `verifyBrandAccess()` (commit `b0881e5`) replaces raw `BrandMembership` filter on all user-facing routes. ### Integration layer **Nango** (`src/lib/integrations/nango.ts`): - Supports: GSC, GA4, GBP, Bing - `getValidAccessToken(brandId, integration)` -- auto-refresh - `getCredentials(brandId, integration)` -- stored in `BrandIntegration.credentialsEnc` (AES-256-GCM) - `BrandIntegration.tokenExpiresAt` tracked; marks expired on 2+ refresh failures - Nango token refresh cron: hourly `/api/cron/nango-sync` **Connection pool:** - `DATABASE_URL` query params: `connection_limit=20, pool_timeout=30` - Neon pooler endpoint for multiplexing - `recomputeSessionScore` removed from ingest hot path (`accf550`) -- was causing 25P03 idle-in-transaction leaks ### Redis cache **File:** `src/lib/cache/redis-cache.ts` - Thin wrapper over `redis` npm client; resolves `REDIS_URL` env var - Fails open (silently disabled) if `REDIS_URL` not set - Used by: analytics endpoints (5-min TTL), confidence route (30s timeout), pulse heartbeat - Commit: `7593eb8` (analytics), `4f96111` (confidence), `2d38e9e` (pulse) ### Inngest **Status: Infrastructure only. No production functions beyond ping.** - `src/lib/inngest/functions/ping.ts`: event `test/ping` - Serve route registered, client configured - Commit: `636ac66` ### prisma-safety-check **File:** `scripts/prisma-safety-check.mjs` - Blocks `prisma db push` if diff contains: `DROP COLUMN`, `DROP TABLE`, `DROP CONSTRAINT`, `DROP INDEX`, `ALTER COLUMN ... DROP` - Protects the ~22GB `SiteEvent` table indexes (commit `ecd5a48`) - Override: `ALLOW_SCHEMA_DROPS=true` - Self-test: `scripts/test-safety-check.mjs` --- ## 4. Audit Flows ### First-party technical audit (authenticated brand) **Entry:** Brand member clicks "Run Audit" on `/technical-audit`. **Route chain:** 1. `POST /api/technical-audit/run` (maxDuration 800s) - Auth: `verifyBrandAccess()` (commit `b0881e5`) - Creates `TechnicalAudit` row (status: pending) - Calls `runTechnicalAudit()` inside `waitUntil()` so Lambda stays alive after 200 (commit `b62fdae`) 2. `GET /api/technical-audit/[auditId]/status` -- polled every 5s - Returns `{ status, pagesScanned, errorCount, warningCount, passedCount }` 3. `GET /api/technical-audit/history` -- historical runs **DataForSEO integration:** - Task submission: `submitCrawl(domain)` inside `site-audit-runner.ts` -- POST to DFS `/v3/on_page/task_post` - Payload: `url`, `max_crawl_pages`, `crawl_delay` (default 200ms, commit `d207a05`), `store_raw_html: false` - Minimum 60s before accepting DFS "finished" (commit `3ce9c07`) - Polling window: 300s (raised from 240s in `1f1fe87`) - Results: `fetchAllCrawledPages()`, `fetchRedirectChains()`, `fetchDuplicateTitles()`, `fetchDuplicateDescriptions()` - On DFS unconfigured: limited-mode banner (commit `4028dd2`) **Synthesizer pipeline** (inside `runSiteAudit()`): - `seo-synthesizer.ts`: Sonnet 4.6 (14K tokens, 280s) + Haiku 4.5 (1.5K tokens, 45s) in `Promise.all` - `real-user-technical.ts`, `ecommerce.ts`, `multilingual.ts`, `compliance.ts`, `b2b.ts` - `industry-classifier.ts`: returns `string[]` for multi-industry support **Crawl-independent sections** (run in degraded mode when DFS crawl fails, commit `8c761f9`): SERP competitor analysis, GSC performance, Site Tag signals, AI visibility. **Cron cleanup:** `/api/cron/site-audit-cleanup` -- re-kicks stuck >15 min, fails >2 hours (commit `6a69290`). ### Third-party / public audit **Entry:** Anonymous visitor at public audit URL. **Route chain:** 1. `POST /api/public-audit/...` -- no auth, validates domain, issues OTP 2. Lead capture: firstName, lastName, email, companyName, phoneNumber, websiteUrl 3. Rate limit: 3 req/IP/hour 4. Returning leads (by email) bypass OTP 5. Same synthesizer pipeline as first-party (degraded crawl-independent sections) 6. maxDuration 800s 7. `LockedFirstPartyTeaser.tsx` CTA for anonymous users ### Audit runner internals (pre-git, audit-runner.ts) **7-phase orchestration** (`src/lib/audit/audit-runner.ts`, 2038 lines): 1. Brand context assembly (BrandProfile, integrations, plan) 2. Data fetch (GSC/GA4/Site Tag/competitors in parallel) 3. External APIs: DataForSEO SERP + backlinks + on-page, Firecrawl 5-10 key pages (markdown + metadata) 4. 26 dimension aggregators (run in parallel batches) 5. LLM synthesis (Sonnet 4.6 narrative + Haiku 4.5 structured) 6. Output assembly (SeoSection object, 40+ fields) 7. DB write (AnalysisJob upsert, score history) **Cost controls:** - `COST_PER_CRAWL_PAGE = 0.0001`, `COST_PER_SONNET_CALL = 0.05` - Hard abort at $4.00 (first-party), $2.00 (third-party) - Prompt caching: `cache_control: { type: "ephemeral" }` on synthesis input **Degraded mode:** `pages.length === 0` triggers Site Tag telemetry fallback. --- ## 5. AI Strategist **Page:** `src/app/ai-strategist/page.tsx` (1342 lines) **API:** `POST /api/ai-strategist/chat` (maxDuration 120s) ### Context sources Assembled by `getEnrichmentContext()` and `buildEnrichmentPrompt()`: - Brand profile: name, industry, locations[], primaryServices[] - DashboardData: SEO score, AI visibility, crawl health, monthly traffic - GSC metrics: searches, impressions, CTR, avg position - Site Tag metrics: sessions, conversions, top pages (SiteTagDailyRollup or live) - AnalysisJob.result: full JSON grounding for recommendations - Paid media: ad spend, ROAS (if AdPlatformIntegration configured) - LLMO context: AI visibility snapshot if feature enabled ### Models + logging - claude-sonnet-4-6, 12K max output tokens (`ARTIFACT_MAX_TOKENS`) - Every call logged to `AiInteraction` (userId, brandId, feature, userPrompt 4KB cap, aiResponse 16KB cap, guardrailFlags[], flagSeverity) - `trackEvent()` -> `PlatformEvent` - Admin surface: `/admin/ai-monitor` ### Artifacts - `` tags in response extracted by `parseArtifacts()` from `src/lib/ai-strategist/artifacts.ts` - Rendered in iframe sandbox via `ArtifactRenderer` -- supports JSX dashboards, markdown, structured reports - Session persistence: `StrategistSession`, `StrategistMessage` **Gaps and unknowns:** Token budget enforcement beyond max_tokens UNCONFIRMED. PII guardrail detection logic in `guardrailFlags` was not read from code. --- ## 6. JSX Dashboards ### Audit dashboard components (46 files) Location: `src/components/audit/dashboard/` Each receives a slice of the `Analysis` object from `AnalysisJob.result`. | Component | Renders | |-----------|---------| | `AIConversionGapChart.tsx` | AI-driven conversion gap analysis | | `AiEnginePersonalitySection.tsx` | LLM engine personality for brand | | `AiSearchVerificationSection.tsx` | AI search mention rates | | `AiSnippetEligibilitySection.tsx` | Featured snippet capture readiness | | `AiSourceAttributionComparisonSection.tsx` | First-party vs third-party attribution | | `AuditHero.tsx` | Report hero with brand + score | | `AuditTOC.tsx` | Table of contents with section links | | `BacklinkProfileSection.tsx` | Backlink quality, toxicity, domain authority | | `BehavioralActionBridgeSection.tsx` | Site Tag behavioral signals -> actions | | `BilingualContentSection.tsx` | Multi-language content gaps | | `BrandAuthorityScoreSection.tsx` | Domain authority + trust signals | | `CompetitiveLandscapeSection.tsx` | Top competitors, overlap, threat level | | `CompetitorDeepDiveSection.tsx` | Per-competitor detailed analysis | | `CompetitorVisualComparisonSection.tsx` | Side-by-side competitor metrics | | `ConversionHeatMapSection.tsx` | Conversion-weighted heatmap overlays | | `EeatAuditSection.tsx` | E-E-A-T signals | | `ExecutiveBriefSection.tsx` | Executive summary prose | | `ExecutiveSummaryCard.tsx` | Summary card with score breakdown | | `ImplementationAppendixSection.tsx` | Implementation roadmap table | | `IndustryBenchmarkSection.tsx` | Brand vs industry median | | `KeyFindingsSection.tsx` | Top 5-7 prioritized findings | | `LockedFirstPartyTeaser.tsx` | Upgrade CTA for anonymous audits | | `LostOpportunityCalculatorSection.tsx` | Revenue impact of ranking gaps | | `LiveVisualAnnotationsSection.tsx` | Real-time behavioral annotations | | `MethodologySection.tsx` | Audit methodology explanation | | `MultiTouchAiPathSection.tsx` | Multi-touch attribution AI path | | `OrganicSearchSection.tsx` | Organic traffic, clicks, impressions | | `PageSpeedInsightsDeepDiveSection.tsx` | Core Web Vitals deep dive | | `PageStrategicPassSection.tsx` | Per-page strategic recommendations | | `PredictiveConversionScoringSection.tsx` | ML-based conversion probability | | `PriorityActionsSection.tsx` | Ranked action items | | `QuickWinsGrid.tsx` | Fast-ROI action grid | | `RevenueAttributionSection.tsx` | Revenue per channel | | `SchemaOpportunityForecastSection.tsx` | Schema markup gap analysis | | `ScoreBreakdown.tsx` | Score sub-component breakdown | | `ScoreProjection.tsx` | Score trend projection | | `SignalCenterSection.tsx` | Signal aggregation | | `SnippetCaptureStrategySection.tsx` | Snippet capture tactics | | `StrategicRoadmapSection.tsx` | 90-day implementation roadmap | | `TechnicalHealthSection.tsx` | Technical crawl health summary | | `ThirdVsFirstPartyComparison.tsx` | GA4/GSC vs Site Tag comparison | | `VoiceSearchEligibilitySection.tsx` | Voice search schema + NLP readiness | | `index.ts` | Barrel export | ### Other component directories - `src/components/ai-strategist/artifact-renderer.tsx` -- iframe sandbox renderer - `src/components/execution-hub/calendar-view/` -- task calendar - `src/components/execution-hub/pipeline-board/` -- kanban board - `src/lib/audit/pdf/` -- `@react-pdf/renderer`: `AuditPdf.tsx`, `ExecutiveAuditPdf.tsx`, `pages/`, `styles.ts` --- ## 7. Guardrails ### Deadline wrappers **`withDeadline`** (`src/app/api/site-tag/analytics/route.ts` lines 46-51): ```typescript async function withDeadline(promise: Promise, ms: number): Promise> { return Promise.race([ promise.then((data) => ({ winner: "ok", data })), new Promise>((resolve) => setTimeout(() => resolve({ winner: "deadline" }), ms)), ]); } ``` - `HANDLER_DEADLINE_MS = 20_000` (20s) - On deadline: returns `{ partial: true, analytics: null }` with `eventsToday: 0` from spread defaults **`maxDuration` settings:** | Route | maxDuration | |-------|-------------| | `/api/site-audit/run` | 800s | | `/api/technical-audit/run` | 800s | | `/api/cron/site-tag-rollup` | 300s | | `/api/cron/ai-visibility` | 300s | | `/api/cron/rank-check` | 300s | | `/api/cron/ai-citation-check` | 300s | | `/api/ai-strategist/chat` | 120s | | `/api/keyword-research/analyze` | 120s | | `/api/keyword-research/search` | 60s | | `/api/performance-audit/run` | 60s | **`waitUntil` usage:** `/api/site-audit/run` keeps Lambda alive after 200 (commit `b62fdae`). ### Rate limits **Site Tag ingest** (`src/lib/services/site-tag-cache.ts`): - 100 events/min per siteId - In-memory token bucket, resets every 60s - Returns 200 on drop (not 429) to prevent client console flood **t.js error rate limit** (commit `f909229`): - Counter-based per-session dedup for JS errors **Public audit** (`src/app/api/public-audit/route.ts`): - 3 req/IP/hour ### Rollup fast paths **`SiteTagDailyRollup` fast path** (`src/lib/services/site-tag-analytics.ts`): - `_getTrafficSummaryUncached`, `_getConversionSummaryUncached`, `getOutboundClickSummary` - Guard: `range === "30d" || range === "28d" || range === "7d"` (extended to 28d in commit `543f271`) - Threshold: `rollupRows.length >= expectedDays - 1` - Turns 1M+ row `TrackedEvent` group-by into a 30-row indexed lookup on `(brandId, date)` **Rollup cron** (`/api/cron/site-tag-rollup`, hourly `0 * * * *`, maxDuration 300s): - Force-recomputes today + yesterday on every run - Backfills last 30 days if row missing or `conversionsByType IS NULL` - UTC midnight date key: `new Date(Date.UTC(y, m, d))` **Today rollup fast path (setup page)** (`src/app/api/site-tag/route.ts`): - `siteTagDailyRollup.findFirst` for `eventsToday`, `eventBreakdown`, `sessionsToday` - Falls back to live `TrackedEvent` only if row absent or `eventCountsByType` is null **activePagesCount bound** (commit `d6dfed4`): - `COUNT(DISTINCT CASE ...)` bounded to `timestamp >= todayStart` (was 7 days) - CASE expression cannot use an index regardless of window; the bound limits rows scanned ### Dedup and dedup resolvers **`submissionId` token** (commit `5990d96`): - `sessionStorage`-keyed token attached to `form_submit` - 30-second fixed-from-first window prevents re-fire duplicates **Rapid-fire dedup query** (`/api/collect/route.ts`): - `WHERE brandId = ? AND sessionId = ? AND pageUrl = ? AND conversionType = ? AND timestamp >= NOW() - 30s` - Index: `@@index([brandId, sessionId, pageUrl, conversionType, timestamp])` **Phone call dedup** (commit `2d1c1b9`): - Unified across Path A (`/api/collect`) and Path B (`/api/site-tag/conversion`) **Multi-source dedup (CanonicalEvent system)** (`src/lib/services/site-tag-reconciliation.ts`): - `ingestSourceEvent()`: matches existing CanonicalEvent within +-5 min by eventType + pageUrl - `reconcileTrackedEvents()`: fire-and-forget, max 500 TrackedEvents per call, NO date filter (R1) - `getReconciliationSummary()`: all 9 queries bounded to 48h (commit `07e37ef`) **Deterministic dedup clustering** (`src/lib/conversions/dedup.ts`): - SHA1-based `computeDedupKey()` for form/phone events - Survivor selection: method priority -> touchpoint richness -> earliest timestamp - `isDuplicateOf` pointer written; never hard-delete **Cron dedup** (`/api/cron/dedup-conversions`, daily): - Processes unresolved conversions (`isDuplicateOf IS NULL`) - Backfills `classifiedChannel` after dedup ### Connection pool protection - `DATABASE_URL`: `connection_limit=20, pool_timeout=30` - `SITE_TAG_SIDE_EFFECTS_ENABLED` flag (`src/app/api/collect/route.ts` line 43): gates all bounded-async side effects in ingest hot path (commit `8f5df21`) ### Middleware exclusions Ingest beacon paths excluded from Next.js middleware matcher (commit `dad4b60`) to eliminate per-request middleware overhead. --- ## 8. Customer-Facing Surfaces ### Actions a customer can run | Surface | Action | Route | |---------|--------|-------| | Audit page | Trigger full site audit | `POST /api/site-audit/run` | | Technical audit | Run Lighthouse + schema crawl | `POST /api/technical-audit/run` | | AI Strategist | Send message | `POST /api/ai-strategist/chat` | | AI Strategist | Load/save session | `StrategistSession` CRUD | | Site Tag | Install snippet | `GET /api/site-tag/script/[name]` | | Site Tag | Add/remove domain | `PATCH /api/site-tag` action=add_domain / remove_domain | | Site Tag | Toggle active | `PATCH /api/site-tag` action=toggle_status | | Site Tag | Regenerate key | `PATCH /api/site-tag` action=regenerate_key | | Conversion config | Set per-type tier | `POST /api/brands/[id]/conversion-config` | | Keyword Research | Search keywords | `POST /api/keyword-research/search` | | Keyword Research | Analyze gap | `POST /api/keyword-research/analyze` | | Keyword Research | Send to tracker | `POST /api/keyword-research/send-to-tracker` | | Content Brief | Generate brief | `POST /api/content-briefs/generate` | | Schema Generator | Generate JSON-LD | `POST /api/schema/generate-for-brief` | | Execution Hub | Create/update/complete task | `POST/PATCH /api/tasks` | | Integrations | Connect/disconnect | `/api/integrations/connect`, `/disconnect` | | Email Reports | Create/edit schedule | `POST /api/email-reports` | | Alert Rules | Create/edit alert | Alert CRUD endpoints | | White-label | Configure portal | `PUT /api/white-label` | | Client Portal | Invite client user | `POST /api/client-portal/users/invite` | | AI Visibility | Add/manage queries | `/api/ai-visibility/queries` CRUD | | AI Visibility | Trigger check | `POST /api/ai-visibility/check` | | Competitors | Discover / add | `/api/competitors/discover` | | Page Launch Tracker | Mark as launched | `POST /api/page-launches` | | GSC | Backfill history | `POST /api/gsc/backfill` | ### Downloadable reports | Report | Format(s) | Route | |--------|-----------|-------| | Full audit report | PDF (comprehensive 40-50pp + executive 6pp) | `GET /api/admin/audits/[id]/pdf?mode=comprehensive` | | Report Studio custom | PDF, DOCX, Markdown, JSON | `POST /api/report-studio/export` | | Report Studio PPTX | PPTX | `POST /api/report-studio/export/pptx` | | Compliance export | JSON | `GET /api/admin/compliance/events` (admin-only) | | Platform metrics | CSV/JSON | `GET /api/admin/export/platform-metrics` | ### Site Tag event types (74 total) | Category | Events | |----------|--------| | Core | `page_view`, `session_start` | | Forms | `form_start`, `form_submit` | | Phone | `click_to_call`, `sms_click`, `phone_number_visible`, `call_tracking_detected` | | Booking | `booking_submit` | | Video | `video_play` | | E-commerce | `purchase` (Shopify Web Pixel + standard) | | Engagement | `outbound_click`, `file_download`, `pdf_opened`, `content_engagement` | | Competitors | `competitor_visit`, `competitor_referral`, `competitive_journey` | | UX Signals | `rage_click`, `dead_click`, `scroll_bounce` (via SiteEvent) | | Performance | `web_vitals`, `slow_interaction`, `js_error`, `resource_error` | | Custom | `plugin_detected`, `custom_event` | --- ## 9. Schema and Crons ### Prisma models (129 models, 268 indexes) **Core: Users and Orgs** - `User` -- Clerk-synced, email, timezone, emailPrefs (JSON) - `Organization` -- name, slug, plan - `OrgMembership` -- userId + orgId unique - `BrandMembership` -- per-brand role (owner|admin|member|viewer) **Brands** - `Brand` -- domain, industry, locations[], primaryServices[], renderTier, isDemo, healthScore - `BrandIntegration` -- integrationId, credentialsEnc (AES-256-GCM), tokenExpiresAt, status - `BrandConversionConfig` -- per-type isCounted + tierOverride + displayLabel - `BrandProfile` -- logoUrl, brandColors[], brandVoice, extractedAt - `BrandAsset` -- filename, url, type, fileSize, dimensions **Analytics + Metrics** - `MetricSnapshot` -- daily per-brand per-source (ga4|gsc|site_tag|daily|manual|seed) - `ChannelMetric` -- daily per-channel (organic|ai_search|social|gbp|direct) - `ConversionEvent` -- type, channel, value, landingPage **Site Tag (Real-Time)** - `TrackedSite` -- siteId (`ms_` prefix), trackingKey (secret), allowedDomains[], customEvents (JSON), detectedPlatform (JSON: name/version/confidence/category/hosting) - `TrackedSession` -- sessionId (unique), device, browser, country, city, channelGroup, predictedConversionScore - `TrackedEvent` -- eventType, pageUrl, timestamp, synthetic, metadata (JSON) - Indexes: `(trackedSiteId, eventType, timestamp)`, `(trackedSiteId, pageUrl, timestamp)`, `(trackedSiteId, timestamp)`, `(timestamp)` - `SiteConversion` -- conversionType, visitorId, sessionId, pageUrl, formProvider, phoneNumber, touchpoints (JSON), conversionValue, orderId, lineItems (JSON), isDuplicateOf, invalidatedReason, classifiedChannel, submissionId - Indexes: `(brandId, timestamp)`, `(brandId, isDuplicateOf)`, `(brandId, sessionId, pageUrl, conversionType, timestamp)` - `HeatmapEvent` -- visitorId, sessionId, pageUrl, eventType, maxScrollPercent, sectionTimes (JSON), pageWidth/Height - `RealUserMetric` -- lcp, cls, inp, ttfb, fcp, device, connection - `SiteEvent` -- eventType (rage_click|dead_click|hesitation|exit_intent|js_error|outbound_click|...), eventData (JSON) - Indexes: `(trackedSiteId, eventType, timestamp)`, `(sessionId, timestamp)` (commit `ecd5a48`) - `SiteTagDailyRollup` -- brandId, date, pageViews, sessions, conversions, topPages (JSON), byReferrer (JSON), conversionsByType (JSON), conversionsByTier (JSON), eventCountsByType (JSON), outboundClickCount - Unique: `(brandId, date)` - `PageRecord`, `PageDailyMetric`, `NavSnapshot`, `PageSEOSnapshot` **Conversions + Dedup** - `CanonicalEvent` -- eventType, pageUrl, timestamp, confidence (0-100), status (verified|inferred|unmatched) - `SourceEvent` -- source (site_tag|ga4|gtm|gsc|crm), isDuplicate, rawData (JSON) **Audit + Analysis** - `Audit` -- auditToken, sections (JSON), completedAt - `AuditScoreHistory` -- healthScore, errorCount, warningCount - `SiteAuditRun` -- status, taskId (DataForSEO), pagesProcessed, crawlError - `SiteAuditPage` -- url, title, h1, wordCount, statusCode, crawlHealth - `SiteAuditIssue` -- type, severity (error|warning|notice), affectedUrls (JSON), evidence (JSON), status - `TechnicalAudit`, `TechnicalIssue`, `PerformanceAudit`, `PerformancePageScore` **AI Visibility + Market Position** - `AiVisibilityCheck`, `AiVisibilityQuery`, `AiVisibilitySnapshot` - `ProbeRun` -- vertical, locale, engine, status (enum ProbeRunStatus: PENDING|RUNNING|COMPLETED|FAILED), sampleCount - `VisibilityObservation` -- probeRunId, query, engine, brandKey, mentioned, position, cited, sentiment, sourceUrls[], sampleIndex - `CompetitorPanel` -- vertical, locale, members[]; unique `(vertical, locale)` - `QueryCorpusEntry` -- vertical, locale, query, source (enum QuerySource: FRAGMENT_SEEDED|CATEGORY_COVERAGE|CURATED), conversionWeight - `CalibrationRun` -- vertical, version, coefficients (JSON), fittedAt, sampleSize - `MarketPositionSnapshot` -- brandId, vertical, locale, calibratedScore, conversionWeightedSov, rawSov, rankInPanel, panelSize, divergence, stabilityBand, computedAt **Execution + Content** - `Task` -- title, category, status, priority, source, sourceId, dueDate, pipelineStage, briefId, assignedTo - `ContentBrief` -- type (seo|llmo), status, primaryKeyword, sections (JSON), humanizationMode, contentScore - `SchemaMarkup` -- type, markup (JSON-LD), appliedPages (JSON) **Page Lifecycle** - `PageLaunch`, `PageLaunchSnapshot` -- baseline vs current GSC metrics - `DeadPage`, `DeadPageSource` -- 404 detection + redirect suggestions **Competitors** - `Competitor`, `CompetitorSnapshot`, `CompetitorRanking`, `CompetitorPage`, `CompetitorDiscoveryJob` **Backlinks** - `Backlink` -- sourceDomain, anchorText, linkType, toxicityScore, qualityTier, status (active|lost|disavowed) - `BacklinkSnapshot` -- totals, avgDA, toxicCount **GSC + GBP** - `GscQuery`, `GscPage` -- daily click/impression/position - `GbpLocation`, `GbpReview`, `GbpMetric` **Rank Tracking** - `TrackedKeyword` -- keyword, location, device, isActive - `KeywordRanking` -- position, previousPosition, serpFeatures[], source **CRM + Revenue** - `CrmDeal`, `RevenueAttribution`, `UserJourney` **Email + Notifications** - `EmailReportSchedule`, `SentEmailReport`, `AlertRule`, `Notification` **White-Label + Portal** - `WhiteLabelConfig` -- orgId (unique), customDomain, logoUrl, primaryColor, removeMeseoBranding - `CustomDomain` -- hostname (unique), status (enum DomainStatus: PENDING|VERIFIED|FAILED), verificationToken - `ClientPortal` -- brandId (unique), allowedPages[], hideSourceBranding - `ClientUser` -- email, role, status, accessToken **AI Strategist** - `AnalysisJob` -- type, status, prompt, result (JSON), shareToken, clientVersion (JSON), hiddenSections (JSON) - `StrategistSession`, `StrategistMessage` **Logging + Telemetry** - `AiUsageLog` -- provider, model, feature, inputTokens, outputTokens, estimatedCost - `AiInteraction` -- userId, brandId, feature, userPrompt (4KB), aiResponse (16KB), guardrailFlags[], flagSeverity - `PlatformEvent`, `PlatformLog`, `ComplianceEvent` **Predictions + Other** - `Prediction`, `DashboardSnapshot`, `IndustryBenchmark`, `AdCampaignData`, `AdClickEvent`, `AdPlatformIntegration` ### Cron jobs (34 routes under `/api/cron/`) | Route | Schedule | maxDuration | Key models | Notes | |-------|----------|-------------|------------|-------| | `site-tag-rollup` | Hourly `0 * * * *` | 300s | TrackedEvent, SiteTagDailyRollup | Pre-aggregates last 30 days. Force-recomputes today+yesterday. Commit `3d1c64e`. | | `site-audit-cleanup` | Every 5 min | -- | SiteAuditRun | Re-kicks stuck >15 min; fails >2 hours. Commit `6a69290`. | | `rank-check` | Daily | 300s | TrackedKeyword, KeywordRanking | DataForSEO live SERP. Soft cap $5/day, hard cap 100 keywords/brand. | | `ai-visibility` | Weekly `0 0 * * 1` | 300s | AiVisibilityQuery, AiVisibilitySnapshot | 7-source mention checks. | | `signal-detection` | Hourly | -- | MetricSnapshot, ConversionEvent | Anomaly detection -> AlertRule -> Notification. | | `dedup-conversions` | Daily | -- | SiteConversion | Deterministic cluster dedup + classifiedChannel backfill. | | `ai-citation-check` | Weekly | 300s | AiCitation | Citation verification. | | `backlink-sync` | Daily | -- | Backlink, BacklinkSnapshot | DataForSEO new/lost + toxicity scoring. | | `gbp-sync` | Daily | -- | GbpLocation, GbpReview, GbpMetric | Google Business Profile sync. | | `email-reports` | Per schedule (nextSendAt) | -- | EmailReportSchedule, SentEmailReport | Assemble + send scheduled reports. | | `industry-intel` | Daily | -- | IndustryBriefing | External event correlation. | | `competitor-crawl` | Weekly | -- | Competitor, CompetitorSnapshot | Sitemap monitoring, threat scoring. | | `paid-media-rebuild` | Daily | -- | AdCampaignData | Ad spend + conversion reconciliation. | | `track-outcomes` | Daily | -- | Task, RecommendationOutcome | ROI scoring for completed recommendations. | | `ai-strategy-recommendations` | Weekly | -- | AnalysisJob | Re-generate AI recs for all brands. | | `data-retention` | Daily | -- | TrackedEvent, SiteConversion, SiteEvent | Delete rows older than `brand.dataRetentionDays` (default 395 days). | | `nango-sync` | Hourly | -- | BrandIntegration (Nango) | OAuth token refresh. | | `refresh-tokens` | Daily | -- | BrandIntegration (Google) | Google OAuth refresh. | | `check-alerts` | Every 5 min | -- | AlertRule, Notification | Evaluate rules, fire notifications. | | `compute-benchmarks` | Weekly | -- | IndustryBenchmark | Platform-wide industry median aggregation. | | `brand-health` | Daily | -- | Brand (healthScore) | Synthetic health score + email on decline. | | `platform-snapshot` | 1st of month | -- | PlatformSnapshot | Monthly MAU/DAU/churn/feature-adoption archive. | | `cleanup-stale-audits` | Daily | -- | Audit, TechnicalAudit | Delete stale >90 days. | | `precompute-dashboard` | Every 30 min | -- | MetricSnapshot | Pre-aggregate KPIs with budget guard. Commit `c376913`. | | `integration-sync` | Hourly | -- | BrandIntegration, GSC/GA4/GBP | Sync search data from connected integrations. | | `signals-competitor-regional` | Hourly | -- | Competitor, CompetitorSnapshot | Regional threat detection. | | `signals-site-tag-ai` | Hourly | -- | SiteConversion, TrackedSession | AI signal detection from behavioral data. | **Gaps and unknowns:** Schedule values are from comments in each route file; vercel.json was not read. `competitor-discovery-queue` and `brand-sitemap-crawl` exist but their schedules and enabled status were not confirmed. --- ## 10. Risk Register Items not yet fully converted to rollup/bounded paths. ### Active risks **R1 -- `reconcileTrackedEvents` is unbounded (fire-and-forget, pool risk)** - File: `src/lib/services/site-tag-reconciliation.ts` lines 152-188 - Fetches up to 500 `TrackedEvent` rows with NO date filter. For a high-volume brand, the oldest 500 rows are weeks/months old; each unmatched event fires 1 `findFirst` + 1-2 `create` = up to ~1500 serial round-trips. - Status: Fire-and-forget (`void`) so it does not block HTTP response, but holds DB connections. - Fix: Add `timestamp: { gte: window48h }` to the `findMany`. **R2 -- `SourceEvent` lacks a composite `(trackedSiteId, timestamp)` index** - File: `prisma/schema.prisma` - `SourceEvent` only has `@@index([trackedSiteId, source])` and `@@index([trackedSiteId, isDuplicate])`. The 48h bound (`07e37ef`) filters on `timestamp` but without a `(trackedSiteId, timestamp)` index, Postgres must post-filter. - Risk level: Medium (48h bound still dramatically reduces scan; not as fast as it could be). - Fix: Add `@@index([trackedSiteId, timestamp])` to `SourceEvent`. **R3 -- `hourlySample` findMany on TrackedEvent (setup page)** - File: `src/app/api/site-tag/route.ts` - `prisma.trackedEvent.findMany({ where: { trackedSiteId, timestamp >= todayStart }, take: 1000 })`. Runs on every 30s poll. - `(trackedSiteId, timestamp)` index exists so this should be fast, but for very high volume brands the today-bounded scan still reads a large index range before the `take: 1000` stops it. - Risk level: Low. Monitor via `?debugTiming=1`. **R4 -- customEventsList fallback: 90-day `TrackedEvent.groupBy`** - File: `src/app/api/site-tag/route.ts` - If `SiteTagDailyRollup` has no rows (new brand, cron lag), falls back to `trackedEvent.groupBy` over 90 days. - Risk level: Low for established brands. High for new brands if cron is delayed. **R5 -- `domainStatus` per-domain count queries** - File: `src/app/api/site-tag/route.ts` - For each allowed domain: 2 parallel queries (`count` + `findFirst`). With `startsWith` (sargable on `(trackedSiteId, pageUrl, timestamp)` index) and `Promise.all`, this is acceptable for 1-3 domains. - Risk level: Low for 1-3 domains. Medium for brands with 10+ allowed domains. **R6 -- Live `TrackedEvent` aggregate fallback when rollup missing** - File: `src/lib/services/site-tag-analytics.ts` - If `rollupRows.length < expectedDays - 1`, falls back to full-range `TrackedEvent` group-by scans. - Risk level: Low for steady-state brands. Medium for brands recovering from cron failures. **R7 -- `getEventTrends` always runs live (intentional revert)** - Commit: `40990aa revert(getEventTrends): remove rollup early-return, always run live query` - 30-day `TrackedEvent.groupBy` per call. - Risk level: Medium -- depends on call frequency and brand volume. **R8 -- `getSiteTagInstalledAt` and `getSiteTagMaturity` internals not confirmed** - Called in `Promise.all` at end of setup page handler (outside `withDeadline`). - Risk level: UNCONFIRMED. **R9 -- `site-tag-rollup` cron processes ALL brands serially** - File: `src/app/api/cron/site-tag-rollup/route.ts` - Outer loop: `for site of sites`, inner loop: `for date of datesToProcess`. 9 DB round-trips per iteration. At 100 brands x 30 days = 2700 iterations x 9 = 24,300 DB calls within 300s maxDuration. - Risk level: Medium at scale. Would fail silently (partial rollup) if maxDuration hit. ### Resolved risks (this session and prior) - `activePagesCount` 7-day unbounded CASE expression -- resolved `d6dfed4` - `getReconciliationSummary` 9 unbounded full-table scans -- resolved `07e37ef` - `recomputeSessionScore` in ingest hot path -- resolved `accf550` --- ## 11. Patent Map ### Site Tag ID system **Status: WORKING IMPLEMENTATION** - `public/t.js` + source `public/t.source.js` -- served from `/api/site-tag/script/[name]` - Session stitching: `TrackedSession` with `predictedConversionScore` ML field - Dedup: submissionId token, rapid-fire query, SHA1 clustering, `isDuplicateOf` pointer - Attribution: `touchpoints JSON`, `firstTouchChannel`, `lastTouchChannel`, `classifiedChannel` - Platform detection: `detectedPlatform` JSON on `TrackedSite` (name, version, confidence, category, hosting) - Plugin detection: `plugin_detected` events with `plugins[]` payload ### Multi-source dedup + CanonicalEvent **Status: WORKING IMPLEMENTATION** - `CanonicalEvent` + `SourceEvent` models; 5-source reconciliation (site_tag|ga4|gtm|gsc|crm) - `ingestSourceEvent()`: +-5 min match window, confidence boosting (75 + sourceCount*8, max 100) - `computeDedupKey()`: SHA1 hash, deterministic survivor selection - Daily `dedup-conversions` cron for backfill - Note: `reconcileTrackedEvents` still has unbounded oldest-first scan (R1) ### Conversion funnel (multi-tier) **Status: WORKING IMPLEMENTATION** - `CONVERSION_TIERS` in `src/lib/conversions/tiers.ts`: 41 conversion types across 3 tiers (completed|intent|signal) - `BrandConversionConfig`: per-type override (isCounted, tierOverride, displayLabel) - `SiteConversion`: full funnel data including touchpoints, value, orderId, lineItems (JSON) - Configure Conversions modal in Site Tag Analytics UI ### AI Strategist **Status: WORKING IMPLEMENTATION** - `POST /api/ai-strategist/chat` (120s, claude-sonnet-4-6) - Context: brand profile + GSC + Site Tag + AnalysisJob.result + paid media + LLMO - Artifact renderer: JSX dashboard generation in iframe sandbox - Session persistence + guardrail logging ### AI Visibility / LLMO attribution **Status: WORKING IMPLEMENTATION (base), PARTIAL (market position calibration)** - `AiVisibilityCheck`, `AiVisibilityQuery`, `AiVisibilitySnapshot` -- fully wired - Weekly cron, 7-source visibility checker using Anthropic `web_search_20250305` tool - Market Position Block 1+2: schema + corpus seeding implemented (`cdb15a9`, `c735d74`) - Calibration runner producing `MarketPositionSnapshot.calibratedScore`: **schema-only, no runner service confirmed** ### Heatmap **Status: WORKING IMPLEMENTATION** - `HeatmapEvent` model, `POST /api/site-tag/heatmap`, `GET /api/site-tag/analytics/heatmap` - UI tab on Site Tag Analytics ### Execution Hub **Status: WORKING IMPLEMENTATION** - `Task` model: 7 categories, pipeline stage, briefId, assignedTo - Calendar view + pipeline board - AI explanations per task ### Audit Runner (7-phase pipeline) **Status: WORKING IMPLEMENTATION** - 2038-line orchestration, 26 dimension aggregators, 7 synthesizers - $4/$2 cost ceilings; prompt caching; degraded mode - 14-industry lens, vendor confidentiality enforced ### Report Studio **Status: WORKING IMPLEMENTATION** - Section reordering, custom assembly - PDF, DOCX, Markdown, JSON, PPTX exports - `@react-pdf/renderer` for audit PDFs (comprehensive + executive modes) ### Multi-tenant / White-label **Status: WORKING IMPLEMENTATION** - `WhiteLabelConfig`: per-org logo, colors, domain, removeMeseoBranding - `CustomDomain`: PENDING|VERIFIED|FAILED with Vercel domain integration - `ClientPortal`: allowedPages[], hideSourceBranding, per-user roles ### Local GEO **Status: SCHEMA + CRON READY, UI STUBBED** - `GbpLocation`, `GbpReview`, `GbpMetric` + daily `gbp-sync` cron active - `/local-geo` intentionally shows "Coming Soon" - Local ranking and AI search citation at location level: ABSENT **Gaps and unknowns:** - Market Position calibration runner: Block 3+ (models + seeding exist, runner not found) - `PageLaunchSnapshot` data population mechanism (GSC direct vs MetricSnapshot): UNCONFIRMED - Inngest: infrastructure only, no production functions beyond ping - Voice search synthesizer call path not fully traced from audit-runner.ts --- *End of inventory. All code paths confirmed via file reads or agent exploration unless marked UNCONFIRMED or ABSENT. Commit hashes reference dmazzei-star/meseo main branch as of 2026-06-10 HEAD 07e37ef.*