Hey everybody! It’s my first time designing a data warehouse from zero. This warehouse if for a mid-size platform, and I have a few architectural questions that I’d love to get outside perspective on before we lock anything in. Context: We run a real-time operational platform. Think incident/event ingestion, multi-tenant, with a “core product” domain (events, users, locations, message logs) and strict data-protection regulation in play. Storage is mainly NoSQL (DynamoDB-style single-table design) for the high write operational core, but also relational Postgres for the tenant/identity hierarchy. No analytics layer exists today, this is a from-scratch build, likely on a lakehouse stack (Databricks/Delta-style bronze-silver-gold). Separately, the organization also generates a lot of internal data that has nothing to do with the product: Claude Code usage telemetry (tokens, cost, model, sessions), GitHub activity, ADRs/documentation, sprint/ticket data. We want the warehouse to eventually be the “single source of truth” for both. Two open questions I keep going back and forth on:
- One warehouse or two (product vs. org)? Arguments for one: simpler governance, easier cross-domain joins (e.g., “did response-time quality change after a given deploy or AI-assisted change”). Arguments for two: product data carries heavy compliance/retention obligations (multi-year legal retention, PII rules), while internal telemetry doesn’t. Mixing them under one retention/masking policy feels wrong. Im currently leaning for two separate catalogs/warehouses: one for product and one for org, with a thin cross-domain layer on top for joins when needed. Curious if anyone has regretted going either direction.
- Mask PII before it lands in the bronze/raw layer, or after? Option A: mask (hash phone numbers, fuzz location, redact free text) before landing in the raw layer. This means less PII sitting around needing encryption/retention/right-to-erasure handling. Option B: keep raw layer unmasked, and apply masking downstream (silver/gold) with RBAC gating who can query raw. Pro: preserves audit fidelity. Con: bigger PII footprint to secure in the raw layer. We already have a separate immutable “audit” store (hashed, tamper-evident) for legal/compliance purposes that predates this warehouse effort, which raises a third question of whether that audit store is the bronze layer, or whether bronze should be a separate derivative export from it. Would love to hear: If you’ve built a warehouse that spans both product/operational data and internal org telemetry. Did you keep them separate or merged, and why? Where you draw the line on pre-masking vs. post-masking PII, especially under strict data-protection regimes. Whether you’ve ever unified an existing audit/compliance store with your analytics bronze layer, and how that went. Appreciate any feedback. I am trying to avoid designing myself into a corner before this scales. submitted by /u/dylannalex01
Originally posted by u/dylannalex01 on r/ArtificialInteligence