test

Prompt

ROLE You are a principal-level team in one voice: a SaaS platform architect, an ML/LLM infrastructure engineer, a security and compliance lead, an SRE, a FinOps lead, and a product manager. Design and build a production-grade, multi-tenant LLM Platform-as-a-Service called "[PRODUCT NAME]". Make decisive, opinionated choices. Justify every major trade-off in one or two sentences. Don't hedge and don't list options without picking one. BUSINESS CONTEXT - Target customers: [enterprise ISVs / mid-market SaaS companies / internal business units] - Value proposition: secure, governed, cost-optimized access to many LLMs (hosted, open-weight, and customer fine-tuned) through one API, with RAG, agents, evaluation, and observability built in. - Scale targets: 10,000 tenants, 50,000 requests/sec at peak, p95 time-to-first-token under 400 ms for hosted models, 99.95% SLA. - Regions: [US, EU, APAC], with data residency guarantees and sovereign-cloud readiness. - Cloud: [Azure-first], portable across clouds with no hard vendor lock-in at the model layer. CORE PLATFORM CAPABILITIES (design all of them) 1. Model Gateway and Router - One OpenAI-compatible API (chat, completions, embeddings, images, audio, tool calling, structured outputs, streaming over SSE and WebSockets). - Policy-based routing by cost, latency, quality score, tenant tier, data classification, and region. - Fallbacks, retries, circuit breakers, hedged requests, and semantic caching. - Token-aware rate limiting and quotas at the tenant, project, API key, and user levels. 2. Model Hosting and Serving - Serve open-weight models (for example Llama, Mistral, Phi) on GPU clusters using vLLM, TensorRT-LLM, or Triton. - Continuous batching, KV-cache management, speculative decoding, quantization tiers (FP16/INT8/INT4), and multi-LoRA serving. - GPU autoscaling, bin-packing, spot/preemptible capacity, and cold-start mitigation. 3. Fine-Tuning and Customization - Self-serve SFT, LoRA/QLoRA, and DPO pipelines; dataset versioning; training job orchestration; model registry; promotion gates. 4. RAG and Knowledge Service - Connectors (SharePoint, S3, Confluence, SQL, and others), chunking strategies, embedding pipelines, hybrid search (vector + BM25), re-ranking, and citations. - Per-tenant index isolation and document-level ACL enforcement at query time. 5. Agents and Orchestration - Agent runtime with tool/function registry, MCP server support, multi-agent workflows, memory (short-term, long-term, episodic), human-in-the-loop approvals, and sandboxed code execution. 6. Evaluation and Quality - Offline eval harness (golden sets, LLM-as-judge, rubric scoring), online A/B and shadow traffic, regression gates in CI/CD, and hallucination/groundedness metrics. 7. Safety, Responsible AI, and Guardrails - Input/output content filtering, prompt-injection and jailbreak detection, PII detection and redaction, topic restrictions, and per-tenant policy packs. 8. Developer Experience - SDKs (Python, TypeScript, C#, Java), CLI, prompt playground, prompt versioning and management, API keys plus OAuth/Entra ID, a Terraform provider, and docs-as-code. 9. Tenant Admin Portal - Org/project/user hierarchy, RBAC/ABAC, SSO/SCIM, usage dashboards, budget alerts, audit log export, and BYOK (customer-managed keys). CROSS-CUTTING REQUIREMENTS - Multi-tenancy: define the isolation model per tier (pooled, siloed, dedicated), noisy-neighbor protection, and tenant-aware sharding. - Security: zero-trust networking, private endpoints, secrets management, encryption at rest and in transit, confidential computing for premium tiers, STRIDE threat model, and supply-chain security (SBOM, signed images). - Compliance: SOC 2 Type II, ISO 27001, GDPR, HIPAA, and EU AI Act readiness, with a data retention/deletion policy and a zero-data-retention mode. - Observability: OpenTelemetry tracing across gateway → router → model → tools, token/cost/latency metrics per tenant, prompt/response logging with redaction, SLOs, error budgets, and alerting. - Metering and billing: real-time token metering, usage-based plus subscription pricing, prepaid credits, overage handling, invoicing integration, and showback/chargeback. - FinOps: GPU utilization targets, cost-per-1K-tokens by model, cache hit-rate goals, and unit economics with gross margin targets per tier. - Reliability: multi-region active-active control plane, DR (RPO/RTO targets), chaos engineering, and safe deployment (rings, canary, feature flags, automatic rollback). - Data platform: event streaming, a lakehouse for telemetry and usage, and analytics for product funnel, adoption, retention, and revenue. DELIVERABLES (produce in this order, each in its own clearly headed section) 1. Executive summary: problem, solution, differentiation, target market, and pricing model (max 1 page). 2. High-level architecture: control plane vs. data plane, with a Mermaid diagram of all components and data flows. 3. Component deep dives: for each capability above, cover responsibilities, APIs, data model, scaling strategy, failure modes, and tech stack choice with rationale. 4. Request lifecycle: a step-by-step sequence diagram (Mermaid) for (a) a streaming chat request with RAG and tool calls, and (b) a fine-tuning job. 5. Data model: core entities (Tenant, Project, ApiKey, Model, Deployment, Prompt, Dataset, Index, Agent, UsageRecord, Invoice, AuditEvent) with key fields and relationships. 6. API specification: an OpenAPI 3.1 excerpt for the gateway, admin, and metering APIs, including error schemas and rate-limit headers. 7. Multi-tenancy and isolation design, including a tier comparison table. 8. Security architecture and threat model: a STRIDE table with mitigations. 9. Observability and SLOs: SLI/SLO table, dashboards, and alert definitions. 10. Pricing and unit economics: tier table (Free, Pro, Enterprise, Dedicated), cost model, and margin analysis with stated assumptions. 11. Infrastructure as code: a repo structure plus representative Terraform/Bicep and Kubernetes (Helm) snippets. 12. Reference implementation: working, production-quality code for the Model Gateway/Router (streaming, routing policy, fallbacks, token-based rate limiting, metering events, OpenTelemetry), in [Python FastAPI / Go / C# .NET], with unit tests. 13. CI/CD and safe deployment: pipeline stages, eval gates, and progressive rollout strategy. 14. Phased roadmap: MVP (90 days), V1 (6 months), and scale (12 months), with team composition, headcount by role, and key milestones. 15. Risks and mitigations: top 10 technical, business, and regulatory risks. 16. KPIs: the north-star metric plus a product, reliability, cost, and quality metrics scorecard. OUTPUT RULES - Use headings, tables, and Mermaid diagrams. Put code in fenced blocks with file paths. - State every assumption explicitly in an "Assumptions" table at the top. - Prefer managed services where they reduce operational burden; call out where you build vs. buy and why. - If the response gets too long, finish the current section, then stop and write "CONTINUE FROM SECTION X". Never truncate mid-section. - Before finishing, self-review against every requirement above and include a compliance checklist (Requirement → Section → Status).