# Internal Developer Platform

Developer experience, self-service, and internal tooling for AI platform teams: golden paths, model serving as a service, prompt management, and cost showback.

## Platform Capabilities

| Capability | Purpose | Example Tools |
|---|---|---|
| **Model Serving as a Service** | Managed inference endpoints abstracting infrastructure | Triton, vLLM on K8s, SageMaker Endpoints |
| **Prompt Management** | Versioned, reviewed prompt templates shared across teams | LangSmith, PromptLayer, custom registry |
| **Cost Showback** | Attributing AI inference/training costs to teams/products | Tagged billing, Kubecost, CloudHealth |
| **Golden Paths** | Paved-road templates for common AI workloads | RAG template, fine-tuning template, eval template |
| **Self-Service Evals** | Teams run their own evals without platform team involvement | LangSmith, Arize, custom eval harness |
| **IDP** | Self-service platform for AI teams to deploy models and apps | Backstage, Humanitec, AWS Proton |

## Golden Path: RAG Workload

```yaml
# Paved-road RAG deployment template
apiVersion: platform.ai/v1
kind: RAGWorkload
metadata:
  name: customer-docs-rag
  team: support
spec:
  embedding:
    model: BAAI/bge-small-en-v1.5
    vector_store: pgvector
  retrieval:
    top_k: 5
    reranker: cross-encoder/ms-marco-MiniLM-L-6-v2
  generation:
    model: meta-llama/Llama-3.1-8B-Instruct
    serving_engine: vllm
    replicas: 2
  guardrails:
    input: [pii, jailbreak]
    output: [toxicity, hallucination]
  eval:
    enabled: true
    judge_model: gpt-4o-mini
    sample_rate: 0.1
  cost:
    budget_usd_per_month: 500
    alert_threshold: 0.8
```

## Decision Guidance

| Situation | Recommended Approach |
|---|---|
| Many AI teams, one platform | Build an IDP with golden paths for RAG and fine-tuning |
| Cost visibility needed | Tag inference/training spend by team; monthly showback |
| Prompt sprawl | Central prompt registry with versioning and review |
| Repeated infra setup | Paved-road templates (Terraform/Helm) for common workloads |
| Platform team bottleneck | Self-service evals and deployment; platform enables, doesn't gate |

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Every team reinvents model serving | Golden path: one supported serving stack |
| No cost visibility | Tag all AI spend; showback to teams |
| Prompts hardcoded in apps | Central prompt registry with versioning |
| Platform team becomes bottleneck | Self-service evals and deployment; platform enables, doesn't gate |
| No eval standardization | Provide golden eval path with default metrics |

## Related Notes

- [[Platform/Prompt-Management-And-Cost-Showback.md]] — Prompt registry and cost attribution
- [[AgenticEngineering/Production/Deployment-Architecture.md]] — Serverless, containers, edge, hybrid
- [[AgenticEngineering/Production/Cost-Optimization.md]] — Caching, model selection, token budgets
- [[MLOps/]] — Deployment and training pipelines
- [[Inference/]] — Model serving and inference engines
- [[Observability/]] — Monitoring AI systems

## Prompt Registry Example

```yaml
# prompts/registry.yaml
prompts:
  - id: support-faq-v2
    name: Customer Support FAQ
    version: "2"
    owner: support-team
    stage: Production
    template: |
      You are a helpful customer support agent. Answer the user's question
      using only the provided context. If you don't know, say so.
      
      Context: {{context}}
      Question: {{question}}
      
      Answer:
    variables:
      - context
      - question
    guardrails:
      - pii_detection
      - toxicity_check
    eval:
      metrics: [faithfulness, relevance]
      threshold: 0.85
    created_at: "2026-01-10"
    updated_at: "2026-01-15"
```

## Cost Showback Example

```sql
-- Monthly AI cost by team
SELECT
  team,
  model,
  SUM(prompt_tokens) as total_prompt_tokens,
  SUM(completion_tokens) as total_completion_tokens,
  SUM(cost_usd) as total_cost_usd,
  COUNT(*) as request_count
FROM ai_inference_events
WHERE event_date >= DATE_TRUNC('month', CURRENT_DATE)
GROUP BY team, model
ORDER BY total_cost_usd DESC;
```

## Sources

* Backstage, Humanitec, AWS Proton, Google Cloud AI Platform, Azure ML, Internal Developer Platform patterns, Team Topologies, platform engineering community.*