Internal Developer Platform

Developer experience, self-service, and internal tooling for AI platform teams: golden paths, model serving as a service, prompt management, and cost showback.

Platform Capabilities

Capability	Purpose	Example Tools
Model Serving as a Service	Managed inference endpoints abstracting infrastructure	Triton, vLLM on K8s, SageMaker Endpoints
Prompt Management	Versioned, reviewed prompt templates shared across teams	LangSmith, PromptLayer, custom registry
Cost Showback	Attributing AI inference/training costs to teams/products	Tagged billing, Kubecost, CloudHealth
Golden Paths	Paved-road templates for common AI workloads	RAG template, fine-tuning template, eval template
Self-Service Evals	Teams run their own evals without platform team involvement	LangSmith, Arize, custom eval harness
IDP	Self-service platform for AI teams to deploy models and apps	Backstage, Humanitec, AWS Proton

Golden Path: RAG Workload

# Paved-road RAG deployment template
apiVersion: platform.ai/v1
kind: RAGWorkload
metadata:
  name: customer-docs-rag
  team: support
spec:
  embedding:
    model: BAAI/bge-small-en-v1.5
    vector_store: pgvector
  retrieval:
    top_k: 5
    reranker: cross-encoder/ms-marco-MiniLM-L-6-v2
  generation:
    model: meta-llama/Llama-3.1-8B-Instruct
    serving_engine: vllm
    replicas: 2
  guardrails:
    input: [pii, jailbreak]
    output: [toxicity, hallucination]
  eval:
    enabled: true
    judge_model: gpt-4o-mini
    sample_rate: 0.1
  cost:
    budget_usd_per_month: 500
    alert_threshold: 0.8

Decision Guidance

Situation	Recommended Approach
Many AI teams, one platform	Build an IDP with golden paths for RAG and fine-tuning
Cost visibility needed	Tag inference/training spend by team; monthly showback
Prompt sprawl	Central prompt registry with versioning and review
Repeated infra setup	Paved-road templates (Terraform/Helm) for common workloads
Platform team bottleneck	Self-service evals and deployment; platform enables, doesn't gate

Anti-Patterns

Anti-Pattern	Fix
Every team reinvents model serving	Golden path: one supported serving stack
No cost visibility	Tag all AI spend; showback to teams
Prompts hardcoded in apps	Central prompt registry with versioning
Platform team becomes bottleneck	Self-service evals and deployment; platform enables, doesn't gate
No eval standardization	Provide golden eval path with default metrics

Prompt Management And Cost Showback — Prompt registry and cost attribution
Deployment Architecture — Serverless, containers, edge, hybrid
Cost Optimization — Caching, model selection, token budgets
MLOps — Deployment and training pipelines
Inference — Model serving and inference engines
Observability — Monitoring AI systems

Prompt Registry Example

# prompts/registry.yaml
prompts:
  - id: support-faq-v2
    name: Customer Support FAQ
    version: "2"
    owner: support-team
    stage: Production
    template: |
      You are a helpful customer support agent. Answer the user's question
      using only the provided context. If you don't know, say so.

      Context: {{context}}
      Question: {{question}}

      Answer:
    variables:
      - context
      - question
    guardrails:
      - pii_detection
      - toxicity_check
    eval:
      metrics: [faithfulness, relevance]
      threshold: 0.85
    created_at: "2026-01-10"
    updated_at: "2026-01-15"

Cost Showback Example

-- Monthly AI cost by team
SELECT
  team,
  model,
  SUM(prompt_tokens) as total_prompt_tokens,
  SUM(completion_tokens) as total_completion_tokens,
  SUM(cost_usd) as total_cost_usd,
  COUNT(*) as request_count
FROM ai_inference_events
WHERE event_date >= DATE_TRUNC('month', CURRENT_DATE)
GROUP BY team, model
ORDER BY total_cost_usd DESC;

Sources

Backstage, Humanitec, AWS Proton, Google Cloud AI Platform, Azure ML, Internal Developer Platform patterns, Team Topologies, platform engineering community.*

From the Systems Engineering research vault · CC BY 4.0 · all notes · raw markdown