AI Microservices Architecture

Learn design patterns for AI microservices including service boundaries, communication protocols, data flow, and deployment topologies.

Defining Service Boundaries

The most important architectural decision is where to draw service boundaries. For AI systems, consider these dimensions:

• Model boundary: Each ML model or model family becomes its own service (recommendation service, NLP service, vision service).
• Pipeline stage: Separate preprocessing, inference, and postprocessing into distinct services.
• Data domain: Group services around business domains (user service, product service, content service).
• Compute requirement: Separate GPU-intensive services from CPU-only services for efficient resource allocation.

Communication Patterns

# gRPC service definition for model serving
syntax = "proto3";

service ModelService {
  // Synchronous prediction
  rpc Predict(PredictRequest) returns (PredictResponse);

  // Streaming prediction for LLMs
  rpc StreamPredict(PredictRequest) returns (stream PredictChunk);

  // Health check
  rpc HealthCheck(Empty) returns (HealthStatus);
}

message PredictRequest {
  string model_id = 1;
  bytes input_data = 2;
  map<string, string> parameters = 3;
}

message PredictResponse {
  bytes output_data = 1;
  float latency_ms = 2;
  TokenUsage usage = 3;
}

Design Patterns for AI Microservices

Kubernetes Deployment

Kubernetes is the standard platform for orchestrating AI microservices:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sentiment-model
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sentiment-model
  template:
    metadata:
      labels:
        app: sentiment-model
    spec:
      containers:
      - name: model-server
        image: ml-registry/sentiment:v2.1
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: "8Gi"
          requests:
            memory: "4Gi"
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 10