The distinction between custom metrics and external metrics in Kubernetes HPA is important as they serve different purposes and have different sources:

Custom Metrics vs External Metrics

Custom Metrics:

Custom metrics are metrics collected from workloads running inside your Kubernetes cluster. They're typically:

• Directly associated with Kubernetes objects (pods, deployments, services)
• Collected by in-cluster monitoring systems (like Prometheus)
• Referenced by object name or label selector

Example of a Custom Metric in HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods  # This indicates a custom metric related to pods
    pods:
      metric:
        name: http_requests_per_second  # Metric collected from pods
      target:
        type: AverageValue
        averageValue: 1k

In this example, the metric is directly collected from the pods in the web-app deployment.

External Metrics:

External metrics come from sources outside your Kubernetes cluster. They're typically:

• Not tied to any Kubernetes object
• Collected from external systems (cloud provider metrics, SaaS monitoring, etc.)
• Independent of your cluster's internal state
• Selected purely by metric name and optional labels

Example of an External Metric in HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-processor-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-processor
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: External  # This indicates an external metric
    external:
      metric:
        name: sqs_messages_visible  # Metric from AWS SQS
        selector:
          matchLabels:
            queue_name: orders-queue
      target:
        type: AverageValue
        averageValue: 30

In this example, the metric comes from AWS SQS (a service outside the cluster) and has no direct relationship to any Kubernetes object.

Real-world examples:

Custom Metrics:

• Number of HTTP requests handled by your application pods
• JVM heap usage in your Java applications
• Queue length in an in-cluster message queue
• Number of active WebSocket connections

External Metrics:

• AWS SQS queue depth
• Azure Service Bus message count
• Google Cloud Pub/Sub subscription backlog
• Stripe API request rate
• External load balancer request rate

Implementation details:

1. Custom Metrics Adapter:

   • Connects to in-cluster monitoring (e.g., Prometheus)
   • Exposes metrics via the custom.metrics.k8s.io API
   • Example: prometheus-adapter, Stackdriver adapter

2. External Metrics Adapter:

   • Connects to external monitoring/metric systems
   • Exposes metrics via the external.metrics.k8s.io API
   • Example: AWS metrics adapter, GCP stackdriver adapter

The primary distinction is whether the metric originates from within your Kubernetes cluster (custom) or from an external system (external). This separation allows for different authentication, access patterns, and data models appropriate to each source type.