Kubernetes monitoring

Monitoring setup: expected result

Use your existing monitoring and logging systems. The application chart exposes eight Service /metrics endpoints. Local values support observability.metricsDiscovery and an optional serviceMonitor when its CRD is available.

1. First verify the application with helm status/test and Pod state.
2. Set observability.metricsDiscovery.enabled=true in work/app-values.yaml; enable ServiceMonitor only with an installed customer-managed CRD.
3. Check discovery with the command below and Prometheus network access to all eight endpoints.

kubectl get service \
  --namespace <namespace> \
  --selector massaccess.net/metrics=true \
  --output custom-columns=NAME:.metadata.name,PORTS:.spec.ports[*].port

1. Allow Prometheus to reach the endpoints

With chart-managed NetworkPolicy, Prometheus in another namespace needs a separate allowance. Choose infrastructure-managed networking in local values and prepare the full policy set. The example below allows only metrics ingress from the selected collector; substitute namespaces and a stable Pod label.

• Replace <namespace> and <monitoring-namespace> with exact non-system namespace names
• Replace <collector-label-key> and <collector-label-value> with a stable label selecting only your Prometheus Pods; do not allow the entire monitoring namespace
• Do not apply this example as the only policy: infrastructure must separately preserve required internal, edge, DNS, and egress flows for MassAccess
• Review the manifest in the infrastructure repository before applying it; then prove an allowed scrape, working internal flows, and a denied connection from an unrelated Pod

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-massaccess-metrics
  namespace: <namespace>
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: massaccess
      app.kubernetes.io/instance: massaccess
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: <monitoring-namespace>
          podSelector:
            matchLabels:
              <collector-label-key>: <collector-label-value>
      ports:
        - {protocol: TCP, port: 8001}
        - {protocol: TCP, port: 8002}
        - {protocol: TCP, port: 8003}
        - {protocol: TCP, port: 8004}
        - {protocol: TCP, port: 8005}
        - {protocol: TCP, port: 8006}
        - {protocol: TCP, port: 10002}

2. Add a job to the existing Prometheus

Add this scrape_config to managed Prometheus configuration and replace <namespace>. The job selects only Services with massaccess.net/metrics=true and only health or admin ports, excluding frontend and the gateway public port. The Prometheus ServiceAccount needs get/list/watch for Services in the target namespace. Validate the configuration with the validator supplied by your Prometheus distribution and perform a controlled reload.

scrape_configs:
  - job_name: massaccess
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: service
        namespaces:
          names: [<namespace>]
    relabel_configs:
      - action: keep
        source_labels: [__meta_kubernetes_service_label_massaccess_net_metrics]
        regex: "true"
      - action: keep
        source_labels: [__meta_kubernetes_service_port_name]
        regex: "health|admin"
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: service
      - source_labels:
          [__meta_kubernetes_service_label_app_kubernetes_io_component]
        target_label: component

Application metrics endpoints

These eight targets must appear in Prometheus. The application promises no other metrics endpoints.

3. Verify the result in Prometheus

Run the three PromQL queries in order. Expected values are 8 discovered targets, a minimum up value of 1, and 8 unique components. Any other result means setup is incomplete.

1. Open the Targets page: job massaccess must have eight targets with no duplicate targets or scrape errors
2. Confirm that namespace, service, and component labels are populated and contain no company, license, or customer identifier
3. For a DOWN target, check Service DNS, the selected port, NetworkPolicy, and the /metrics response first; do not increase scrape_timeout before finding the cause

count(up{job="massaccess", namespace="<namespace>"})
min(up{job="massaccess", namespace="<namespace>"})
count(count by (component) (up{job="massaccess", namespace="<namespace>"}))

4. Connect logs and Kubernetes Events

Configure the customer's existing DaemonSet or agent to collect stdout/stderr from containers with app.kubernetes.io/part-of=massaccess. The package ships no separate log agent.

• Add namespace, pod, container, and app.kubernetes.io/component to records; use event_id, correlation_id, and parent_event_id only where a service actually emits them
• Never use license keys, company or customer identifiers, tokens, DSNs, Secret values, or webhook payloads as fields or labels
• For Kubernetes Events retain type, reason, object, and a bounded category; the free-form message may contain infrastructure details
• Set retention, access controls, and volume limits under customer policy; prove a test request can be found by component and correlation_id
• Collect Kubernetes Events as a separate stream and alert on FailedScheduling, FailedMount, ImagePullBackOff, and OOMKilled

5. Load rules and route alerts

Prepare your own rules file and validate it with promtool before loading it. Use available metric sources only and configure recipients, deduplication and a test notification. The materials do not include an alerts.yaml file.

• MassAccessKubernetesWorkloadUnavailable, RestartLoop, and JobFailed require kube-state-metrics series with Kubernetes labels
• MassAccessKubernetesPersistentVolumePressure requires kubelet_volume_stats; prove that the CSI and kubelet publish them
• MassAccessExternalDependencyProbeFailed requires a separate customer-owned authenticated probe with probe_success, massaccess_signal=external-dependency, and a bounded dependency label
• LicenseHeartbeatLost, LicenseInactive, and MeteringFailure use application metrics from the configured massaccess job

promtool check rules /secure/path/customer-massaccess-alerts.yaml

6. Acceptance criteria

• Prometheus shows eight UP targets for at least two intervals, with correct component/namespace/service labels.
• In an approved test window verify alert delivery and recovery, allowed scraping and denied access from an unrelated Pod.
• Logs can be found by component, have agreed retention and do not expose tokens, DSNs, Secrets or message contents.
• Helm tests and application checks pass; the operator reviews and redacts diagnostic records before sharing.

helm --kube-context "$MA_CONTEXT" status massaccess -n "$MA_NAMESPACE"
helm --kube-context "$MA_CONTEXT" test massaccess -n "$MA_NAMESPACE" --logs
kubectl --context "$MA_CONTEXT" -n "$MA_NAMESPACE" get deployment,statefulset,pod,service,job,pvc