Kubernetes monitoring

Metrics, logs, Events, alerts, and the customer infrastructure responsibility boundary.

Monitoring setup: expected result

After setup, Prometheus must report exactly eight MassAccess targets as UP, Pod logs must reach customer storage, and alert rules must load only when every source metric exists.

1. Confirm that installation completed without FAIL and healthcheck exits with code 0
2. Verify the current kube context and namespace, then run the command below: it must return archiver, auditor, bot-max, bot-telegram, core, gateway, webhook, and worker
3. Inspect configuration/installation-config.yaml: the current package generator fixes metricsDiscovery.enabled=true and serviceMonitor.enabled=false
4. Choose the network path before downloading the package: for Prometheus in another namespace, select infrastructure-managed NetworkPolicy in the panel and prepare an exact ingress allowance

kubectl get service \
  --namespace <namespace> \
  --selector massaccess.net/metrics=true \
  --output custom-columns=NAME:.metadata.name,PORTS:.spec.ports[*].port

1. Allow Prometheus to reach the endpoints

With chart-managed NetworkPolicy, ingress is limited to MassAccess Pods in the same namespace; Prometheus in a separate monitoring namespace cannot scrape metrics. For that topology, select infrastructure as the NetworkPolicy owner in the panel, download the package again, and add this allowance on top of a complete customer-owned policy set after replacing the three placeholders.

• Replace <namespace> and <monitoring-namespace> with exact non-system namespace names
• Replace <collector-label-key> and <collector-label-value> with a stable label selecting only your Prometheus Pods; do not allow the entire monitoring namespace
• Do not apply this example as the only policy: infrastructure must separately preserve required internal, edge, DNS, and egress flows for MassAccess
• Review the manifest in the infrastructure repository before applying it; then prove an allowed scrape, working internal flows, and a denied connection from an unrelated Pod

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-massaccess-metrics
  namespace: <namespace>
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: massaccess
      app.kubernetes.io/instance: massaccess
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: <monitoring-namespace>
          podSelector:
            matchLabels:
              <collector-label-key>: <collector-label-value>
      ports:
        - {protocol: TCP, port: 8001}
        - {protocol: TCP, port: 8002}
        - {protocol: TCP, port: 8003}
        - {protocol: TCP, port: 8004}
        - {protocol: TCP, port: 8005}
        - {protocol: TCP, port: 8006}
        - {protocol: TCP, port: 10002}

2. Add a job to the existing Prometheus

Add this scrape_config to managed Prometheus configuration and replace <namespace>. The job selects only Services with massaccess.net/metrics=true and only health or admin ports, excluding frontend and the gateway public port. The Prometheus ServiceAccount needs get/list/watch for Services in the target namespace. Validate the configuration with the validator supplied by your Prometheus distribution and perform a controlled reload.

scrape_configs:
  - job_name: massaccess
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: service
        namespaces:
          names: [<namespace>]
    relabel_configs:
      - action: keep
        source_labels: [__meta_kubernetes_service_label_massaccess_net_metrics]
        regex: "true"
      - action: keep
        source_labels: [__meta_kubernetes_service_port_name]
        regex: "health|admin"
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: service
      - source_labels:
          [__meta_kubernetes_service_label_app_kubernetes_io_component]
        target_label: component

Application metrics endpoints

These eight targets must appear in Prometheus. The application promises no other metrics endpoints.

3. Verify the result in Prometheus

Run the three PromQL queries in order. Expected values are 8 discovered targets, a minimum up value of 1, and 8 unique components. Any other result means setup is incomplete.

1. Open the Targets page: job massaccess must have eight targets with no duplicate targets or scrape errors
2. Confirm that namespace, service, and component labels are populated and contain no company, license, or customer identifier
3. For a DOWN target, check Service DNS, the selected port, NetworkPolicy, and the /metrics response first; do not increase scrape_timeout before finding the cause

count(up{job="massaccess", namespace="<namespace>"})
min(up{job="massaccess", namespace="<namespace>"})
count(count by (component) (up{job="massaccess", namespace="<namespace>"}))

4. Connect logs and Kubernetes Events

Configure the customer's existing DaemonSet or agent to collect stdout/stderr from containers with app.kubernetes.io/part-of=massaccess. The package ships no separate log agent.

• Add namespace, pod, container, and app.kubernetes.io/component to records; use event_id, correlation_id, and parent_event_id only where a service actually emits them
• Never use license keys, company or customer identifiers, tokens, DSNs, Secret values, or webhook payloads as fields or labels
• For Kubernetes Events retain type, reason, object, and a bounded category; the free-form message may contain infrastructure details
• Set retention, access controls, and volume limits under customer policy; prove a test request can be found by component and correlation_id
• Collect Kubernetes Events as a separate stream and alert on FailedScheduling, FailedMount, ImagePullBackOff, and OOMKilled

5. Load rules and route alerts

Extract alerts.yaml from the downloaded chart, validate it with promtool, and load it into your rule loader. Do not load a rule until its data source exists. Configure the recipient, deduplication, repeat interval, and a link to the customer's internal runbook.

• MassAccessKubernetesWorkloadUnavailable, RestartLoop, and JobFailed require kube-state-metrics series with Kubernetes labels
• MassAccessKubernetesPersistentVolumePressure requires kubelet_volume_stats; prove that the CSI and kubelet publish them
• MassAccessExternalDependencyProbeFailed requires a separate customer-owned authenticated probe with probe_success, massaccess_signal=external-dependency, and a bounded dependency label
• LicenseHeartbeatLost, LicenseInactive, and MeteringFailure use application metrics from the configured massaccess job

tar -xOf release/charts/massaccess-<chart-version>.tgz \
  massaccess/files/observability/alerts.yaml \
  > /secure/path/massaccess-alerts.yaml

promtool check rules /secure/path/massaccess-alerts.yaml

6. Acceptance criteria

• Prometheus reports 8/8 targets UP for at least two consecutive scrape intervals
• A controlled stop of one test Pod appears in metrics, logs, and an allowed test alert; after recovery the target returns to UP
• An unrelated Pod cannot read /metrics while Prometheus can
• Logs and labels contain no Secrets, tokens, DSNs, license/company/customer identifiers, or message payloads
• healthcheck and status pass; a local support-bundle is created with mode 0600, inspected by the operator, and not sent automatically

./bin/healthcheck.sh --namespace <namespace>
./bin/massaccess-k8s status --namespace <namespace>
./bin/massaccess-k8s support-bundle --namespace <namespace>