Skip to content

Prometheus Probe

The prometheus probe allows users to run Prometheus queries and match the resulting output against specific conditions. The intent behind this probe is to allow users to define metrics-based SLOs in a declarative way and determine the experiment verdict based on its success. The probe runs the query on a Prometheus server defined by the endpoint, and checks whether the output satisfies the specified criteria. It can be executed by setting type as promProbe inside .spec.experiments[].spec.probe.

View the prometheus probe schema

Field .name
Description Flag to hold the name of the probe
Type Mandatory
Range n/a (type: string)
Notes The .name holds the name of the probe. It can be set based on the usecase

Field .type
Description Flag to hold the type of the probe
Type Mandatory
Range httpProbe, k8sProbe, cmdProbe, promProbe
Notes The .type supports four type of probes. It can one of the httpProbe, k8sProbe, cmdProbe, promProbe

Field .mode
Description Flag to hold the mode of the probe
Type Mandatory
Range SOT, EOT, Edge, Continuous, OnChaos
Notes The .mode supports five modes of probes. It can one of the SOT, EOT, Edge, Continuous, OnChaos

Field .promProbe/inputs.endpoint
Description Flag to hold the prometheus endpoints for the promProbe
Type Mandatory
Range n/a {type: string}
Notes The .promProbe/inputs.endpoint contains the prometheus endpoints

Field .promProbe/inputs.query
Description Flag to hold the promql query for the promProbe
Type Mandatory
Range n/a {type: string}
Notes The .promProbe/inputs.query contains the promql query to extract out the desired prometheus metrics via running it on the given prometheus endpoint

Field .promProbe/inputs.queryPath
Description Flag to hold the path of the promql query for the promProbe
Type Optional
Range n/a {type: string}
Notes The .promProbe/inputs.queryPath This field is used in case of complex queries that spans multiple lines, the queryPath attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR.

Field .promProbe/inputs.comparator.criteria
Description Flag to hold criteria for the comparision
Type Mandatory
Range it supports {>=, <=, ==, >, <, !=, oneOf, between} criteria
Notes The .promProbe/inputs.comparator.criteria contains criteria of the comparision, which should be fulfill as part of comparision operation.

Field .promProbe/inputs.comparator.value
Description Flag to hold value for the comparision
Type Mandatory
Range n/a {type: string}
Notes The .promProbe/inputs.comparator.value contains value of the comparision, which should follow the given criteria as part of comparision operation.

Field .runProperties.probeTimeout
Description Flag to hold the timeout for the probes
Type Mandatory
Range n/a {type: integer}
Notes The .runProperties.probeTimeout represents the time limit for the probe to execute the specified check and return the expected data

Field .runProperties.retry
Description Flag to hold the retry count for the probes
Type Mandatory
Range n/a {type: integer}
Notes The .runProperties.retry contains the number of times a check is re-run upon failure in the first attempt before declaring the probe status as failed.

Field .runProperties.interval
Description Flag to hold the interval for the probes
Type Mandatory
Range n/a {type: integer}
Notes The .runProperties.interval contains the interval for which probes waits between subsequent retries

Field .runProperties.probePollingInterval
Description Flag to hold the polling interval for the probes(applicable for Continuous mode only)
Type Optional
Range n/a {type: integer}
Notes The .runProperties.probePollingInterval contains the time interval for which continuous probe should be sleep after each iteration

Field .runProperties.initialDelaySeconds
Description Flag to hold the initial delay interval for the probes
Type Optional
Range n/a {type: integer}
Notes The .runProperties.initialDelaySeconds represents the initial waiting time interval for the probes.

Field .runProperties.stopOnFailure
Description Flags to hold the stop or continue the experiment on probe failure
Type Optional
Range false {type: boolean}
Notes The .runProperties.stopOnFailure can be set to true/false to stop or continue the experiment execution after probe fails

Common Probe Tunables

Refer the common attributes to tune the common tunables for all the probes.

Prometheus Query(query is a simple)

It contains the promql query to extract out the desired prometheus metrics via running it on the given prometheus endpoint. The prometheus query can be provided in the query field. It can be executed by setting .promProbe/inputs.query field.

Use the following example to tune this:

# contains the prom probe which execute the query and match for the expected criteria
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-delete-sa
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "check-probe-success"
        type: "promProbe"
        promProbe/inputs:
          # endpoint for the promethus service
          endpoint: "<prometheus-endpoint>"
          # promql query, which should be executed
          query: "<promql-query>"
          comparator:
            # criteria which should be followed by the actual output and the expected output
            #supports >=,<=,>,<,==,!= comparision
            criteria: "==" 
            # expected value, which should follow the specified criteria
            value: "<value-for-criteria-match>"
        mode: "Edge"
        runProperties:
          probeTimeout: 5
          interval: 5
          retry: 1

Prometheus Query(query is a complex

In case of complex queries that spans multiple lines, the queryPath attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR. It can be executed by setting promProbe/inputs.queryPath field.

NOTE: It is mutually exclusive with the query field. If query is set then it will use the query field otherwise, it will use the queryPath field.

Use the following example to tune this:

# contains the prom probe which execute the query and match for the expected criteria
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-delete-sa
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "check-probe-success"
        type: "promProbe"
        promProbe/inputs:
          # endpoint for the promethus service
          endpoint: "<prometheus-endpoint>"
          # the configMap should be mounted to the experiment which contains promql query
          # use the mounted path here
          queryPath: "<path of the query>"
          comparator:
            # criteria which should be followed by the actual output and the expected output
            #supports >=,<=,>,<,==,!= comparision
            criteria: "==" 
            # expected value, which should follow the specified criteria
            value: "<value-for-criteria-match>"
        mode: "Edge"
        runProperties:
          probeTimeout: 5
          interval: 5
          retry: 1