Pod HTTP Status Code

Introduction

• It injects http status code chaos inside the pod which modifies the status code of the response from the provided application server to desired status code provided by user on the service whose port is provided as TARGET_SERVICE_PORT by starting proxy server and then redirecting the traffic through the proxy server.
• It can test the application's resilience to error code http responses from the provided application server.

Scenario: Modify http response status code of the HTTP request

Uses

View the uses of the experiment

coming soon

Prerequisites

Verify the prerequisites

• Ensure that Kubernetes Version > 1.17
• Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus).If not, install from here
• Ensure that the pod-http-status-code experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here

Default Validations

View the default validations

The application pods should be in running state before and after chaos injection.

Minimal RBAC configuration example (optional)

NOTE

If you are using this experiment as part of a litmus workflow scheduled constructed & executed from chaos-center, then you may be making use of the litmus-admin RBAC, which is pre installed in the cluster as part of the agent setup.

View the Minimal RBAC permissions

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-http-status-code-sa
  namespace: default
  labels:
    name: pod-http-status-code-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-http-status-code-sa
  namespace: default
  labels:
    name: pod-http-status-code-sa
    app.kubernetes.io/part-of: litmus
rules:
  # Create and monitor the experiment & helper pods
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update", "deletecollection"]
  # Performs CRUD operations on the events inside chaosengine and chaosresult
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create","get","list","patch","update"]
  # Fetch configmaps details and mount it to the experiment pod (if specified)
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get","list",]
  # Track and get the runner, experiment, and helper pods log
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  # for creating and managing to execute comands inside target container
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["get","list","create"]
  # deriving the parent/owner details of the pod(if parent is anyof {deployment, statefulset, daemonsets})
  - apiGroups: ["apps"]
    resources: ["deployments","statefulsets","replicasets", "daemonsets"]
    verbs: ["list","get"]
  # deriving the parent/owner details of the pod(if parent is deploymentConfig)
  - apiGroups: ["apps.openshift.io"]
    resources: ["deploymentconfigs"]
    verbs: ["list","get"]
  # deriving the parent/owner details of the pod(if parent is deploymentConfig)
  - apiGroups: [""]
    resources: ["replicationcontrollers"]
    verbs: ["get","list"]
  # deriving the parent/owner details of the pod(if parent is argo-rollouts)
  - apiGroups: ["argoproj.io"]
    resources: ["rollouts"]
    verbs: ["list","get"]
  # for configuring and monitor the experiment job by the chaos-runner pod
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create","list","get","delete","deletecollection"]
  # for creation, status polling and deletion of litmus chaos resources used within a chaos workflow
  - apiGroups: ["litmuschaos.io"]
    resources: ["chaosengines","chaosexperiments","chaosresults"]
    verbs: ["create","list","get","patch","update","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-http-status-code-sa
  namespace: default
  labels:
    name: pod-http-status-code-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-http-status-code-sa
subjects:
- kind: ServiceAccount
  name: pod-http-status-code-sa
  namespace: default

Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Experiment tunables

check the experiment tunables

Mandatory Fields

Variables	Description	Notes
TARGET_SERVICE_PORT	Port of the service to target	This should be the port on which the application container runs at the pod level, not at the service level. Defaults to port 80
STATUS_CODE	Modified status code for the HTTP response	If no value is provided, then a random value is selected from the list of supported values. Multiple values can be provided as comma separated, a random value from the provided list will be selected Supported values: [200, 201, 202, 204, 300, 301, 302, 304, 307, 400, 401, 403, 404, 500, 501, 502, 503, 504]. Defaults to random status code
MODIFY_RESPONSE_BODY	Whether to modify the body as per the status code provided.	If true, then the body is replaced by a default template for the status code. Defaults to true

Optional Fields

Variables	Description	Notes
RESPONSE_BODY	Body string to overwrite the http response body	This will be used only if MODIFY_RESPONSE_BODY is set to true. If no value is provided, response will be an empty body. Defaults to empty body
CONTENT_ENCODING	Encoding type to compress/encodde the response body	Accepted values are: gzip, deflate, br, identity. Defaults to none (no encoding)
CONTENT_TYPE	Content type of the response body	Defaults to text/plain
PROXY_PORT	Port where the proxy will be listening for requests	Defaults to 20000
NETWORK_INTERFACE	Network interface to be used for the proxy	Defaults to eth0
TOXICITY	Percentage of HTTP requests to be affected	Defaults to 100
CONTAINER_RUNTIME	container runtime interface for the cluster	Defaults to containerd, supported values: docker, containerd and crio for litmus and only docker for pumba LIB
SOCKET_PATH	Path of the containerd/crio/docker socket file	Defaults to /run/containerd/containerd.sock
TOTAL_CHAOS_DURATION	The duration of chaos injection (seconds)	Default (60s)
TARGET_PODS	Comma separated list of application pod name subjected to pod http status code chaos	If not provided, it will select target pods randomly based on provided appLabels
PODS_AFFECTED_PERC	The Percentage of total pods to target	Defaults to 0 (corresponds to 1 replica), provide numeric value only
LIB_IMAGE	Image used to run the netem command	Defaults to litmuschaos/go-runner:latest
RAMP_TIME	Period to wait before and after injection of chaos in sec
SEQUENCE	It defines sequence of chaos execution for multiple target pods	Default value: parallel. Supported: serial, parallel

Experiment Examples

Common and Pod specific tunables

Refer the common attributes and Pod specific tunable to tune the common tunables for all experiments and pod specific tunables.

Target Service Port

It defines the port of the targeted service that is being targeted. It can be tuned via TARGET_SERVICE_PORT ENV. This should be the port where the application runs at the pod level, not at the service level. This means if the application pod is running the service at port 8080 and we create a service exposing that at port 80, then the target service port should be 8080 and not 80, which is the port at pod-level.

Use the following example to tune this:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"
        # modified status code for the http response
        - name: STATUS_CODE
          value: '500'

Proxy Port

It defines the port on which the proxy server will listen for requests. It can be tuned via PROXY_PORT ENV.

Use the following example to tune this:

## provide the port for proxy server
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        # provide the port for proxy server
        - name: PROXY_PORT
          value: '8080'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"
        # modified status code for the http response
        - name: STATUS_CODE
          value: '500'

Status Code

It defines the status code value for the http response. It can be tuned via STATUS_CODE ENV.

Use the following example to tune this:

## modified status code for the http response
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        # modified status code for the http response
        # if no value is provided, a random status code from the supported code list will selected
        # if multiple comma separated values are provided, then a random value from the provided list will be selected
        # if an invalid status code is provided, the experiment will fail
        # supported status code list: [200, 201, 202, 204, 300, 301, 302, 304, 307, 400, 401, 403, 404, 500, 501, 502, 503, 504]
        - name: STATUS_CODE
          value: '500'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Modify Response Body

It defines whether to modify the respone body with a pre-defined template to match with the status code value of the http response. It can be tuned via MODIFY_RESPONSE_BODY ENV.

Use the following example to tune this:

##  whether to modify the body as per the status code provided
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        #  whether to modify the body as per the status code provided
        - name: "MODIFY_RESPONSE_BODY"
          value: "true"
        # modified status code for the http response
        - name: STATUS_CODE
          value: '500'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Toxicity

It defines the toxicity value to be added to the http request. It can be tuned via TOXICITY ENV. Toxicity value defines the percentage of the total number of http requests to be affected.

Use the following example to tune this:

## provide the toxicity
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        # toxicity is the probability of the request to be affected
        # provide the percentage value in the range of 0-100
        # 0 means no request will be affected and 100 means all request will be affected
        - name: TOXICITY
          value: "100"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

RESPONSE BODY

It defines the body string that will overwrite the http response body. It can be tuned via RESPONSE_BODY and MODIFY_RESPONSE_BODY ENV. The MODIFY_RESPONSE_BODY ENV should be set to true to enable this feature.

Use the following example to tune this:

## provide the response body value
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        # provide the body string to overwrite the response body. This will be used only if MODIFY_RESPONSE_BODY is set to true
        - name: RESPONSE_BODY
          value: '<h1>Hello World</h1>'
        #  whether to modify the body as per the status code provided
        - name: "MODIFY_RESPONSE_BODY"
          value: "true"
        # modified status code for the http response
        - name: STATUS_CODE
          value: '500'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Content Encoding and Content Type

It defines the content encoding and content type of the response body. It can be tuned via CONTENT_ENCODING and CONTENT_TYPE ENV.

Use the following example to tune this:

##  whether to modify the body as per the status code provided
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        # provide the encoding type for the response body
        # currently supported value are gzip, deflate
        # if empty no encoding will be applied
        - name: CONTENT_ENCODING
          value: 'gzip'
        # provide the content type for the response body
        - name: CONTENT_TYPE
          value: 'text/html'
        #  whether to modify the body as per the status code provided
        - name: "MODIFY_RESPONSE_BODY"
          value: "true"
        # modified status code for the http response
        - name: STATUS_CODE
          value: '500'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Network Interface

It defines the network interface to be used for the proxy. It can be tuned via NETWORK_INTERFACE ENV.

Use the following example to tune this:

## provide the network interface for proxy
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        # provide the network interface for proxy
        - name: NETWORK_INTERFACE
          value: "eth0"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: '80'
        # modified status code for the http response
        - name: STATUS_CODE
          value: '500'

Container Runtime Socket Path

It defines the CONTAINER_RUNTIME and SOCKET_PATH ENV to set the container runtime and socket file path.

• CONTAINER_RUNTIME: It supports docker, containerd, and crio runtimes. The default value is docker.
• SOCKET_PATH: It contains path of docker socket file by default(/run/containerd/containerd.sock). For other runtimes provide the appropriate path.

Use the following example to tune this:

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-http-status-code-sa
  experiments:
  - name: pod-http-status-code
    spec:
      components:
        env:
        # runtime for the container
        # supports docker, containerd, crio
        - name: CONTAINER_RUNTIME
          value: 'containerd'
        # path of the socket file
        - name: SOCKET_PATH
          value: '/run/containerd/containerd.sock'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"
        # modified status code for the http response
        - name: STATUS_CODE
          value: '500'