OpenShift | Luis Cacho

Trigger a scheduled job manually on Kubernetes

Thu, 22 Jun 2023 00:00:00 +0000

# Mockup command
kubectl create job --from=cronjob/<cronjob-name> <job-name> -n <namespace-name>

References:

¹How can I trigger a Kubernetes Scheduled Job manually?

https://stackoverflow.com/questions/40401795/how-can-i-trigger-a-kubernetes-scheduled-job-manually ↩︎

Understanding the Compliance Operator

Tue, 16 Aug 2022 00:00:00 +0000

Table of Contents

Compliance studies a company’s security processes. It details their security at a moment in time and compares it to a specific set of regulatory requirements. These requirements come in the form of legislation, industry regulations, or standards created from best practices.

Every company can protect its data accordingly if they follow Compliance frameworks and have quality security in place. To have proper protection, companies must understand that Compliance is not the same thing as security. However, security is a big part of Compliance.

Becoming secure and compliant means securing information assets, preventing damage, protecting it, eliminating potential attack vectors and decreasing the system’s attack surface.

Compliance Operator Introduction

The Compliance Operator was released in OpenShift Container Platform v4.6, and allows OpenShift Container Platform administrators describe the desired compliance state of a cluster and provides them with an overview of gaps and ways to remediate them. The Compliance Operator assesses compliance of both the Kubernetes API resources of OpenShift Container Platform, as well as the nodes running the cluster. The Compliance Operator uses OpenSCAP, a NIST-certified tool, to scan and enforce security policies provided by the content.¹

The Compliance Operator is available for Red Hat Enterprise Linux CoreOS (RHCOS) deployments only.

To secure your OpenShift cluster, it is necessary to consider both; platform (Kubernetes/OpenShift API) and host OS (RHCOS) perspectives, because Kubernetes is composed of control plane machines (master) and worker machines (node), then the Kubernetes services such as API Server, etcd, or controller manager run on the control plane to manage the workloads on the worker machines.

Platform and Host Perspective

The basic approach to perform hardening in a RHCOS host is described in the documentation and it is based in the RHEL 8 Security Hardening. For instance, the hardening consists in validate if the file has appropriately restrictive file permissions, if the file ownership is appropriately set, if required systemd services or processes are launched with appropriate arguments or parameters in the configuration, or if the appropriate kernel parameters are set. Then, hardening the control plane is specific to the Kubernetes services and includes the control plane components or the master configuration files. It should validate if the API server has started with restrictive arguments regarding allowing specific admission plug-ins, enabling audit logging, applying etcd server and peer configurations, and restricting RBAC, among others. For clusters that use RHCOS, updating or upgrading are designed to become automatic events from the central control plane, because OpenShift completely controls the systems and services that run on each machine, including the operating system itself through the Machine Config Operator.

Why Compliance Operator?

Why is the Compliance Operator needed to validate the hardening and apply changes in the configuration of the operating system and the platform? The Compliance Operator is defined as follows:

The Compliance Operator lets OpenShift Container Platform administrators describe the required compliance state of a cluster and provides them with an overview of gaps and ways to remediate them. The Compliance Operator assesses compliance of both the Kubernetes API resources of OpenShift Container Platform, as well as the nodes running the cluster. The Compliance Operator uses OpenSCAP, a NIST-certified tool, to scan and enforce security policies provided by the content.¹

In other words, the Compliance Operator checks the host and the platform to detect gaps in compliance by specifying profiles for scan and creates summary reports about security compliance so that you will be able to find if there is any configuration that violates the policy in the cluster. The reports also show which remediations are applied, so you can choose if you want to apply the recommended configuration by hand, step-by-step, or automatically. In short, the whole process goes like this: choosing a profile for scanning, specifying the scan settings, then initiating the scan, and generating the reports.

Compliance Operator Overview

The Compliance Operator leverages OpenSCAP, a NIST-certified tool, to scan and enforce security policies, and the security policies for the compliance checks are derived through SCAP content and built from the community-based ComplianceAsCode/content project. A bundle of security policies, or profiles created by default when the operator is installed and profiles scheduled include NIST 800-53 Moderate (FedRAMP), Australian Cyber Security Centre (ACSC) Essential Eight, CIS OpenShift Benchmark, and others, so far. You can also create or tailor your own profiles so that you can pick the rules you want to run other than the profiles provided by default.

How can I implement Compliance Operator in my cluster?

First of all, you will need to install the Compliance Operator, for that, you can follow the documentation steps

Great, the Operator is installed and functional, now lets check how to create, run and evaluate a compliance scan.

References:

https://docs.openshift.com/container-platform/4.8/security/compliance_operator/compliance-operator-understanding.html ↩︎ ↩︎

Delete OpenShift Operator

Mon, 15 Aug 2022 00:00:00 +0000

# Mockup commands

oc get subscription <operator> -n openshift-operators -o yaml | grep
currentCSV

oc delete subscription <operator> -n openshift-operators

oc delete clusterserviceversion <currentCSV> -n openshift-operators

# Example commands

oc get subscription jaeger -n openshift-operators -o yaml | grep currentCSV

oc delete subscription jaeger -n openshift-operators

oc delete clusterserviceversion jaeger-operator.v1.8.2 -n openshift-operators

Delete OpenShift Project

Mon, 15 Aug 2022 00:00:00 +0000

Issue, can’t delete a namespace/project and is stuck in a “Terminating” status

oc delete project fishbone

oc get projects fishbone
NAME DISPLAY NAME STATUS
fishbone Fishbone Terminating

Solution

Define the namespace/project to delete

export NAMESPACE=fishbone

Cleanup the finalizers section from the namespace definition and define a json file with that

oc get namespace $NAMESPACE -o=json | jq '.spec = {"finalizers":[]}' > $NAMESPACE.json
{
 "apiVersion": "v1",
 "kind": "Namespace",
 "metadata": {
 "annotations":
 "openshift.io/description": "",
 "openshift.io/display-name": "Fishbone",
 "openshift.io/requester": "system:admin",
 "openshift.io/sa.scc.mcs": "s0:c24,c14",
 "openshift.io/sa.scc.supplemental-groups": "1000580000/10000",
 "openshift.io/sa.scc.uid-range": "1000580000/10000"
 },
 "creationTimestamp": "2021-02-20T03:29:59Z",
 "deletionTimestamp": "2021-02-20T03:31:35Z",
 "name": "fishbone",
 "resourceVersion": "270664",
 "selfLink": "/api/v1/namespaces/fishbone",
 "uid": "885c4440-6f05-4e72-a103-28e61705406f"
 },
 "spec": {
 "finalizers": [
 "kubernetes"
 ]
 },
 "status": {
 "conditions": [
 .
 .
 .
 ],
 "phase": "Terminating"
}

Apply the json file from the previous step to delete the finalizers and therefore the deletion of the namespace/project

curl -k -H "authorization: Bearer $(oc whoami --show-token)" -H "Content-Type: application/json" -X PUT --data-binary @$NAMESPACE.json $(oc whoami --show-server)/api/v1/namespaces/$NAMESPACE/finalize

Verify that the namspace/project has been deleted

oc get projects $NAMESPACE
Error from server (NotFound): namespaces "fishbone" not found

References:

Kubernetes Advanced Topics¹
Kubernetes Namespace Stuck²
Red Hat article[^3]

How to Retrieve secrets values

Mon, 15 Aug 2022 00:00:00 +0000

oc get secret my-secret -n my-project --template={{.data.my-secret-key}} | base64 -d

oc get secret my-secret -n my-project -o go-template --template="{{.data.my-secret-key|base64decode}}"

oc extract secrets/my-secret -n my-project

oc get secret my-secret -n my-project -o jsonpath="{.data['tls\.key']}" | base64 -d

How to test the OpenShift Cluster and Machine Autoscaler

Mon, 15 Aug 2022 00:00:00 +0000

oc new-project work-queue


echo 'apiVersion: batch/v1
kind: Job
metadata:
 generateName: work-queue-
spec:
 template:
 spec:
 tolerations:
 - effect: NoSchedule
 key: node-role.kubernetes.io/infra
 value: reserved
 - effect: NoExecute
 key: node-role.kubernetes.io/infra
 value: reserved
 containers:
 - name: work
 image: busybox
 command: ["sleep", "300"]
 resources:
 requests:
 memory: 2Gi
 cpu: 4
 restartPolicy: Never
 parallelism: 50
 completions: 250' | oc create -f - -n work-queue

oc get jobs -n work-queue

oc get pods -n work-queue 

oc get machineset -n openshift-machine-api

oc get machines -n openshift-machine-api

oc get nodes

oc delete project work-queue

How to list the taints on kubernetes nodes

Sun, 14 Aug 2022 00:00:00 +0000

oc get nodes -o json | jq '.items[].spec.taints'
oc get nodes -o json | jq '.items[] | {node_name:.metadata.name,taints:.spec.taints[]?}

oc get nodes -o=custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

References:

¹ StackOverflow

]: https://stackoverflow.com/questions/43379415/how-can-i-list-the-taints-on-kubernetes-nodes ↩︎

OpenID autodiscovery URL integration in OpenShift

Mon, 23 May 2022 00:00:00 +0000

# Discover the available URLs from your autodiscovery URL
curl https://<idp_host>/.well-known/openid-configuration

CLIENT_SECRET=$(oc get secret -n openshift-config my-id-secret my-id-secret -o jsonpath='{.items[0].data.clientSecret}' | base64 -D)

curl -s -X POST https://<idp_host>/idp/userinfo.openid -H 'content-type: application/x-www-form-urlencoded' -d "client_id=ocp4testawsuswest2QA&client_secret=${CLIENT_SECRET}&access_token=${ACCESS_TOKEN}" | jq

References: - ¹ https://access.redhat.com/solutions/4605141

https://access.redhat.com/solutions/4605141 ↩︎

How to access an OpenShift 4 Node

Thu, 16 Sep 2021 00:00:00 +0000

In order to review the underline node, you need to follow the next steps:

Get the node name that is having issues or your need to login

oc get nodes

Run the oc debug command into the node

oc debug node/ip-10-0-129-255.us-east-2.compute.internal

Once you are logged in the node, you’ll need to run:

chroot /host

References:

¹ OCP4 Documentation

https://docs.openshift.com/container-platform/4.10/support/gathering-cluster-data.html#support-collecting-network-trace_gathering-cluster-data ↩︎

Get ServiceAccount token for MTC

Fri, 06 Aug 2021 00:00:00 +0000

oc sa get-token migration-operator -n openshift-migration

Command to perform a curl to a PodIP

Thu, 05 Aug 2021 00:00:00 +0000

curl $(oc get pod <pod-name> --template '{{ .status.podIP}}):8080

Approve pending CSRs in OpenShift

Wed, 04 Aug 2021 00:00:00 +0000

for id in $(oc get csr | grep Pending | awk '{print $1}'); do oc adm certificate approve $id; done

Get OpenShift Console URL

Mon, 02 Aug 2021 00:00:00 +0000

oc whoami --show-console

oc -n openshift-console get routes

Kubernetes debug commands

Sun, 01 Aug 2021 00:00:00 +0000

Kubernetes error messages

kubectl get events -n kube-system

Kubernetes pods logs

stern prox -n kube-system

Debug with busybox

kubectl run --generator=run-pod/v1 -i --tty busybox --image=busybox --restart=Never -- sh

CodeReady Containers commands

Tue, 20 Jul 2021 00:00:00 +0000

crc config view # View Actual config
crc config set autostart-tray false # Disable autostart
crc config set consent-telemetry false # Disable telemetry
crc config set enable-cluster-monitoring true # Enable Monitoring, needs more memory
crc config set host-network-access true # Enable TCP/IP connections
crc config set kubeadmin-password $MY_PASSW0RD # Define my own kubeadmin-password
crc config set memory 14336 # Increase Memory to 14GB
crc config set pull-secret-file $PATH_TO_PULL_SECRET_FILE # Path to a pull-secret file (download from https://cloud.redhat.com/openshift/create/local)
crc setup
crc start
crc ip
crc stop

Wait for pod to be ready

Thu, 15 Jul 2021 00:00:00 +0000

export POD_NAME=hello-d8d8d7455-j9nzw
while [[ $(oc get pods $POD_NAME -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}') != "True" ]]; do echo "waiting for pod" && sleep 1; done

If you want to wait until all the pods with a common label you can do:

while [[ $(kubectl get pods -l app=hello -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}') != "True" ]]; do echo "waiting for pod" && sleep 1; done

OpenShift | Luis Cacho

Trigger a scheduled job manually on Kubernetes

Understanding the Compliance Operator

Compliance Operator Introduction

Why Compliance Operator?

How can I implement Compliance Operator in my cluster?

Delete OpenShift Operator

Delete OpenShift Project

How to Retrieve secrets values

How to test the OpenShift Cluster and Machine Autoscaler

How to list the taints on kubernetes nodes

OpenID autodiscovery URL integration in OpenShift

How to access an OpenShift 4 Node

Get ServiceAccount token for MTC

Command to perform a curl to a PodIP

Approve pending CSRs in OpenShift

Get OpenShift Console URL

Kubernetes debug commands

CodeReady Containers commands

Wait for pod to be ready

Differences between Kubernetes and Openshift

Differences between Kubernetes and Openshift

Luis Cacho

Senior Container Infrastructure Consultant

@Red Hat

Kubernetes

Some assembly required

Kubernetes

Some assembly required

Kubernetes

Some assembly required

Openshift

Value Added

Openshift

Value Added

Openshift

Value Added

Openshift contains Kubernetes

Openshift contains Kubernetes

Conclusion

Openshit IS a Kubernetes flavor

Thank You