K8s:开源安全平台 kubescape 实现 Pod 的安全合规检查/镜像漏洞扫描
【摘要】 写在前面生产环境中的 k8s 集群安全不可忽略，即使是内网环境容器化的应用部署虽然本质上没有变化，始终是机器上的一个进程但是提高了安全问题的处理的复杂性分享一个开源的 k8s 集群安全合规检查/漏洞扫描 工具 kubescape博文内容涉及：kubescape 简介介绍kubescape 命令行工具安装，扫描运行的集群kubescape 在集群下安装，通过 kubescape Clound ...
写在前面
-
生产环境中的 k8s 集群安全不可忽略，即使是内网环境 -
容器化的应用部署虽然本质上没有变化，始终是机器上的一个进程 -
但是提高了安全问题的处理的复杂性 -
分享一个开源的 k8s 集群安全合规检查/漏洞扫描 工具 kubescape -
博文内容涉及： -
kubescape简介介绍 -
kubescape命令行工具安装，扫描运行的集群 -
kubescape在集群下安装，通过 kubescape Clound 可视化查看扫描信息
-
-
理解不足小伙伴帮忙指正
对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》
简单介绍
k8s 安全问题不可忽略
镜像的不可变性使我们能够方便地部署、测试和发布镜像到其他环境，这是一个很大的优势，但也带来了潜在的风险：镜像及其依赖在过时或者被弃用时，无法自动更新或打新的安全补丁，尤其一些基础镜像，重新做镜像，需要依赖一些厂商，或者开源项目 team 来维护。
RedHat 在 2022 年 Kubernetes 安全报告中对 300 多名 DevOps、工程和安全专业人士进行了调查，发现对 容器安全威胁的担忧以及对容器安全投资的缺乏 是 31% 的受访者对容器策略最普遍的担忧。
支持这些担忧的是 93% 的受访者在过去 12 个月内在其 Kubernetes 环境中经历过至少一次安全事件，这些事件有时会导致收入或客户流失。超过一半的受访者 (55%) 在过去一年中还因为安全问题而不得不推迟应用程序的推出。
尽管媒体广泛关注网络攻击，但该报告强调，实际上是错误配置让 IT 专业人员彻夜难眠。Kubernetes 是高度可定制的，具有可以影响应用程序安全状况的各种配置选项。因此，受访者最担心的是容器和 Kubernetes 环境中的错误配置导致的风险暴露 (46%) —— 几乎是对攻击的担忧程度 (16%) 的三倍。尽可能自动化配置管理有助于缓解这些问题，因此安全工具 —— 而不是人类 —— 提供了帮助开发人员和 DevOps 团队更安全地配置容器和 Kubernetes 的护栏。
以上内容来自：Redhat Blog(The State of Kubernetes Security in 2022)
Kubescape 简单介绍
Kubescape 是一个开源的 Kubernetes 安全平台。它的功能包括 风险分析、安全合规性 和 错误配置扫描。针对 DevSecOps 从业者或平台工程师，提供易于使用的 CLI 界面、灵活的输出格式和自动扫描功能。同时对于小集群提供了免费的 在线 面板工具，它为 Kubernetes 用户和管理员节省了宝贵的时间、精力和资源。
Kubescape 可以扫描运行的集群、静态 YAML 文件和 本地 Helm Charts。它根据多个框架（包括 NSA-CISA、MITRE ATT&CK®和CIS Benchmark）检测错误配置。
Kubescape 由 ARMO 创建，是一个 Cloud Native Computing Foundation (CNCF) 沙盒项目。
Kubescape 如果小伙伴觉得重的话， kubectl 有一个类似的插件，个人十分推荐，叫做 kube-score ,很轻量，也可以做一些简单的合规性扫描。但是只有扫描合规性的提示，没有规范出处。
Kubescape 安装
当前的集群信息
┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl get nodes
NAME STATUS ROLES AGE VERSION
vms100.liruilongs.github.io Ready control-plane 8d v1.25.1
vms101.liruilongs.github.io Ready control-plane 8d v1.25.1
vms102.liruilongs.github.io Ready control-plane 8d v1.25.1
vms103.liruilongs.github.io Ready <none> 8d v1.25.1
vms105.liruilongs.github.io Ready <none> 8d v1.25.1
vms106.liruilongs.github.io Ready <none> 8d v1.25.1
vms107.liruilongs.github.io Ready <none> 8d v1.25.1
vms108.liruilongs.github.io Ready <none> 8d v1.25.1
┌──[root@vms100.liruilongs.github.io]-[~]
└─$
集群节点信息
┌──[root@vms100.liruilongs.github.io]-[~]
└─$hostnamectl
Static hostname: vms100.liruilongs.github.io
Icon name: computer-vm
Chassis: vm
Machine ID: e93ae3f6cb354f3ba509eeb73568087e
Boot ID: a1150b6d97dc4afbb81dae58f131a487
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 5.4.230-1.el7.elrepo.x86_64
Architecture: x86-64
┌──[root@vms100.liruilongs.github.io]-[~]
└─$
Kubescape 命令行工具安装
Kubescape CLI 安装
通过下面的方式自动安装
curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash
找一台可以访问的集群，下载 install.sh 文件，按照下面方式修改，获取 curl 命令，自行下载。
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$vim install.sh
.......
# curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT
echo "curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT"
exit 1
...
运行 shell，获取下载命令，找有网的机器下载
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$sh install.sh
Installing Kubescape...
curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape
上传到指定位置
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$ls
install.sh kubescape-ubuntu-latest
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$mv kubescape-ubuntu-latest /root/.kubescape/kubescape
mv：是否覆盖"/root/.kubescape/kubescape"？ y
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$
修改脚本，再次运行。
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$vim install.sh
# exit 1
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$sh install.sh
Installing Kubescape...
curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape
Finished Installation.
Your current version is: v2.0.183 [git enabled in build: true]
Usage: $ kubescape scan --enable-host-scan
到这里 ，kubescape 命令行工具即安装成功，扫描当前运行的集群可以运行如下命令。
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubescape scan --enable-host-scan --format html --output results.html --verbose
[info] Kubescape scanner starting
[info] Installing host scanner
[info] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[info] Accessing Kubernetes objects
[success] Accessed to Kubernetes objects
[info] Requesting images vulnerabilities results
[success] Requested images vulnerabilities results
[info] Requesting Host scanner data
[info] Host scanner version : v1.0.39
◑[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-xshr6; error: the server could not find the requested resource (get pods http:host-scanner-xshr6:7888)
◒[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-4tgnp; error: the server could not find the requested resource (get pods http:host-scanner-4tgnp:7888)
.......
[success] Done scanning. Cluster: kubernetes-admin-kubernetes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
################################################################################
ApiVersion: hostdata.kubescape.cloud/v1beta0
Kind: KubeletInfo
Name: vms105.liruilongs.github.io
Controls: 21 (Failed: 6, Excluded: 0)
+----------+--------------------------------+------------------------------------+------------------------+
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
+----------+--------------------------------+------------------------------------+------------------------+
| High | CIS-4.1.7 Ensure that the | https://hub.armosec.io/docs/c-0168 | |
| | certificate authorities file | | |
| | permissions are set to 600 or | | |
| | more restrictive | | |
+ +--------------------------------+------------------------------------+------------------------+
| | CIS-4.1.9 If the kubelet | https://hub.armosec.io/docs/c-0170 | |
| | config.yaml configuration | | |
| | file is being used validate | | |
| | permissions set to 600 or more | | |
| | restrictive | | |
+----------+--------------------------------+------------------------------------+------------------------+
........
+----------+--------------------------------+------------------------------------+------------------------+
| Low | CIS-4.2.6 Ensure that the | https://hub.armosec.io/docs/c-0177 | protectKernelDefaults |
| | --protect-kernel-defaults | | |
| | argument is set to true | | |
+ +--------------------------------+------------------------------------+------------------------+
| | CIS-4.2.7 Ensure that the | https://hub.armosec.io/docs/c-0178 | makeIPTablesUtilChains |
| | --make-iptables-util-chains | | |
| | argument is set to true | | |
+----------+--------------------------------+------------------------------------+------------------------+
################################################################################
ApiVersion: v1
Kind: Namespace
Name: kubescape
Controls: 18 (Failed: 17, Excluded: 0)
+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| High | CIS-5.2.11 Minimize the | https://hub.armosec.io/docs/c-0202 | metadata.labels[pod-security.kubernetes.io/enforce]=baseline |
| | admission of Windows | | |
| | HostProcess Containers | | |
+ +--------------------------------+------------------------------------+ +
| | CIS-5.2.2 Minimize the | https://hub.armosec.io/docs/c-0193 | |
| | admission of privileged | | |
| | containers | | |
+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| Medium | CIS-5.2.1 Ensure that | https://hub.armosec.io/docs/c-0192 | metadata.labels[pod-security.kubernetes.io/enforce]=YOUR_VALUE |
| | the cluster has at least | | |
| | one active policy control | | |
| | mechanism in place | | |
+ +--------------------------------+------------------------------------+----------------------------------------------------------------+
.......
..........
################################################################################
ApiVersion: hostdata.kubescape.cloud/v1beta0
Kind: ControlPlaneInfo
Name: vms102.liruilongs.github.io
Controls: 25 (Failed: 1, Excluded: 0)
+----------+--------------------------------+------------------------------------+-----------------------+
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
+----------+--------------------------------+------------------------------------+-----------------------+
| High | CIS-1.1.20 Ensure that the | https://hub.armosec.io/docs/c-0111 | |
| | Kubernetes PKI certificate | | |
| | file permissions are set to | | |
| | 600 or more restrictive | | |
+----------+--------------------------------+------------------------------------+-----------------------+
################################################################################
ApiVersion: apps/v1
Kind: Deployment
Name: local-path-provisioner
Namespace: local-path-storage
Controls: 35 (Failed: 18, Excluded: 0)
...
........
......
输出的合规信息和漏洞信息。
异常问题
我在 kubescape 多次次扫描中，集群因部分节点因为端口问题，无法发生调度，节点上的 kubescape 对于的 pod 无法自行删除，如果上一次扫描的 pod 或则 ns 没有删除，那么下一次的扫描无法进行，之前创建的 pod 和 ns 状态一直为 Terminating, 解决办法需要 对命名空间进行彻底删除。
这里执行完命令会进入阻塞状态。
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubevirt]
└─$kubectl delete ns kubescape-host-scanner
namespace "kubescape-host-scanner" deleted
运行脚本删除
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript]
└─$cat delete_namespace.sh
#!/bin/bash
coproc kubectl proxy --port=30990 &
if [ $# -eq 0 ] ; then
echo "后面加上你所要删除的ns."
exit 1
fi
kubectl get namespace $1 -o json > logging.json
sed -i '/"finalizers"/{n;d}' logging.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @logging.json http://127.0.0.1:30990/api/v1/namespaces/${1}/finalize
kill %1
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript]
└─$sh delete_namespace.sh kubescape-host-scanner
也可以离线运行Kubescape，时间网络原因，这里不做分享，有需要的小伙伴可以到 github 上的项目地址查看详细信息。
一些其他的用法
扫描正在运行的 Kubernetes 集群：
kubescape scan --enable-host-scan --verbose
使用替代的 kubeconfig 文件：
kubescape scan --kubeconfig cluster.conf
扫描特定的命名空间：
kubescape scan --include-namespaces development,staging,production
排除某些命名空间：
kubescape scan --exclude-namespaces kube-system,kube-public
部署前扫描本地 YAML/JSON 文件：
kubescape scan *.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible/helm]
└─$kubescape scan --enable-host-scan kube-prometheus-stack.yaml --format html --output resout.html
[info] Kubescape scanner starting
[warning] in setContextMetadata. case: git-url; error: repository host 'gitee.com' not supported
[info] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[info] Accessing local objects
[success] Done accessing local objects
[info] Scanning GitLocal
[success] Done scanning GitLocal
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Controls: 55 (Failed: 31, Excluded: 0, Skipped: 0)
Failed Resources by Severity: Critical — 0, High — 34, Medium — 151, Low — 24
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| SEVERITY | CONTROL NAME | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| High | Resources memory limit and request | 7 | 0 | 7 | 100% |
| High | Resource limits | 7 | 0 | 7 | 100% |
| High | List Kubernetes secrets | 4 | 0 | 7 | 57% |
。。。。。。。。。。。。。。。。。。。。。
| Medium | Allow privilege escalation | 6 | 0 | 7 | 86% |
| Medium | Ingress and Egress blocked | 7 | 0 | 7 | 100% |
。。。。。。。。。。。。。。。。。。。。。
| Medium | CIS-5.4.1 Prefer using secrets as files over secrets as environment variables | 1 | 0 | 7 | 14% |
| Medium | CIS-5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions | 7 | 0 | 7 | 100% |
| Medium | CIS-5.7.4 The default namespace should not be used | 56 | 0 | 61 | 92% |
| Low | Immutable container filesystem | 6 | 0 | 7 | 86% |
| Low | Configured readiness probe | 5 | 0 | 7 | 71% |
| Low | Malicious admission controller (validating) | 1 | 0 | 1 | 100% |
| Low | Pods in default namespace | 7 | 0 | 7 | 100% |
| Low | Naked PODs | 1 | 0 | 1 | 100% |
| Low | Label usage for resources | 3 | 0 | 7 | 43% |
| Low | K8s common labels usage | 1 | 0 | 7 | 14% |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| | RESOURCE SUMMARY | 66 | 0 | 72 | 43.26% |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
FRAMEWORKS: ArmoBest (risk: 33.51), cis-v1.23-t1.0.1 (risk: 62.75), DevOpsBest (risk: 61.72), AllControls (risk: 36.46), MITRE (risk: 11.79), NSA (risk: 33.74)
[success] Scan results saved. filename: resout.html
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scan results have not been submitted: run kubescape with the '--account' flag
For more details: https://hub.armosec.io/docs/installing-kubescape?utm_campaign=Submit&utm_medium=CLI&utm_source=GitHub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run with '--verbose'/'-v' flag for detailed resources view
┌──[root@vms100.liruilongs.github.io]-[~/ansible/helm]
└─$l
从 Git 存储库扫描 Kubernetes 清单文件：
kubescape scan https://github.com/kubescape/kubescape
扫描 Helm 图表,kubescape 将加载默认的 VALUES 文件。
kubescape scan </path/to/directory>
扫描 Kustomize 目录,Kubescape 将使用 kustomize 文件生成 Kubernetes YAML 对象并扫描它们以确保安全。
kubescape scan </path/to/directory>
使用 NSA 框架扫描正在运行的 Kubernetes 集群：
kubescape scan framework nsa
使用 MITRE ATT&CK® 框架扫描正在运行的 Kubernetes 集群：
kubescape scan framework mitre
使用控件名称或控件 ID 扫描特定控件。请参阅控件列表。
kubescape scan control "Privileged container"
指定报告输出格式
JSON：
kubescape scan --format json --format-version v2 --output results.json
XML：
kubescape scan --format junit --output results.xml
PDF:
kubescape scan --format pdf --output results.pdf
普罗米修斯指标：
kubescape scan --format prometheus
HTML
kubescape scan --format html --output results.html
显示所有扫描到的资源（包括通过的资源）：
kubescape scan --verbose
Kubescape 在集群下安装
Kubescape 也可以在集群下安装，通过集群安装，可以在 cloud 里的 UI 界面查看具体扫描信息，集群中安装 Kubescape，先决条件
-
确保您拥有 Kubescape Cloud 帐户——如果没有，请在此处注册 -
您需要拥有对集群的安装权限（您应该能够创建 Deployments、CronJobs、ConfigMaps 和 Secrets） -
你必须有 Kubectl 和 Helm
集群要求
-
Kubescape 运算符组件至少需要 400Mib RAM 和 400m CPU
具体的安装可以参考下的教程： 需要注册 Kubescape Cloud，并且 只有的工作节点小于 10 个的时候才免费。
https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
注册登录
这里登录完会弹出一个安装部署，安装安装部署安装即可
添加 Helm 源，并更新
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm repo add kubescape https://kubescape.github.io/helm-charts/
"kubescape" has been added to your repositories
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "rancher-stable" chart repository
...Successfully got an update from the "botkube" chart repository
...Successfully got an update from the "awx-operator" chart repository
...Successfully got an update from the "kubescape" chart repository
Update Complete. ⎈Happy Helming!⎈
运行 charts
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm upgrade --install kubescape kubescape/kubescape-cloud-operator -n kubescape --create-namespace --set account=97f09924-0c06-42e4-bdad-5b333321af77 --set clusterName=`kubectl config current-context`
Release "kubescape" does not exist. Installing it now.
.........
......................
NAME: kubescape
LAST DEPLOYED: Sat Feb 4 10:03:36 2023
NAMESPACE: kubescape
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing kubescape-cloud-operator version 1.9.5.
In a few minutes your scan results will be available in the following link:
https://cloud.armosec.io/config-scanning/kubernetes-admin-kubernetes
You can see and change the values of your's recurring configurations daily scan in the following link:
https://cloud.armosec.io/settings/assets/clusters/scheduled-scans?cluster=kubernetes-admin-kubernetes
> kubectl -n kubescape get cj kubescape-scheduler -o=jsonpath='{.metadata.name}{"\t"}{.spec.schedule}{"\n"}'
You can see and change the values of your's recurring images daily scan in the following link:
https://cloud.armosec.io/settings/assets/images
> kubectl -n kubescape get cj kubevuln-scheduler -o=jsonpath='{.metadata.name}{"\t"}{.spec.schedule}{"\n"}'
See you!!!
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$
运行完成，安装提示操作，点击刚才注册生成的页面。
验证 Kubescape 在集群中运行状态
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$kubectl -n kubescape get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
gateway 1/1 1 1 19m
kubescape 1/1 1 1 19m
kubevuln 1/1 1 1 19m
operator 1/1 1 1 19m
kubescape 会定期扫描
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$kubectl get cronjobs.batch -n kubescape
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
kubescape-scheduler 21 3 * * * False 0 <none> 20m
kubevuln-scheduler 13 22 * * * False 0 <none> 20m
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$
在登录的主页中可以看到集群和第一次扫描结果
安全合规性扫描信息
可以通过不同的维度来看
合规性的维度查看 
集群 Pod/Deploy 的维度查看 
具体的 合规配置信息查看
漏洞扫描
关于 kubescape 和小伙伴分享到这里，时间关系，没有深入太多。偶尔听大佬谈到，所以研究一下，有需要的小伙伴快去尝试吧。
博文部分内容参考
文中涉及参考链接内容版权归原作者所有，如有侵权请告知
https://github.com/kubescape/kubescape
https://www.redhat.com/en/blog/state-kubernetes-security-2022-1
https://betterprogramming.pub/image-vulnerability-scanning-for-optimal-kubernetes-security-c3ba933190ef
https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
© 2018-2023 liruilonger@gmail.com，All rights reserved. 保持署名-非商用-自由转载-相同方式共享(创意共享 3.0 许可证)
【声明】本内容来自华为云开发者社区博主，不代表华为云及华为云开发者社区的观点和立场。转载时必须标注文章的来源（华为云社区）、文章链接、文章作者等基本信息，否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容，欢迎发送邮件进行举报，并提供相关证据，一经查实，本社区将立刻删除涉嫌侵权内容，举报邮箱：
cloudbbs@huaweicloud.com
- 点赞
- 收藏
- 关注作者
评论（0）