WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content

Hubble UI & relay Crashloopback #1678

New issue
New issue
Open
Open
Hubble UI & relay Crashloopback#1678
@cloudcafetech

Description

@cloudcafetech
cloudcafetech
opened on May 4, 2025

Running on RKE2.

  • POD status
NAMESPACE     NAME                                                    READY   STATUS             RESTARTS         AGE
kube-system   cilium-envoy-h9vn7                                      1/1     Running            0                124m
kube-system   cilium-operator-56668b95f-4hk6c                         1/1     Running            0                124m
kube-system   cilium-zg642                                            1/1     Running            0                124m
kube-system   cloud-controller-manager-ubuntu2004                     1/1     Running            0                151m
kube-system   etcd-ubuntu2004                                         1/1     Running            0                151m
kube-system   helm-install-rke2-coredns-zcntk                         0/1     Completed          0                152m
kube-system   helm-install-rke2-metrics-server-8dwp4                  1/1     Running            24 (5m22s ago)   152m
kube-system   helm-install-rke2-runtimeclasses-vxklc                  1/1     Running            24 (5m26s ago)   152m
kube-system   hubble-relay-846d47bdc5-558rv                           0/1     CrashLoopBackOff   35 (3m5s ago)    124m
kube-system   hubble-ui-5f75567858-9sgst                              1/2     CrashLoopBackOff   43 (3m40s ago)   124m
kube-system   kube-apiserver-ubuntu2004                               1/1     Running            0                151m
kube-system   kube-controller-manager-ubuntu2004                      1/1     Running            0                151m
kube-system   kube-scheduler-ubuntu2004                               1/1     Running            0                151m
kube-system   rke2-coredns-rke2-coredns-86f64967bd-jk5db              0/1     CrashLoopBackOff   29 (3m6s ago)    151m
kube-system   rke2-coredns-rke2-coredns-autoscaler-5b89b754bd-xm9pn   0/1     CrashLoopBackOff   33 (2m32s ago)   151m
  • Logs
k  logs hubble-ui-5f75567858-9sgst -n kube-system -c frontend

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2025/05/04 05:49:31 [notice] 1#1: using the "epoll" event method
2025/05/04 05:49:31 [notice] 1#1: nginx/1.27.3
2025/05/04 05:49:31 [notice] 1#1: built by gcc 13.2.1 20240309 (Alpine 13.2.1_git20240309)
2025/05/04 05:49:31 [notice] 1#1: OS: Linux 5.4.0-214-generic
2025/05/04 05:49:31 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2025/05/04 05:49:31 [notice] 1#1: start worker processes
2025/05/04 05:49:31 [notice] 1#1: start worker process 23
2025/05/04 05:49:31 [notice] 1#1: start worker process 24
2025/05/04 05:49:31 [notice] 1#1: start worker process 25
2025/05/04 05:50:00 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
2025/05/04 05:50:00 [notice] 24#24: gracefully shutting down
2025/05/04 05:50:00 [notice] 24#24: exiting
2025/05/04 05:50:00 [notice] 25#25: gracefully shutting down
2025/05/04 05:50:00 [notice] 25#25: exiting
2025/05/04 05:50:00 [notice] 24#24: exit
2025/05/04 05:50:00 [notice] 25#25: exit
2025/05/04 05:50:00 [notice] 23#23: gracefully shutting down
2025/05/04 05:50:00 [notice] 23#23: exiting
2025/05/04 05:50:00 [notice] 23#23: exit
2025/05/04 05:50:01 [notice] 1#1: signal 17 (SIGCHLD) received from 24
2025/05/04 05:50:01 [notice] 1#1: worker process 24 exited with code 0
2025/05/04 05:50:01 [notice] 1#1: worker process 25 exited with code 0
2025/05/04 05:50:01 [notice] 1#1: signal 29 (SIGIO) received
2025/05/04 05:50:01 [notice] 1#1: signal 17 (SIGCHLD) received from 25
2025/05/04 05:50:01 [notice] 1#1: signal 17 (SIGCHLD) received from 23
2025/05/04 05:50:01 [notice] 1#1: worker process 23 exited with code 0
2025/05/04 05:50:01 [notice] 1#1: exit
root@ubuntu2004:~#

k logs hubble-ui-5f75567858-9sgst -n kube-system -c backend
time="2025-05-04T03:46:20Z" level=info msg="using fallback value for env var" fallback=false var=GOPS_ENABLED
time="2025-05-04T03:46:20Z" level=info msg="using fallback value for env var" fallback=false var=TLS_TO_RELAY_ENABLED
time="2025-05-04T03:46:20Z" level=info msg="TLS to hubble-relay is not enabled"
time="2025-05-04T03:46:20Z" level=info msg="using fallback value for env var" fallback=false var=CORS_ENABLED
time="2025-05-04T03:46:20Z" level=info msg="using fallback value for env var" fallback=false var=E2E_TEST_MODE
time="2025-05-04T03:46:20Z" level=info msg="using fallback value for env var" fallback= var=E2E_LOGFILES_BASEPATH
time="2025-05-04T03:46:20Z" level=info msg="running ListenAndServe" apipath=/api component=APIServer port=8090

logs -f hubble-relay-846d47bdc5-558rv -n kube-system
time="2025-05-04T05:50:54.249128997Z" level=info msg="Starting gRPC health server..." addr=":4222" subsys=hubble-relay
time="2025-05-04T05:50:54.249228916Z" level=info msg="Starting gRPC server..." options="{peerTarget:hubble-peer.kube-system.svc.cluster.local.:443 retryTimeout:30000000000 listenAddress::4245 healthListenAddress::4222 metricsListenAddress::9966 log:0xc0006b81c0 serverTLSConfig:<nil> insecureServer:true clientTLSConfig:0xc0006b6480 clusterName:default insecureClient:false observerOptions:[0x2dc27a0 0x2dc2880] grpcMetrics:0xc0000300e0 grpcUnaryInterceptors:[0x2e27b60] grpcStreamInterceptors:[0x2e27dc0]}" subsys=hubble-relay
time="2025-05-04T05:50:54.25000414Z" level=info msg="Starting metrics server..." address=":9966" subsys=hubble-relay
time="2025-05-04T05:51:04.253990267Z" level=info msg="Failed to create peer notify client for peers change notification; will try again after the timeout has expired" connection timeout=30s error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup hubble-peer.kube-system.svc.cluster.local. on 10.96.0.10:53: read udp 10.244.0.198:52737->10.96.0.10:53: i/o timeout" subsys=hubble-relay
time="2025-05-04T05:51:44.26316915Z" level=info msg="Failed to create peer notify client for peers change notification; will try again after the timeout has expired" connection timeout=30s error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup hubble-peer.kube-system.svc.cluster.local. on 10.96.0.10:53: read udp 10.244.0.198:49718->10.96.0.10:53: i/o timeout" subsys=hubble-relay
time="2025-05-04T05:52:03.985796824Z" level=info msg="Stopping server..." subsys=hubble-relay
time="2025-05-04T05:52:03.985957259Z" level=info msg="Server stopped" subsys=hubble-relay

logs -f rke2-coredns-rke2-coredns-86f64967bd-jk5db -n kube-system
maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration SHA512 = d805f5b24f40eeb7f956f336b9ce5b4fd9e00338df867ac1e296637c429923127f746d12258dd5aa757d0328f07b53c17cefb97387608dfe13da6b476c3ecd55
CoreDNS-1.12.1
linux/amd64, go1.23.6 X:boringcrypto, 707c7c10a
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:59905->192.168.29.1:53: i/o timeout
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:56323->192.168.29.1:53: i/o timeout
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:36399->192.168.29.1:53: i/o timeout
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:44670->192.168.29.1:53: i/o timeout
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:37611->192.168.29.1:53: i/o timeout
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:50664->192.168.29.1:53: i/o timeout
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:47740->192.168.29.1:53: i/o timeout
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:60966->192.168.29.1:53: i/o timeout
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:56759->192.168.29.1:53: i/o timeout
[ERROR] plugin/errors: 2 244314236201260380.3098061246195956319. HINFO: read udp 10.244.0.137:54451->192.168.29.1:53: i/o timeout
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 10s
  • Deployment
cilium install --version 1.17.3 \
  --helm-set ipam.mode=kubernetes \
  --helm-set kubeProxyReplacement=true \
  --helm-set cgroup.autoMount.enabled=false \
  --helm-set cgroup.hostRoot=/sys/fs/cgroup \
  --helm-set cni.exclusive=false \
  --helm-set l2announcements.enabled=true \
  --helm-set externalIPs.enabled=true \
  --helm-set socketLB.hostNamespaceOnly=true \
  --helm-set k8sServiceHost=localhost \
  --helm-set k8sServicePort=6443 \
  --helm-set hubble.enabled=true \
  --helm-set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
  --helm-set hubble.metrics.dashboards.enabled=true \
  --helm-set hubble.relay.enabled=true \
  --helm-set hubble.relay.prometheus.enabled=true \
  --helm-set hubble.ui.enabled=true

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions