Created
October 25, 2019 15:22
-
-
Save imranrazakhan/e006282baf0200d0f384a1bc4510796a to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oct 25 16:08:35 xdww002.my.local origin-node[18869]: E1025 16:08:35.035241 18869 event.go:212] Unable to write event: 'Patch https://xdww001.my.local:8443/api/v1/namespaces/kube-system/events/master-etcd-xdww002.my.local.15d0e894c53b9128: dial tcp 10.11.0.178:8443: i/o timeout' (may retry after sleeping) | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:35.205055 I | raft: 1f1b1f1969afb782 is starting a new election at term 366 | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:35.205078 I | raft: 1f1b1f1969afb782 became candidate at term 367 | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:35.205090 I | raft: 1f1b1f1969afb782 received MsgVoteResp from 1f1b1f1969afb782 at term 367 | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:35.205103 I | raft: 1f1b1f1969afb782 [logterm: 243, index: 23857] sent MsgVote request to bbda256326db0d46 at term 367 | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: E1025 14:08:35.425646 19700 reflector.go:136] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to list *v1.HostSubnet: Get https://xdww001.my.local:8443/apis/network.openshift.io/v1/hostsubnets?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:35.458163068Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: I1025 14:08:35.558185 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:38793->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:35.747473516Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:288: Failed to list *v1.Service: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/services?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:35 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:35.985546858Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/default/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.004009 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.004803 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.005821 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.006926 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.007911 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.009100 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.010082 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.011169 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.012403 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.013423 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.104128 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-scheduler: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-scheduler: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:36.139627501Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:36.13964985Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:36.176417352Z caller=notifier.go:473 component=notifier alertmanager=https://10.128.0.25:9094/api/v1/alerts count=1 msg="Error sending alert" err="Post https://10.128.0.25:9094/api/v1/alerts: dial tcp 10.128.0.25:9094: connect: no route to host" | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: I1025 14:08:36.324719 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:34328->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: E1025 16:08:36.351791 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-controller-manager: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-controller-manager: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:36 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:36.986033671Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/default/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.004835 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.005574 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.006647 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.007722 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.008742 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.009798 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.010896 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.011925 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.013097 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.014117 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local origin-node[18869]: W1025 16:08:37.089450 18869 status_manager.go:482] Failed to get status for pod "master-etcd-xdww002.my.local_kube-system(691e4b894fec0cd2e377c5bdab4d51f5)": Get https://xdww001.my.local:8443/api/v1/namespaces/kube-system/pods/master-etcd-xdww002.my.local: dial tcp 10.11.0.178:8443: i/o timeout | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:37.352568812Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: I1025 14:08:37.558669 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:58699->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.571103 1 webhook.go:90] Failed to make webhook authenticator request: Post https://xdww002.my.local:8443/apis/authentication.k8s.io/v1beta1/tokenreviews: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: E1025 16:08:37.571123 1 authentication.go:62] Unable to authenticate the request due to an error: Post https://xdww002.my.local:8443/apis/authentication.k8s.io/v1beta1/tokenreviews: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:37,648 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.SocketTimeoutException: connect timed out | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: time="2019-10-25T14:08:37.849046896Z" level=info msg=response go.version=go1.10.3 http.request.host="10.129.0.18:5000" http.request.id=7f29018c-ff06-4a69-9dcf-59a9247660b9 http.request.method=GET http.request.remoteaddr="10.129.0.1:35030" http.request.uri=/healthz http.request.useragent=kube-probe/1.11+ http.response.duration="38.82µs" http.response.status=200 http.response.written=0 instance.id=f5871a25-5a0c-4fca-9b4f-734a14005449 | |
Oct 25 16:08:37 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:37.986508853Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/default/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.005545 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.006407 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.007480 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.008636 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.009644 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.010680 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.011810 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.012882 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.014047 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: E1025 16:08:38.015031 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.205140 I | raft: 1f1b1f1969afb782 is starting a new election at term 367 | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.205165 I | raft: 1f1b1f1969afb782 became candidate at term 368 | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.205179 I | raft: 1f1b1f1969afb782 received MsgVoteResp from 1f1b1f1969afb782 at term 368 | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.205192 I | raft: 1f1b1f1969afb782 [logterm: 243, index: 23857] sent MsgVote request to bbda256326db0d46 at term 368 | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:38.224713 W | rafthttp: health check for peer bbda256326db0d46 could not connect: dial tcp 10.11.0.178:2380: i/o timeout | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: I1025 14:08:38.324952 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:33658->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:38.365215677Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host" | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:38,648 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.ConnectException: Connection refused (Connection refused) | |
Oct 25 16:08:38 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:38.717964893Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.006231 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.007052 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.008223 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.009254 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.010298 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.011439 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.012502 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.013623 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.014584 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.015645 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.238993 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-scheduler: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-scheduler: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:39.365789942Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: I1025 14:08:39.558815 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:42478->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:39 xdww002.my.local origin-node[18869]: E1025 16:08:39.569016 18869 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://xdww001.my.local:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dxdww002.my.local&limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 14:08:39.569079 19700 reflector.go:136] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to list *v1.NetNamespace: Get https://xdww001.my.local:8443/apis/network.openshift.io/v1/netnamespaces?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 14:08:39.569970 19700 reflector.go:136] github.com/openshift/client-go/network/informers/externalversions/factory.go:101: Failed to list *v1.EgressNetworkPolicy: Get https://xdww001.my.local:8443/apis/network.openshift.io/v1/egressnetworkpolicies?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:39,649 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.ConnectException: Connection refused (Connection refused) | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:39.718550669Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.835269 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-controller-manager: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-controller-manager: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 16:08:39.865195 1 leaderelection.go:234] error retrieving resource lock kube-system/openshift-master-controllers: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/openshift-master-controllers: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:39 xdww002.my.local dockerd-current[6249]: E1025 14:08:39.950580 1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: Get https://172.30.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.007027 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.007843 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.008872 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.009919 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.011037 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.012016 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.013056 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.014070 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.015245 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 16:08:40.016227 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: I1025 14:08:40.325269 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:60711->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:40.366313792Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: 2019/10/25 14:08:40 Failed to dial backend: 'dial tcp 172.30.0.1:443: connect: connection timed out' | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 14:08:40.597674 19700 reflector.go:136] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: Failed to list *core.Service: Get https://xdww001.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:40,649 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.ConnectException: Connection refused (Connection refused) | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:40.719101285Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: E1025 16:08:40.878633 18869 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://xdww001.my.local:8443/api/v1/nodes?fieldSelector=metadata.name%3Dxdww002.my.local&limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout | |
Oct 25 16:08:40 xdww002.my.local dockerd-current[6249]: E1025 14:08:40.878615 19700 reflector.go:136] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: Failed to list *core.Endpoints: Get https://xdww001.my.local:8443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.11.0.178:8443: i/o timeout | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: I1025 16:08:40.943821 18869 kuberuntime_manager.go:513] Container {Name:api Image:docker.io/openshift/origin-control-plane:v3.11.0 Command:[/bin/bash -c] Args:[#!/bin/bash | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: set -euo pipefail | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: if [[ -f /etc/origin/master/master.env ]]; then | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: set -o allexport | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: source /etc/origin/master/master.env | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: fi | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: exec openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=${DEBUG_LOGLEVEL:-2} | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: ] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:master-config ReadOnly:false MountPath:/etc/origin/master/ SubPath: MountPropagation:<nil>} {Name:master-cloud-provider ReadOnly:false MountPath:/etc/origin/cloudprovider/ SubPath: MountPropagation:<nil>} {Name:master-data ReadOnly:false MountPath:/var/lib/origin/ SubPath: MountPropagation:<nil>} {Name:master-pki ReadOnly:false MountPath:/etc/pki SubPath: MountPropagation:<nil>} {Name:host-localtime ReadOnly:false MountPath:/etc/localtime SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:healthz,Port:8443,Host:,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:45,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:healthz/ready,Port:8443,Host:,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:10,TimeoutSeconds:10,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it. | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: I1025 16:08:40.943899 18869 kuberuntime_manager.go:757] checking backoff for container "api" in pod "master-api-xdww002.my.local_kube-system(b24b15710309f0062b93e07af49cb464)" | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: I1025 16:08:40.943984 18869 kuberuntime_manager.go:767] Back-off 2m40s restarting failed container=api pod=master-api-xdww002.my.local_kube-system(b24b15710309f0062b93e07af49cb464) | |
Oct 25 16:08:40 xdww002.my.local origin-node[18869]: E1025 16:08:40.944003 18869 pod_workers.go:186] Error syncing pod b24b15710309f0062b93e07af49cb464 ("master-api-xdww002.my.local_kube-system(b24b15710309f0062b93e07af49cb464)"), skipping: failed to "StartContainer" for "api" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=api pod=master-api-xdww002.my.local_kube-system(b24b15710309f0062b93e07af49cb464)" | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.007756 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.008604 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.009667 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.010654 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.011740 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.012868 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.013995 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.014975 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.016175 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: E1025 16:08:41.017220 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:41.366792497Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: I1025 14:08:41.559195 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:34154->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:41.665228351Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: 2019-10-25 14:08:41,874 ERROR [org.hawkular.openshift.auth.org.hawkular.openshift.namespace.NamespaceListener] (Thread-83) java.net.NoRouteToHostException: No route to host (Host unreachable) | |
Oct 25 16:08:41 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:41.875147621Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: connect: no route to host" | |
Oct 25 16:08:41 xdww002.my.local origin-node[18869]: E1025 16:08:41.875306 18869 kubelet_node_status.go:391] Error updating node status, will retry: error getting node "xdww002.my.local": Get https://xdww001.my.local:8443/api/v1/nodes/xdww002.my.local?timeout=10s: dial tcp 10.11.0.178:8443: connect: no route to host | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.008530 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.009365 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.010449 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.011434 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.012496 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.013530 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.014754 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.015806 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.016832 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.018003 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 14:08:42.238352 1 summary.go:97] error while getting metrics summary from Kubelet xdww001.my.local(10.11.0.178:10250): Get https://10.11.0.178:10250/stats/summary/: dial tcp 10.11.0.178:10250: connect: connection timed out | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: W1025 14:08:42.238378 1 manager.go:102] Failed to get kubelet_summary:10.11.0.178:10250 response in time | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: I1025 14:08:42.325626 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:45351->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:42.367292157Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:42 xdww002.my.local origin-node[18869]: I1025 16:08:42.594789 18869 prober.go:111] Liveness probe for "master-etcd-xdww002.my.local_kube-system(691e4b894fec0cd2e377c5bdab4d51f5):etcd" failed (failure): cluster may be unhealthy: failed to list members | |
Oct 25 16:08:42 xdww002.my.local origin-node[18869]: Error: client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint https://10.11.0.194:2379 exceeded header timeout | |
Oct 25 16:08:42 xdww002.my.local origin-node[18869]: error #0: client: endpoint https://10.11.0.194:2379 exceeded header timeout | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:42.665730263Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:42.705159 I | raft: 1f1b1f1969afb782 is starting a new election at term 368 | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:42.705182 I | raft: 1f1b1f1969afb782 became candidate at term 369 | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:42.705193 I | raft: 1f1b1f1969afb782 received MsgVoteResp from 1f1b1f1969afb782 at term 369 | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:42.705202 I | raft: 1f1b1f1969afb782 [logterm: 243, index: 23857] sent MsgVote request to bbda256326db0d46 at term 369 | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 16:08:42.778224 1 leaderelection.go:234] error retrieving resource lock kube-system/kube-scheduler: Get https://xdww002.my.local:8443/api/v1/namespaces/kube-system/configmaps/kube-scheduler: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:42.87553612Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: E1025 14:08:42.978434 1 reflector.go:136] github.com/openshift/origin/pkg/router/template/service_lookup.go:32: Failed to list *core.Service: Get https://172.30.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout | |
Oct 25 16:08:42 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:42.97841345Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.009224 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.010078 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.011192 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.012278 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.013322 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.014356 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.015501 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.016496 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.017654 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 16:08:43.018693 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.096446202Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.170857715Z caller=notifier.go:473 component=notifier alertmanager=https://10.129.0.23:9094/api/v1/alerts count=1 msg="Error sending alert" err="context deadline exceeded" | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: 2019/10/25 14:08:43 oauthproxy.go:772: 10.129.0.24:35282 Post https://172.30.0.1:443/apis/authentication.k8s.io/v1beta1/tokenreviews: dial tcp 172.30.0.1:443: getsockopt: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.176656197Z caller=notifier.go:473 component=notifier alertmanager=https://10.129.0.23:9094/api/v1/alerts count=2 msg="Error sending alert" err="bad response status 403 Forbidden" | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:43.219866 E | etcdserver: publish error: etcdserver: request timed out | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: 2019-10-25 16:08:43.224927 W | rafthttp: health check for peer bbda256326db0d46 could not connect: dial tcp 10.11.0.178:2380: i/o timeout | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.367810928Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: I1025 14:08:43.559523 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:34519->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 14:08:43.907088 1 reflector.go:205] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:348: Failed to list *v1beta2.StatefulSet: Get https://172.30.0.1:443/apis/apps/v1beta2/namespaces/openshift-monitoring/statefulsets?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.978835778Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: E1025 14:08:43.978879 1 reflector.go:136] github.com/openshift/origin/pkg/router/template/service_lookup.go:32: Failed to list *core.Service: Get https://172.30.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused | |
Oct 25 16:08:43 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:43.995978024Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.010103 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Service: Get https://xdww002.my.local:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.010890 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.Node: Get https://xdww002.my.local:8443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.011871 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.StorageClass: Get https://xdww002.my.local:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.012913 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.PodDisruptionBudget: Get https://xdww002.my.local:8443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.013921 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolume: Get https://xdww002.my.local:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.015041 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.ReplicationController: Get https://xdww002.my.local:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.016104 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.ReplicaSet: Get https://xdww002.my.local:8443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.017226 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1.PersistentVolumeClaim: Get https://xdww002.my.local:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.018269 1 reflector.go:136] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list *v1.Pod: Get https://xdww002.my.local:8443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: E1025 16:08:44.019366 1 reflector.go:136] k8s.io/client-go/informers/factory.go:130: Failed to list *v1beta1.StatefulSet: Get https://xdww002.my.local:8443/apis/apps/v1beta1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.11.0.194:8443: connect: connection refused | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:44.057971495Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Endpoints: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/endpoints?resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout" | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:44.097047797Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/openshift-monitoring/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: I1025 14:08:44.325923 19700 logs.go:49] skydns: failure to forward request "read udp 10.11.0.194:36292->10.11.0.194:53: i/o timeout" | |
Oct 25 16:08:44 xdww002.my.local dockerd-current[6249]: level=error ts=2019-10-25T14:08:44.368412397Z caller=main.go:218 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:289: Failed to list *v1.Pod: Get https://172.30.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0: dial tcp 172.30.0.1:443: connect: connection refused" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment