Skip to content

Instantly share code, notes, and snippets.

@james64
Last active December 1, 2020 15:59
Show Gist options
  • Save james64/154d18d10b805d37bafb1cb405007475 to your computer and use it in GitHub Desktop.
Save james64/154d18d10b805d37bafb1cb405007475 to your computer and use it in GitHub Desktop.
Node name: 'worker1.k8s'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 10.233.69.200 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
2020-12-01T14:43:21.998Z [WARN] agent: Node name "worker1.k8s" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
==> Starting Consul agent...
Version: '1.9.0'
Node ID: '3b32d665-7338-5da0-d080-d2d3c0f1fb08'
2020-12-01T14:43:22.012Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2020-12-01T14:43:22.012Z [WARN] agent.router.manager: No servers available
2020-12-01T14:43:22.008Z [WARN] agent.auto_config: Node name "worker1.k8s" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2020-12-01T14:43:22.010Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200
2020-12-01T14:43:22.010Z [INFO] agent.router: Initializing LAN area manager
2020-12-01T14:43:22.010Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2020-12-01T14:43:22.010Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2020-12-01T14:43:22.011Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
2020-12-01T14:43:22.011Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
2020-12-01T14:43:22.011Z [INFO] agent: Joining cluster...: cluster=LAN
2020-12-01T14:43:22.012Z [INFO] agent: started state syncer
==> Consul agent running!
2020-12-01T14:43:22.011Z [INFO] agent: Started gRPC server: address=[::]:8502 network=tcp
2020-12-01T14:43:22.011Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2020-12-01T14:43:22.011Z [INFO] agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc]
2020-12-01T14:43:22.018Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a dead message (from: worker1.k8s)
2020-12-01T14:43:22.018Z [INFO] agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
2020-12-01T14:43:22.018Z [INFO] agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
2020-12-01T14:43:22.018Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.53
2020-12-01T14:43:22.018Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.152
2020-12-01T14:43:22.393Z [INFO] agent: Synced node info
2020-12-01T14:43:25.034Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2
2020-12-01T14:43:25.034Z [INFO] agent: (LAN) joined: number_of_nodes=2
2020-12-01T14:43:27.798Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
2020-12-01T14:43:27.869Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T14:43:32.797Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
2020-12-01T14:43:32.798Z [INFO] agent.client: adding server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
2020-12-01T14:43:45.669Z [INFO] agent.client.serf.lan: serf: EventMemberLeave: consul-server-1 10.233.69.152
2020-12-01T14:43:45.669Z [INFO] agent.client: removing server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
2020-12-01T14:43:50.669Z [INFO] agent.client: New leader elected: payload=consul-server-0
2020-12-01T14:44:06.947Z [INFO] agent.client.memberlist.lan: memberlist: Updating address for left or failed node consul-server-1 from 10.233.69.152:8301 to 10.233.69.201:8301
2020-12-01T14:44:06.947Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
2020-12-01T14:44:06.947Z [INFO] agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T14:44:18.869Z [INFO] agent.client.serf.lan: serf: EventMemberLeave: consul-server-0 10.233.67.53
2020-12-01T14:44:18.869Z [INFO] agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
2020-12-01T14:44:20.948Z [INFO] agent.client: New leader elected: payload=consul-server-2
2020-12-01T14:44:32.547Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T14:44:32.547Z [INFO] agent.client.memberlist.lan: memberlist: Updating address for left or failed node consul-server-0 from 10.233.67.53:8301 to 10.233.67.184:8301
2020-12-01T14:44:32.547Z [INFO] agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:01:47.010Z [INFO] agent.client.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:01:47.895Z [INFO] agent.client.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:01:47.895Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:01:48.748Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:01:48.748Z [INFO] agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:01:49.809Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: worker3.k8s)
2020-12-01T15:01:50.296Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:01:50.296Z [INFO] agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:01:51.011Z [WARN] agent.client.memberlist.lan: memberlist: Was able to connect to consul-server-0 but other probes failed, network may be misconfigured
2020-12-01T15:01:52.017Z [INFO] agent.client.serf.lan: serf: attempting reconnect to worker3.k8s 10.233.67.183:8301
2020-12-01T15:01:52.019Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:02:14.011Z [INFO] agent.client.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:02:17.880Z [INFO] agent.client.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:02:17.880Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:02:17.880Z [INFO] agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:18.011Z [INFO] agent.client.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:02:18.092Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:02:21.011Z [WARN] agent.client.memberlist.lan: memberlist: Was able to connect to worker3.k8s but other probes failed, network may be misconfigured
2020-12-01T15:02:22.009Z [INFO] agent.client.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
2020-12-01T15:02:22.009Z [INFO] agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:22.493Z [INFO] agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:22.492Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:02:22.493Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:02:22.615Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: consul-server-2 10.233.68.21
2020-12-01T15:02:22.616Z [INFO] agent.client: removing server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
2020-12-01T15:02:22.680Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
2020-12-01T15:02:22.680Z [INFO] agent.client: adding server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
2020-12-01T15:02:26.810Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: worker3.k8s)
2020-12-01T15:02:31.011Z [INFO] agent.client.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:02:32.748Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:02:35.011Z [WARN] agent.client.memberlist.lan: memberlist: Was able to connect to consul-server-0 but other probes failed, network may be misconfigured
2020-12-01T15:02:37.260Z [INFO] agent.client.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:02:37.260Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:02:37.260Z [INFO] agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:40.949Z [INFO] agent.client.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
2020-12-01T15:02:40.949Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20
2020-12-01T15:02:40.949Z [INFO] agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:41.081Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
2020-12-01T15:02:52.480Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:02:52.481Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:02:52.481Z [INFO] agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:58.809Z [INFO] agent.client: New leader elected: payload=consul-server-2
2020-12-01T15:03:04.749Z [INFO] agent.client: New leader elected: payload=consul-server-2
2020-12-01T15:08:05.010Z [INFO] agent.client.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:08:07.347Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:08:08.080Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:08:08.080Z [INFO] agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:08:08.080Z [INFO] agent.client.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:08:09.011Z [INFO] agent.client.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:08:10.809Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-2)
2020-12-01T15:08:10.809Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: consul-server-1 10.233.69.201
2020-12-01T15:08:10.809Z [INFO] agent.client: removing server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T15:08:10.880Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:08:10.880Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:08:10.880Z [INFO] agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:08:10.893Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
2020-12-01T15:08:10.893Z [INFO] agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment