Skip to content

Instantly share code, notes, and snippets.

@james64
Created December 1, 2020 16:06
Show Gist options
  • Save james64/882086b05e207f31a9899ccc5ee1b710 to your computer and use it in GitHub Desktop.
Save james64/882086b05e207f31a9899ccc5ee1b710 to your computer and use it in GitHub Desktop.
Node ID: '3048cdda-bdc9-e93e-54eb-809e8c03ae05'
Node name: 'consul-server-1'
Cluster Addr: 10.233.69.201 (LAN: 8301, WAN: 8302)
==> Starting Consul agent...
Version: '1.9.0'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
2020-12-01T14:44:06.692Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers
2020-12-01T14:44:06.702Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers
2020-12-01T14:44:06.726Z [INFO] agent.server.raft: restored from snapshot: id=222-1458580-1606818035001
2020-12-01T14:44:06.885Z [INFO] agent.server.raft: initial configuration: index=1462077 servers="[{Suffrage:Voter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.53:8300} {Suffrage:Nonvoter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300}]"
2020-12-01T14:44:06.885Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.69.201:8300 [Follower]" leader=
2020-12-01T14:44:06.887Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.dc1 10.233.69.201
2020-12-01T14:44:06.891Z [INFO] agent.router: Initializing LAN area manager
2020-12-01T14:44:06.892Z [INFO] agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T14:44:06.892Z [INFO] agent.server: Raft data found, disabling bootstrap mode
2020-12-01T14:44:06.891Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
2020-12-01T14:44:06.892Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-1.dc1 area=wan
2020-12-01T14:44:06.899Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
2020-12-01T14:44:06.899Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
2020-12-01T14:44:06.898Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2020-12-01T14:44:06.899Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2020-12-01T14:44:06.899Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2020-12-01T14:44:06.900Z [INFO] agent: Joining cluster...: cluster=LAN
2020-12-01T14:44:06.900Z [INFO] agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc]
2020-12-01T14:44:06.900Z [INFO] agent: started state syncer
==> Consul agent running!
2020-12-01T14:44:06.912Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
2020-12-01T14:44:06.912Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.53
2020-12-01T14:44:06.912Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
2020-12-01T14:44:06.913Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T14:44:06.913Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200
2020-12-01T14:44:06.913Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a dead message (from: consul-server-1)
2020-12-01T14:44:06.914Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
2020-12-01T14:44:06.914Z [INFO] agent.server: Adding LAN server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
2020-12-01T14:44:06.915Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-2.dc1 10.233.68.21
2020-12-01T14:44:06.915Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.53
2020-12-01T14:44:06.915Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a dead message (from: consul-server-1.dc1)
2020-12-01T14:44:06.915Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-2.dc1 area=wan
2020-12-01T14:44:06.915Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
2020-12-01T14:44:06.925Z [WARN] agent.server.raft: failed to get previous log: previous-index=1462084 last-index=1462077 error="log not found"
2020-12-01T14:44:06.928Z [WARN] agent.server.memberlist.lan: memberlist: Failed to resolve consul-server-1.consul-server.consul.svc: lookup consul-server-1.consul-server.consul.svc on 10.233.0.3:53: no such host
2020-12-01T14:44:06.933Z [INFO] agent: (LAN) joined: number_of_nodes=2
2020-12-01T14:44:06.933Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2
2020-12-01T14:44:07.215Z [INFO] agent: Synced node info
2020-12-01T14:44:07.220Z [INFO] agent: Synced service: service=devportal-be-8080
2020-12-01T14:44:15.067Z [INFO] agent.server.serf.wan: serf: EventMemberLeave: consul-server-0.dc1 10.233.67.53
2020-12-01T14:44:15.067Z [INFO] agent.server: Handled event for server in area: event=member-leave server=consul-server-0.dc1 area=wan
2020-12-01T14:44:18.880Z [INFO] agent.server.serf.lan: serf: EventMemberLeave: consul-server-0 10.233.67.53
2020-12-01T14:44:18.880Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
2020-12-01T14:44:20.892Z [INFO] agent.server: New leader elected: payload=consul-server-2
2020-12-01T14:44:21.218Z [INFO] agent: Synced check: check=service:devportal-be-8080
2020-12-01T14:44:32.514Z [INFO] agent.server.memberlist.lan: memberlist: Updating address for left or failed node consul-server-0 from 10.233.67.53:8301 to 10.233.67.184:8301
2020-12-01T14:44:32.514Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T14:44:32.514Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T14:44:32.516Z [INFO] agent.server.memberlist.wan: memberlist: Updating address for left or failed node consul-server-0.dc1 from 10.233.67.53:8302 to 10.233.67.184:8302
2020-12-01T14:44:32.516Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184
2020-12-01T14:44:32.516Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
2020-12-01T15:01:43.894Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:01:47.892Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:01:47.895Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:01:47.895Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:01:48.881Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:01:48.881Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:01:50.251Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:01:50.251Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:01:52.296Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:02:13.892Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:02:16.888Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc1 has failed, no acks received
2020-12-01T15:02:17.880Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:02:17.880Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:17.892Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:02:17.893Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:02:17.893Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:02:21.950Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:21.950Z [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
2020-12-01T15:02:22.433Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:02:22.433Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:02:22.433Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:29.892Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:02:32.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:02:32.747Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:02:34.892Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:02:36.889Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to consul-server-0.dc1 but other probes failed, network may be misconfigured
2020-12-01T15:02:37.280Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:02:37.280Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:40.811Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: worker3.k8s)
2020-12-01T15:02:40.893Z [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
2020-12-01T15:02:40.893Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:40.896Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20
2020-12-01T15:02:41.096Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
2020-12-01T15:02:45.832Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300
2020-12-01T15:02:46.889Z [INFO] agent.server.memberlist.wan: memberlist: Marking consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2020-12-01T15:02:46.889Z [INFO] agent.server.serf.wan: serf: EventMemberFailed: consul-server-0.dc1 10.233.67.184
2020-12-01T15:02:46.889Z [INFO] agent.server: Handled event for server in area: event=member-failed server=consul-server-0.dc1 area=wan
2020-12-01T15:02:50.745Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184
2020-12-01T15:02:50.745Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
2020-12-01T15:02:52.063Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300
2020-12-01T15:02:52.547Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:02:52.548Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:02:52.548Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:58.738Z [ERROR] agent.server.raft: failed to flush response: error="write tcp 10.233.69.201:8300->10.233.68.21:37574: write: broken pipe"
2020-12-01T15:02:58.747Z [INFO] agent.server: New leader elected: payload=consul-server-2
2020-12-01T15:03:01.392Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300
2020-12-01T15:03:04.747Z [INFO] agent.server: New leader elected: payload=consul-server-2
2020-12-01T15:08:01.892Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:08:05.892Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:08:07.347Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:08:07.947Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:08:07.947Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:08:09.892Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:08:10.747Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-2)
2020-12-01T15:08:10.747Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:08:10.747Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:08:10.747Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment