Skip to content

Instantly share code, notes, and snippets.

@james64
Last active December 1, 2020 15:57
Show Gist options
  • Save james64/f651ae5937b1b19f6ff4af37731c9a18 to your computer and use it in GitHub Desktop.
Save james64/f651ae5937b1b19f6ff4af37731c9a18 to your computer and use it in GitHub Desktop.
==> Starting Consul agent...
Version: '1.9.0'
Node ID: 'c2ce972c-a438-dbeb-6a8c-fc696d3228f7'
Node name: 'worker3.k8s'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 10.233.67.183 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
2020-12-01T14:43:27.809Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2020-12-01T14:43:27.809Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
2020-12-01T14:43:27.810Z [INFO] agent: Started gRPC server: address=[::]:8502 network=tcp
2020-12-01T14:43:27.810Z [INFO] agent: Joining cluster...: cluster=LAN
2020-12-01T14:43:27.810Z [INFO] agent: started state syncer
2020-12-01T14:43:27.810Z [WARN] agent.router.manager: No servers available
2020-12-01T14:43:27.809Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
2020-12-01T14:43:27.810Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2020-12-01T14:43:27.810Z [INFO] agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc]
==> Consul agent running!
2020-12-01T14:43:27.810Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2020-12-01T14:43:27.807Z [WARN] agent.auto_config: Node name "worker3.k8s" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2020-12-01T14:43:27.809Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T14:43:27.809Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2020-12-01T14:43:27.809Z [INFO] agent.router: Initializing LAN area manager
2020-12-01T14:43:27.815Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.152
2020-12-01T14:43:27.815Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.53
2020-12-01T14:43:27.815Z [INFO] agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
2020-12-01T14:43:27.815Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
2020-12-01T14:43:27.815Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a dead message (from: worker3.k8s)
2020-12-01T14:43:27.815Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200
2020-12-01T14:43:27.815Z [INFO] agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
2020-12-01T14:43:27.883Z [INFO] agent: Synced node info
2020-12-01T14:43:27.890Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T14:43:27.898Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T14:43:30.835Z [INFO] agent: (LAN) joined: number_of_nodes=2
2020-12-01T14:43:30.835Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2
2020-12-01T14:43:32.869Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
2020-12-01T14:43:32.869Z [INFO] agent.client: adding server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
2020-12-01T14:43:45.599Z [INFO] agent.client: removing server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
2020-12-01T14:43:45.598Z [INFO] agent.client.serf.lan: serf: EventMemberLeave: consul-server-1 10.233.69.152
2020-12-01T14:43:50.471Z [INFO] agent.client: New leader elected: payload=consul-server-0
2020-12-01T14:44:07.013Z [INFO] agent.client.memberlist.lan: memberlist: Updating address for left or failed node consul-server-1 from 10.233.69.152:8301 to 10.233.69.201:8301
2020-12-01T14:44:07.013Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
2020-12-01T14:44:07.013Z [INFO] agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T14:44:18.880Z [INFO] agent.client.serf.lan: serf: EventMemberLeave: consul-server-0 10.233.67.53
2020-12-01T14:44:18.880Z [INFO] agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
2020-12-01T14:44:20.948Z [INFO] agent.client: New leader elected: payload=consul-server-2
2020-12-01T14:44:32.547Z [INFO] agent.client.memberlist.lan: memberlist: Updating address for left or failed node consul-server-0 from 10.233.67.53:8301 to 10.233.67.184:8301
2020-12-01T14:44:32.547Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T14:44:32.547Z [INFO] agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T14:44:40.071Z [INFO] agent: Deregistered service: service=vault:10.233.67.174:8200
2020-12-01T14:44:40.603Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T14:44:40.609Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T14:45:55.068Z [INFO] agent: Deregistered service: service=vault:10.233.67.174:8200
2020-12-01T14:45:55.531Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T14:45:55.538Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T14:47:10.079Z [INFO] agent: Deregistered service: service=vault:10.233.67.174:8200
2020-12-01T14:47:10.623Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T14:47:10.630Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T14:48:25.067Z [INFO] agent: Deregistered service: service=vault:10.233.67.174:8200
2020-12-01T14:48:25.595Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T14:48:25.603Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T14:48:56.182Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T14:48:56.241Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T14:48:56.249Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:01:44.011Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
2020-12-01T15:01:44.809Z [INFO] agent.client.memberlist.lan: memberlist: Suspect worker1.k8s has failed, no acks received
2020-12-01T15:01:49.809Z [INFO] agent.client.memberlist.lan: memberlist: Suspect worker2.k8s has failed, no acks received
2020-12-01T15:01:50.005Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T15:01:50.014Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:01:51.014Z [ERROR] agent.client: RPC failed to server: method=Session.Apply server=10.233.67.184:8300 error="rpc error making call: rpc error making call: Check 'serfHealth' is in critical state"
2020-12-01T15:01:51.014Z [ERROR] agent.http: Request error: method=PUT url=/v1/session/create from=10.233.67.174:43772 error="rpc error making call: rpc error making call: Check 'serfHealth' is in critical state"
2020-12-01T15:02:06.131Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T15:02:06.143Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:02:12.810Z [INFO] agent.client.memberlist.lan: memberlist: Suspect worker1.k8s has failed, no acks received
2020-12-01T15:02:13.947Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
2020-12-01T15:02:16.811Z [INFO] agent.client.memberlist.lan: memberlist: Suspect consul-server-2 has failed, no acks received
2020-12-01T15:02:22.496Z [INFO] agent.client.memberlist.lan: memberlist: Marking consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:02:22.496Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: consul-server-2 10.233.68.21
2020-12-01T15:02:22.496Z [INFO] agent.client: removing server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
2020-12-01T15:02:22.693Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
2020-12-01T15:02:22.693Z [INFO] agent.client: adding server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
2020-12-01T15:02:23.811Z [WARN] agent.client.memberlist.lan: memberlist: Was able to connect to worker2.k8s but other probes failed, network may be misconfigured
2020-12-01T15:02:25.675Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T15:02:25.703Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:02:28.893Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
2020-12-01T15:02:29.809Z [INFO] agent.client.memberlist.lan: memberlist: Suspect consul-server-1 has failed, no acks received
2020-12-01T15:02:35.809Z [INFO] agent.client.memberlist.lan: memberlist: Suspect consul-server-2 has failed, no acks received
2020-12-01T15:02:40.555Z [ERROR] agent.client: RPC failed to server: method=KVS.Apply server=10.233.69.201:8300 error="rpc error making call: rpc error making call: invalid session "ddd01a54-e467-6a93-49fd-417a68a3a471""
2020-12-01T15:02:40.555Z [ERROR] agent.http: Request error: method=PUT url=/v1/kv/vault/core/lock?acquire=ddd01a54-e467-6a93-49fd-417a68a3a471&flags=3304740253564472344 from=10.233.67.174:43770 error="rpc error making call: rpc error making call: invalid session "ddd01a54-e467-6a93-49fd-417a68a3a471""
2020-12-01T15:02:40.950Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20
2020-12-01T15:02:41.096Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
2020-12-01T15:02:43.099Z [INFO] agent: Deregistered service: service=vault:10.233.67.174:8200
2020-12-01T15:02:44.377Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T15:02:44.403Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:02:53.076Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/core/?keys=&separator=%2F from=10.233.67.174:46846 error="rpc error making call: No cluster leader"
2020-12-01T15:02:53.076Z [ERROR] agent.client: RPC failed to server: method=KVS.ListKeys server=10.233.67.184:8300 error="rpc error making call: No cluster leader"
2020-12-01T15:02:58.748Z [INFO] agent.client: New leader elected: payload=consul-server-2
2020-12-01T15:02:59.265Z [ERROR] agent.client: RPC failed to server: method=KVS.ListKeys server=10.233.67.184:8300 error="rpc error making call: No cluster leader"
2020-12-01T15:02:59.265Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/core/?keys=&separator=%2F from=10.233.67.174:47312 error="rpc error making call: No cluster leader"
2020-12-01T15:03:00.120Z [ERROR] agent.client: RPC failed to server: method=KVS.ListKeys server=10.233.68.21:8300 error="rpc error making call: No cluster leader"
2020-12-01T15:03:00.120Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/core/?keys=&separator=%2F from=10.233.67.174:47356 error="rpc error making call: No cluster leader"
2020-12-01T15:03:01.334Z [ERROR] agent.client: RPC failed to server: method=KVS.Get server=10.233.68.21:8300 error="rpc error making call: No cluster leader"
2020-12-01T15:03:01.334Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/sys/uiconfig_plaintext from=10.233.67.174:47406 error="rpc error making call: No cluster leader"
2020-12-01T15:03:04.503Z [ERROR] agent.http: Request error: method=GET url=/v1/kv/vault/core/?keys=&separator=%2F from=10.233.67.174:47584 error="rpc error making call: No cluster leader"
2020-12-01T15:03:04.503Z [ERROR] agent.client: RPC failed to server: method=KVS.ListKeys server=10.233.68.21:8300 error="rpc error making call: No cluster leader"
2020-12-01T15:03:04.747Z [INFO] agent.client: New leader elected: payload=consul-server-2
2020-12-01T15:03:55.091Z [INFO] agent: Deregistered service: service=vault:10.233.67.174:8200
2020-12-01T15:03:55.576Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T15:03:55.585Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:04:25.801Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:04:25.867Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T15:04:25.879Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:08:01.809Z [INFO] agent.client.memberlist.lan: memberlist: Suspect consul-server-1 has failed, no acks received
2020-12-01T15:08:02.083Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
2020-12-01T15:08:05.809Z [INFO] agent.client.memberlist.lan: memberlist: Suspect worker1.k8s has failed, no acks received
2020-12-01T15:08:09.711Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T15:08:09.723Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
2020-12-01T15:08:10.716Z [ERROR] agent.client: RPC failed to server: method=Session.Apply server=10.233.67.184:8300 error="rpc error making call: rpc error making call: Check 'serfHealth' is in critical state"
2020-12-01T15:08:10.716Z [ERROR] agent.http: Request error: method=PUT url=/v1/session/create from=10.233.67.174:50438 error="rpc error making call: rpc error making call: Check 'serfHealth' is in critical state"
2020-12-01T15:08:10.748Z [INFO] agent.client.memberlist.lan: memberlist: Marking consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:08:10.748Z [INFO] agent.client.serf.lan: serf: EventMemberFailed: consul-server-1 10.233.69.201
2020-12-01T15:08:10.748Z [INFO] agent.client: removing server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T15:08:10.893Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
2020-12-01T15:08:10.893Z [INFO] agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T15:08:12.809Z [WARN] agent.client.memberlist.lan: memberlist: Was able to connect to worker1.k8s but other probes failed, network may be misconfigured
2020-12-01T15:08:25.808Z [INFO] agent: Synced service: service=vault:10.233.67.174:8200
2020-12-01T15:08:25.819Z [INFO] agent: Synced check: check=vault:10.233.67.174:8200:vault-sealed-check
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment