Skip to content

Instantly share code, notes, and snippets.

@barkbay

barkbay/notes.md

Last active March 16, 2022 12:17
Show Gist options
  • Save barkbay/ef1c7b516bf93534d216bfa48827b70d to your computer and use it in GitHub Desktop.
Save barkbay/ef1c7b516bf93534d216bfa48827b70d to your computer and use it in GitHub Desktop.
Download ZIP
RELOCATING shards during restart
Raw
notes.md

Initial state

  • ECK 1.9.0
  • 5 nodes cluster with a cold tier:
pod/elasticsearch-gcs-sample-es-cold-0      1/1     Running   0             28m   10.124.98.26   gke-michael-dev1-default-pool-d46d259a-9cqn   <none>           <none>
pod/elasticsearch-gcs-sample-es-cold-1      1/1     Running   0             30m   10.124.97.32   gke-michael-dev1-default-pool-3c8b09db-lbmw   <none>           <none>
pod/elasticsearch-gcs-sample-es-cold-2      1/1     Running   0             31m   10.124.96.18   gke-michael-dev1-default-pool-b138cacd-cncm   <none>           <none>
pod/elasticsearch-gcs-sample-es-hot-0       1/1     Running   0             53m   10.124.98.25   gke-michael-dev1-default-pool-d46d259a-9cqn   <none>           <none>
pod/elasticsearch-gcs-sample-es-hot-1       1/1     Running   0             55m   10.124.97.29   gke-michael-dev1-default-pool-3c8b09db-lbmw   <none>           <none>
pod/elasticsearch-gcs-sample-es-master-0    1/1     Running   0             79m   10.124.98.21   gke-michael-dev1-default-pool-d46d259a-9cqn   <none>           <none>
  • Initial shards allocation:
09:54:21 CET
.ds-metricbeat-8.0.0-2022.03.16-000014                        0 p STARTED 12597  10.3mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000014                        0 r STARTED 12629  10.3mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000001               0 p STARTED 74649  48.8mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000002               0 p STARTED 72449  47.7mb 10.124.96.15 elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000004               0 p STARTED 70675  47.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000006               0 p STARTED 83033  56.3mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000008               0 p STARTED 28362  19.6mb 10.124.96.15 elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000010               0 p STARTED 28491  19.5mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000012               0 p STARTED 31747  21.6mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0

Operator is upgraded to 2.1

  1. Operator deletes the elasticsearch-gcs-sample-es-cold-2 once it gets a COMPLETE status from the Shutdown API:
elastic-operator-0 - {"log.level":"info","@timestamp":"2022-03-16T09:56:02.430Z","log.logger":"driver","message":"Node shutdown complete, can start node deletion","service.version":"2.1.0+02a8d7c7","service.type":"eck","ecs.version":"1.4.0","namespace":"test-gcs","es_name":"elasticsearch-gcs-sample","type":"RESTART","node":"elasticsearch-gcs-sample-es-cold-2"}
elastic-operator-0 - {"log.level":"info","@timestamp":"2022-03-16T09:56:02.430Z","log.logger":"driver","message":"Deleting pod for rolling upgrade","service.version":"2.1.0+02a8d7c7","service.type":"eck","ecs.version":"1.4.0","es_name":"elasticsearch-gcs-sample","namespace":"test-gcs","pod_name":"elasticsearch-gcs-sample-es-cold-2","pod_uid":"a89b2529-af00-4a60-8317-dac82c85c713"}
  1. Shutdown is logged by Elasticsearch
{
	"type": "server",
	"timestamp": "2022-03-16T09:56:02,275Z",
	"level": "INFO",
	"component": "o.e.x.s.TransportPutShutdownNodeAction",
	"cluster.name": "elasticsearch-gcs-sample",
	"node.name": "elasticsearch-gcs-sample-es-master-0",
	"message": "creating shutdown record {nodeId=[NV95U8goRt2-euvASCg4pg], type=[RESTART], reason=[79026]}",
	"cluster.uuid": "G9KbJxhTRbi_ErW56ylscw",
	"node.id": "uyg2gRP4RwaZfUXZt56ZvA"
}
  1. Shards are relocated around 50 seconds later, from elasticsearch-gcs-sample-es-cold-2 to elasticsearch-gcs-sample-es-cold-0 and elasticsearch-gcs-sample-es-cold-1 why ?
09:56:51 CET
.ds-metricbeat-8.0.0-2022.03.16-000014                        0 p STARTED 27803    19mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000014                        0 r STARTED 27803    19mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 p STARTED  1774   1.9mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 r STARTED  1774   1.8mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000001               0 p STARTED 74649  48.8mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000002               0 p STARTED 72449  47.7mb 10.124.96.15 elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000004               0 p STARTED 70675  47.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000006               0 p STARTED 83033  56.3mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000008               0 p STARTED 28362  19.6mb 10.124.96.15 elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000010               0 p STARTED 28491  19.5mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000012               0 p STARTED 31747  21.6mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
09:56:52 CET
.ds-metricbeat-8.0.0-2022.03.16-000014                        0 p STARTED      27803    19mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000014                        0 r STARTED      27803    19mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 p STARTED       1774   1.9mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 r STARTED       1774   1.8mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000001               0 p STARTED      74649  48.8mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000002               0 p INITIALIZING               10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000004               0 p STARTED      70675  47.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000006               0 p STARTED      83033  56.3mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000008               0 p STARTED      28362  19.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000010               0 p STARTED      28491  19.5mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000012               0 p STARTED      31747  21.6mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
  1. Elasticsearch logs mentions the lost of elasticsearch-gcs-sample-es-cold-2 at the same time:
elasticsearch-gcs-sample-es-master-0 - {"type": "server", "timestamp": "2022-03-16T09:56:52,674Z", "level": "INFO", "component": "o.e.t.ClusterConnectionManager", "cluster.name": "elasticsearch-gcs-sample", "node.name": "elasticsearch-gcs-sample-es-master-0", "message": "transport connection to [{elasticsearch-gcs-sample-es-cold-2}{NV95U8goRt2-euvASCg4pg}{r43_DnD9R8mZAye_V7Mxzw}{10.124.96.15}{10.124.96.15:9300}{c}] closed by remote", "cluster.uuid": "G9KbJxhTRbi_ErW56ylscw", "node.id": "uyg2gRP4RwaZfUXZt56ZvA"  }
elasticsearch-gcs-sample-es-master-0 - {"type": "server", "timestamp": "2022-03-16T09:56:52,676Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elasticsearch-gcs-sample", "node.name": "elasticsearch-gcs-sample-es-master-0", "message": "Cluster health status changed from [GREEN] to [RED] (reason: [{elasticsearch-gcs-sample-es-cold-2}{NV95U8goRt2-euvASCg4pg}{r43_DnD9R8mZAye_V7Mxzw}{10.124.96.15}{10.124.96.15:9300}{c} reason: disconnected]).", "cluster.uuid": "G9KbJxhTRbi_ErW56ylscw", "node.id": "uyg2gRP4RwaZfUXZt56ZvA"  }
elasticsearch-gcs-sample-es-master-0 - {"type": "server", "timestamp": "2022-03-16T09:56:52,679Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elasticsearch-gcs-sample", "node.name": "elasticsearch-gcs-sample-es-master-0", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [{elasticsearch-gcs-sample-es-cold-2}{NV95U8goRt2-euvASCg4pg}{r43_DnD9R8mZAye_V7Mxzw}{10.124.96.15}{10.124.96.15:9300}{c} reason: disconnected]).", "cluster.uuid": "G9KbJxhTRbi_ErW56ylscw", "node.id": "uyg2gRP4RwaZfUXZt56ZvA"  }
elasticsearch-gcs-sample-es-master-0 - {"type": "server", "timestamp": "2022-03-16T09:56:52,679Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "elasticsearch-gcs-sample", "node.name": "elasticsearch-gcs-sample-es-master-0", "message": "node-left[{elasticsearch-gcs-sample-es-cold-2}{NV95U8goRt2-euvASCg4pg}{r43_DnD9R8mZAye_V7Mxzw}{10.124.96.15}{10.124.96.15:9300}{c} reason: disconnected], term: 2, version: 691, delta: removed {{elasticsearch-gcs-sample-es-cold-2}{NV95U8goRt2-euvASCg4pg}{r43_DnD9R8mZAye_V7Mxzw}{10.124.96.15}{10.124.96.15:9300}{c}}", "cluster.uuid": "G9KbJxhTRbi_ErW56ylscw", "node.id": "uyg2gRP4RwaZfUXZt56ZvA"  }
elasticsearch-gcs-sample-es-master-0 - {"type": "server", "timestamp": "2022-03-16T09:56:52,733Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elasticsearch-gcs-sample", "node.name": "elasticsearch-gcs-sample-es-master-0", "message": "removed {{elasticsearch-gcs-sample-es-cold-2}{NV95U8goRt2-euvASCg4pg}{r43_DnD9R8mZAye_V7Mxzw}{10.124.96.15}{10.124.96.15:9300}{c}}, term: 2, version: 691, reason: Publication{term=2, version=691}", "cluster.uuid": "G9KbJxhTRbi_ErW56ylscw", "node.id": "uyg2gRP4RwaZfUXZt56ZvA"  }
elasticsearch-gcs-sample-es-master-0 - {"type": "server", "timestamp": "2022-03-16T09:56:52,742Z", "level": "INFO", "component": "o.e.c.r.DelayedAllocationService", "cluster.name": "elasticsearch-gcs-sample", "node.name": "elasticsearch-gcs-sample-es-master-0", "message": "scheduling reroute for delayed shards in [4.9m] (2 delayed shards)", "cluster.uuid": "G9KbJxhTRbi_ErW56ylscw", "node.id": "uyg2gRP4RwaZfUXZt56ZvA"  }
  1. 2 seconds later shutdown record is removed
{
	"type": "server",
	"timestamp": "2022-03-16T09:56:54,390Z",
	"level": "INFO",
	"component": "o.e.x.s.TransportDeleteShutdownNodeAction",
	"cluster.name": "elasticsearch-gcs-sample",
	"node.name": "elasticsearch-gcs-sample-es-master-0",
	"message": "removing shutdown record for node [NV95U8goRt2-euvASCg4pg]",
	"cluster.uuid": "G9KbJxhTRbi_ErW56ylscw",
	"node.id": "uyg2gRP4RwaZfUXZt56ZvA"
}
  1. Shards are relocated back to elasticsearch-gcs-sample-es-cold-2
09:57:38 CET
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 p STARTED  1774   1.9mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 r STARTED  1774   1.8mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000001               0 p STARTED 74649  48.8mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000002               0 p STARTED 72449  47.7mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000004               0 p STARTED 70675  47.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000006               0 p STARTED 83033  56.3mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000008               0 p STARTED 28362  19.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000010               0 p STARTED 28491  19.5mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000012               0 p STARTED 31747  21.6mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000014               0 p STARTED 27803    19mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1

09:57:39 CET
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 p STARTED     1774   1.9mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 r STARTED     1774   1.8mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000001               0 p STARTED    74649  48.8mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000002               0 p STARTED    72449  47.7mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000004               0 p STARTED    70675  47.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000006               0 p RELOCATING 83033  56.3mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0 -> 10.124.96.18 NV95U8goRt2-euvASCg4pg elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000008               0 p STARTED    28362  19.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000010               0 p STARTED    28491  19.5mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000012               0 p RELOCATING 31747  21.6mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0 -> 10.124.96.18 NV95U8goRt2-euvASCg4pg elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000014               0 p STARTED    27803    19mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
  1. Shards are still relocating until 09:57:43:
09:57:41 CET
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 p STARTED     1774   1.9mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 r STARTED     1774   1.8mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000001               0 p STARTED    74649  48.8mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000002               0 p STARTED    72449  47.7mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000004               0 p STARTED    70675  47.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000006               0 p RELOCATING 83033  56.3mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0 -> 10.124.96.18 NV95U8goRt2-euvASCg4pg elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000008               0 p STARTED    28362  19.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000010               0 p STARTED    28491  19.5mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000012               0 p RELOCATING 31747  21.6mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0 -> 10.124.96.18 NV95U8goRt2-euvASCg4pg elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000014               0 p STARTED    27803    19mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
09:57:42 CET
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 p STARTED     1774   1.9mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 r STARTED     1774   1.8mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000001               0 p STARTED    74649  48.8mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000002               0 p STARTED    72449  47.7mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000004               0 p STARTED    70675  47.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000006               0 p RELOCATING 83033  56.3mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0 -> 10.124.96.18 NV95U8goRt2-euvASCg4pg elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000008               0 p STARTED    28362  19.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000010               0 p STARTED    28491  19.5mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000012               0 p STARTED    31747  21.6mb 10.124.96.18 elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000014               0 p RELOCATING 27803    19mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1 -> 10.124.98.24 BJAuLXBaRhelmJe98L_v1A elasticsearch-gcs-sample-es-cold-0
  • At 09:57:43 shards location is stable:
09:57:43 CET
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 p STARTED  1774   1.9mb 10.124.98.25 elasticsearch-gcs-sample-es-hot-0
.ds-metricbeat-8.0.0-2022.03.16-000016                        0 r STARTED  1774   1.8mb 10.124.97.29 elasticsearch-gcs-sample-es-hot-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000001               0 p STARTED 74649  48.8mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000002               0 p STARTED 72449  47.7mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
restored-.ds-metricbeat-8.0.0-2022.03.16-000004               0 p STARTED 70675  47.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000006               0 p STARTED 83033  56.3mb 10.124.96.18 elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000008               0 p STARTED 28362  19.6mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000010               0 p STARTED 28491  19.5mb 10.124.97.26 elasticsearch-gcs-sample-es-cold-1
restored-.ds-metricbeat-8.0.0-2022.03.16-000012               0 p STARTED 31747  21.6mb 10.124.96.18 elasticsearch-gcs-sample-es-cold-2
restored-.ds-metricbeat-8.0.0-2022.03.16-000014               0 p STARTED 27803    19mb 10.124.98.24 elasticsearch-gcs-sample-es-cold-0
  • Operator deletes the next nodes:
elastic-operator-0 - {"log.level":"info","@timestamp":"2022-03-16T09:57:45.025Z","log.logger":"driver","message":"Node shutdown complete, can start node deletion","service.version":"2.1.0+02a8d7c7","service.type":"eck","ecs.version":"1.4.0","namespace":"test-gcs","es_name":"elasticsearch-gcs-sample","type":"RESTART","node":"elasticsearch-gcs-sample-es-cold-1"}
elastic-operator-0 - {"log.level":"info","@timestamp":"2022-03-16T09:57:45.025Z","log.logger":"driver","message":"Deleting pod for rolling upgrade","service.version":"2.1.0+02a8d7c7","service.type":"eck","ecs.version":"1.4.0","es_name":"elasticsearch-gcs-sample","namespace":"test-gcs","pod_name":"elasticsearch-gcs-sample-es-cold-1","pod_uid":"3f63ee22-a4af-47e2-b457-1299f816b018"}

Same thing is observed during the restart of the 2 other cold nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment