Last active
March 21, 2023 16:06
-
-
Save rayjanoka/82a094110e1de07d75b5a961f1ee0d02 to your computer and use it in GitHub Desktop.
vitastor #1 journals
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- Logs begin at Tue 2023-03-21 12:40:24 UTC. -- | |
| Mar 21 15:42:41 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:42:46 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:42:51 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:42:56 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:01 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:06 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:11 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:12 bmhv3 etcd[465468]: Starting auto-compaction at revision 1022 (retention: 10 revisions) | |
| Mar 21 15:43:12 bmhv3 etcd[465468]: Finished auto-compaction at revision 1022 | |
| Mar 21 15:43:12 bmhv3 etcd[465468]: store.index: compact 1022 | |
| Mar 21 15:43:12 bmhv3 etcd[465468]: finished scheduled compaction at 1022 (took 398.302834ms) | |
| Mar 21 15:43:16 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:21 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:26 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:31 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:36 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:41 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:42 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:42 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:44 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:44 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:45 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:45 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:46 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:46 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:46 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:47 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:47 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:51 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:43:56 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:01 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:06 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:11 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:16 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:21 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:26 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:31 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:36 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:41 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:43 bmhv3 systemd-logind[990]: New session 25 of user ubuntu. | |
| Mar 21 15:44:43 bmhv3 systemd[1]: Started Session 25 of user ubuntu. | |
| Mar 21 15:44:46 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:51 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:44:56 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:01 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:06 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:11 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:16 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:21 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:26 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:31 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:36 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:41 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:46 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:51 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:45:56 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:01 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:06 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:11 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:16 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:21 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:26 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:31 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:36 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:41 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:46 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:52 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:46:57 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:02 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:07 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:12 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:17 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:22 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:27 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:32 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:37 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:38 bmhv3 etcd[465468]: closed an existing TCP streaming connection with peer cfefa00537b3d384 (stream Message writer) | |
| Mar 21 15:47:38 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message writer) | |
| Mar 21 15:47:42 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:44 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:44 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:44 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:44 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:44 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:47:44 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream Message (read tcp 10.10.0.167:43564->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:47:44 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:47:44 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active | |
| Mar 21 15:47:44 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:47:45 bmhv3 etcd[465468]: closed an existing TCP streaming connection with peer cfefa00537b3d384 (stream Message writer) | |
| Mar 21 15:47:45 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message writer) | |
| Mar 21 15:47:45 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:45 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:45 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:45 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:46 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:46 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:46 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:46 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:47 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:47 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:47 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:47 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:47 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:48 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:48 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:48 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:48 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:51 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:47:51 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream Message (read tcp 10.10.0.167:50380->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:47:51 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:47:51 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active | |
| Mar 21 15:47:51 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:47:52 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:56 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader) | |
| Mar 21 15:47:56 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.167:43552->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:47:56 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:47:56 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active | |
| Mar 21 15:47:56 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader) | |
| Mar 21 15:47:57 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:47:57 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:47:57 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream Message (read tcp 10.10.0.167:45822->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:47:57 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:47:59 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active | |
| Mar 21 15:47:59 bmhv3 etcd[465468]: closed an existing TCP streaming connection with peer cfefa00537b3d384 (stream Message writer) | |
| Mar 21 15:47:59 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message writer) | |
| Mar 21 15:47:59 bmhv3 etcd[465468]: failed to dial cfefa00537b3d384 on stream Message (dial tcp 10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:47:59 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:48:01 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active | |
| Mar 21 15:48:01 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:48:02 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:48:07 bmhv3 node[467775]: Waiting to become master | |
| Mar 21 15:48:07 bmhv3 etcd[465468]: closed an existing TCP streaming connection with peer cfefa00537b3d384 (stream Message writer) | |
| Mar 21 15:48:07 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message writer) | |
| Mar 21 15:48:07 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader) | |
| Mar 21 15:48:07 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.167:45846->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:07 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:48:07 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active | |
| Mar 21 15:48:07 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader) | |
| Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 [logterm: 21, index: 2714, vote: cb7af46e9829ddc8] ignored MsgVote from cfefa00537b3d384 [logterm: 21, index: 2714] at term 21 | |
| : lease is not expired (remaining ticks: 1) | |
| Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 [term: 21] received a MsgAppResp message with higher term from cfefa00537b3d384 [term: 22] | |
| Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 became follower at term 22 | |
| Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: raft.node: cb7af46e9829ddc8 lost leader cb7af46e9829ddc8 at term 22 | |
| Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 [term: 22] ignored a MsgAppResp message with lower term from af5ac16031361c6a [term: 21] | |
| Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 [term: 22] ignored a MsgAppResp message with lower term from 607474ecd5112191 [term: 21] | |
| Mar 21 15:48:10 bmhv3 etcd[465468]: raft2023/03/21 15:48:10 INFO: cb7af46e9829ddc8 [logterm: 21, index: 2715, vote: 0] cast MsgVote for af5ac16031361c6a [logterm: 21, index: 2715] at term 22 | |
| Mar 21 15:48:10 bmhv3 etcd[465468]: raft2023/03/21 15:48:10 INFO: raft.node: cb7af46e9829ddc8 elected leader af5ac16031361c6a at term 22 | |
| Mar 21 15:48:10 bmhv3 node[467775]: failed to query etcd: timeout | |
| Mar 21 15:48:10 bmhv3 node[467775]: events.js:174 | |
| Mar 21 15:48:10 bmhv3 node[467775]: throw er; // Unhandled 'error' event | |
| Mar 21 15:48:10 bmhv3 node[467775]: ^ | |
| Mar 21 15:48:10 bmhv3 node[467775]: Error: socket hang up | |
| Mar 21 15:48:10 bmhv3 node[467775]: at createHangUpError (_http_client.js:332:15) | |
| Mar 21 15:48:10 bmhv3 node[467775]: at Socket.socketCloseListener (_http_client.js:373:25) | |
| Mar 21 15:48:10 bmhv3 node[467775]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:10 bmhv3 node[467775]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:10 bmhv3 node[467775]: Emitted 'error' event at: | |
| Mar 21 15:48:10 bmhv3 node[467775]: at Socket.socketCloseListener (_http_client.js:373:11) | |
| Mar 21 15:48:10 bmhv3 node[467775]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:10 bmhv3 node[467775]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:10 bmhv3 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE | |
| Mar 21 15:48:10 bmhv3 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'. | |
| Mar 21 15:48:12 bmhv3 etcd[465468]: health check for peer cfefa00537b3d384 could not connect: dial tcp 10.10.0.164:2380: i/o timeout | |
| Mar 21 15:48:14 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader) | |
| Mar 21 15:48:14 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.167:58980->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:14 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:48:14 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:48:17 bmhv3 etcd[465468]: health check for peer cfefa00537b3d384 could not connect: dial tcp 10.10.0.164:2380: i/o timeout | |
| Mar 21 15:48:20 bmhv3 systemd[1]: vitastor-mon.service: Scheduled restart job, restart counter is at 1. | |
| Mar 21 15:48:20 bmhv3 systemd[1]: Stopped Vitastor monitor. | |
| Mar 21 15:48:20 bmhv3 systemd[1]: Started Vitastor monitor. | |
| Mar 21 15:48:22 bmhv3 etcd[465468]: health check for peer cfefa00537b3d384 could not connect: dial tcp 10.10.0.164:2380: i/o timeout | |
| Mar 21 15:48:22 bmhv3 node[475469]: failed to query etcd: timeout | |
| Mar 21 15:48:22 bmhv3 node[475469]: events.js:174 | |
| Mar 21 15:48:22 bmhv3 node[475469]: throw er; // Unhandled 'error' event | |
| Mar 21 15:48:22 bmhv3 node[475469]: ^ | |
| Mar 21 15:48:22 bmhv3 node[475469]: Error: socket hang up | |
| Mar 21 15:48:22 bmhv3 node[475469]: at createHangUpError (_http_client.js:332:15) | |
| Mar 21 15:48:22 bmhv3 node[475469]: at Socket.socketCloseListener (_http_client.js:373:25) | |
| Mar 21 15:48:22 bmhv3 node[475469]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:22 bmhv3 node[475469]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:22 bmhv3 node[475469]: Emitted 'error' event at: | |
| Mar 21 15:48:22 bmhv3 node[475469]: at Socket.socketCloseListener (_http_client.js:373:11) | |
| Mar 21 15:48:22 bmhv3 node[475469]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:22 bmhv3 node[475469]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:22 bmhv3 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE | |
| Mar 21 15:48:22 bmhv3 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'. | |
| Mar 21 15:48:24 bmhv3 etcd[465468]: raft2023/03/21 15:48:24 INFO: cb7af46e9829ddc8 [term: 22] received a MsgVote message with higher term from 607474ecd5112191 [term: 23] | |
| Mar 21 15:48:24 bmhv3 etcd[465468]: raft2023/03/21 15:48:24 INFO: cb7af46e9829ddc8 became follower at term 23 | |
| Mar 21 15:48:24 bmhv3 etcd[465468]: raft2023/03/21 15:48:24 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732, vote: 0] cast MsgVote for 607474ecd5112191 [logterm: 22, index: 2732] at term 23 | |
| Mar 21 15:48:24 bmhv3 etcd[465468]: raft2023/03/21 15:48:24 INFO: raft.node: cb7af46e9829ddc8 lost leader af5ac16031361c6a at term 23 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732, vote: 607474ecd5112191] rejected MsgVote from af5ac16031361c6a [logterm: 22, index: 2732] at term 2 | |
| 3 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 is starting a new election at term 23 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 became candidate at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 received MsgVoteResp from cb7af46e9829ddc8 at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732] sent MsgVote request to 46d9e5656b97850f at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732] sent MsgVote request to 607474ecd5112191 at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732] sent MsgVote request to af5ac16031361c6a at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732] sent MsgVote request to cfefa00537b3d384 at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 received MsgVoteResp from af5ac16031361c6a at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 has received 2 MsgVoteResp votes and 0 vote rejections | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 received MsgVoteResp from 607474ecd5112191 at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 has received 3 MsgVoteResp votes and 0 vote rejections | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 became leader at term 24 | |
| Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: raft.node: cb7af46e9829ddc8 elected leader cb7af46e9829ddc8 at term 24 | |
| Mar 21 15:48:27 bmhv3 etcd[465468]: health check for peer cfefa00537b3d384 could not connect: dial tcp 10.10.0.164:2380: i/o timeout | |
| Mar 21 15:48:28 bmhv3 etcd[465468]: raft2023/03/21 15:48:28 WARN: cb7af46e9829ddc8 stepped down to follower since quorum is not active | |
| Mar 21 15:48:28 bmhv3 etcd[465468]: raft2023/03/21 15:48:28 INFO: cb7af46e9829ddc8 became follower at term 24 | |
| Mar 21 15:48:28 bmhv3 etcd[465468]: raft2023/03/21 15:48:28 INFO: raft.node: cb7af46e9829ddc8 lost leader cb7af46e9829ddc8 at term 24 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 is starting a new election at term 24 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 became candidate at term 25 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 received MsgVoteResp from cb7af46e9829ddc8 at term 25 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to 46d9e5656b97850f at term 25 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to 607474ecd5112191 at term 25 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to af5ac16031361c6a at term 25 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to cfefa00537b3d384 at term 25 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 received MsgVoteResp from af5ac16031361c6a at term 25 | |
| Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 has received 2 MsgVoteResp votes and 0 vote rejections | |
| Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 is starting a new election at term 25 | |
| Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 became candidate at term 26 | |
| Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 received MsgVoteResp from cb7af46e9829ddc8 at term 26 | |
| Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to 46d9e5656b97850f at term 26 | |
| Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to 607474ecd5112191 at term 26 | |
| Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to af5ac16031361c6a at term 26 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- Logs begin at Fri 2023-03-17 18:39:52 UTC. -- [3740/5199] | |
| Mar 21 15:42:18 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:42:23 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:42:28 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:42:33 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:42:38 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:42:43 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:42:48 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:42:53 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:42:58 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:03 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:08 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:12 bmhv4 etcd[144820]: store.index: compact 1022 | |
| Mar 21 15:43:12 bmhv4 etcd[144820]: finished scheduled compaction at 1022 (took 392.280926ms) | |
| Mar 21 15:43:13 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:18 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:23 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:28 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:33 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:38 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:42 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:42 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:43 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:44 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:44 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:45 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:45 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:46 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:46 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:47 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:47 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:48 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:50 bmhv4 systemd[1]: Started libcontainer container c6129cf43061692a6c756adcc67c96fbc0bc5f6f411038632e7ccd01c7a6b1e0. | |
| Mar 21 15:43:53 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:58 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:43:59 bmhv4 systemd[1]: Started libcontainer container 9e6dac849a8e3d0de5b67ec79ea45e5186531ae89751c82e57f86bed208ff832. | |
| Mar 21 15:44:00 bmhv4 systemd[1]: Started libcontainer container eaa50b1961174b84be5b1df6deb956cd660fa03826d4e41e23b6da40562999b6. | |
| Mar 21 15:44:00 bmhv4 systemd[1]: Started libcontainer container a8da5eb9efaec931320042dac1b4d5975ff2e2bb6fd73f9dd8cfcd1c45bca2dd. | |
| Mar 21 15:44:00 bmhv4 systemd[1]: Started libcontainer container 5454035cd3a7110bec2507f833602a3a3227a8e59019df9ec42a37fec8be1809. | |
| Mar 21 15:44:03 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:08 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:13 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:18 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:23 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:24 bmhv4 systemd-logind[992]: New session 132 of user ubuntu. | |
| Mar 21 15:44:24 bmhv4 systemd[1]: Started Session 132 of user ubuntu. | |
| Mar 21 15:44:28 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:33 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:38 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:43 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:48 bmhv4 node[149964]: Waiting to become master Mar 21 15:44:49 bmhv4 systemd[1]: Started libcontainer container cb852b8a5151ec80f7b3f6f69f869178c71d5fd478298e5d4d2bb65dc530691f. | |
| Mar 21 15:44:53 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:44:58 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:03 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:08 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:13 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:18 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:23 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:28 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:33 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:38 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:43 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:46 bmhv4 systemd[1]: Started libcontainer container 804797332d24c3126e41699fe032bf376f6dd5f36f8a73be3a2dc49218251522. | |
| Mar 21 15:45:47 bmhv4 systemd[1]: Started libcontainer container 4b58780c8e2bb55c3b06c0795db01f599497cb88328f83c2221a87b12b7f487d. | |
| Mar 21 15:45:47 bmhv4 systemd[1]: Started libcontainer container e2736254d0254ac66c6fb221eadc879a23cf54e862c99b52b19b6ace833ea269. | |
| Mar 21 15:45:48 bmhv4 systemd[1]: Started libcontainer container 5e10e33b52a9f93a89dda44cd4a683a3d622f0f89985bd95dca51464d1beddcf. | |
| Mar 21 15:45:48 bmhv4 systemd[1]: Started libcontainer container ff18c58bd28cde69d352af0823616f78c0c40952012d724595b156caa52ac1cc. | |
| Mar 21 15:45:48 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:53 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:45:58 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:03 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:08 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:13 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:18 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:23 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:28 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:33 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:38 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:43 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:48 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:53 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:46:58 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:03 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:05 bmhv4 systemd-logind[992]: New session 133 of user ubuntu. | |
| Mar 21 15:47:05 bmhv4 systemd[1]: Started Session 133 of user ubuntu. | |
| Mar 21 15:47:09 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:12 bmhv4 etcd[144820]: time="2023-03-21T15:47:12Z" level=warning msg="error reading websocket message: websocket: close 1006 (abnormal closure): unexpected EOF" | |
| Mar 21 15:47:14 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:16 bmhv4 systemd[1]: Started libcontainer container bfb6c4f6c4e70041687810231dceb78939b2efb8c80394fcf5988c81de013caf. | |
| Mar 21 15:47:19 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:24 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:29 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:34 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:39 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:44 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:49 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:54 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:47:59 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:48:02 bmhv4 etcd[144820]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:48:02 bmhv4 etcd[144820]: failed to read cfefa00537b3d384 on stream Message (read tcp 10.10.0.166:60352->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:02 bmhv4 etcd[144820]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:48:02 bmhv4 etcd[144820]: peer cfefa00537b3d384 became active | |
| Mar 21 15:48:02 bmhv4 etcd[144820]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:48:04 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:48:09 bmhv4 etcd[144820]: raft2023/03/21 15:48:09 INFO: af5ac16031361c6a [logterm: 21, index: 2714, vote: cb7af46e9829ddc8] ignored MsgVote from cfefa00537b3d384 [logterm: 21, index: 2714] at term 21 | |
| : lease is not expired (remaining ticks: 10) | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a is starting a new election at term 21 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a became candidate at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a received MsgVoteResp from af5ac16031361c6a at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a [logterm: 21, index: 2715] sent MsgVote request to 46d9e5656b97850f at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a [logterm: 21, index: 2715] sent MsgVote request to 607474ecd5112191 at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a [logterm: 21, index: 2715] sent MsgVote request to cb7af46e9829ddc8 at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a [logterm: 21, index: 2715] sent MsgVote request to cfefa00537b3d384 at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: raft.node: af5ac16031361c6a lost leader cb7af46e9829ddc8 at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a received MsgVoteResp rejection from cfefa00537b3d384 at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a has received 1 MsgVoteResp votes and 1 vote rejections | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a received MsgVoteResp from 607474ecd5112191 at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a has received 2 MsgVoteResp votes and 1 vote rejections | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a received MsgVoteResp from cb7af46e9829ddc8 at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a has received 3 MsgVoteResp votes and 1 vote rejections | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a became leader at term 22 | |
| Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: raft.node: af5ac16031361c6a elected leader af5ac16031361c6a at term 22 | |
| Mar 21 15:48:10 bmhv4 node[149964]: Waiting to become master | |
| Mar 21 15:48:10 bmhv4 node[149964]: failed to query etcd: timeout | |
| Mar 21 15:48:11 bmhv4 node[149964]: events.js:174 | |
| Mar 21 15:48:11 bmhv4 node[149964]: throw er; // Unhandled 'error' event | |
| Mar 21 15:48:11 bmhv4 node[149964]: ^ | |
| Mar 21 15:48:11 bmhv4 node[149964]: Error: socket hang up | |
| Mar 21 15:48:11 bmhv4 node[149964]: at createHangUpError (_http_client.js:332:15) | |
| Mar 21 15:48:11 bmhv4 node[149964]: at Socket.socketCloseListener (_http_client.js:373:25) | |
| Mar 21 15:48:11 bmhv4 node[149964]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:11 bmhv4 node[149964]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:11 bmhv4 node[149964]: Emitted 'error' event at: | |
| Mar 21 15:48:11 bmhv4 node[149964]: at Socket.socketCloseListener (_http_client.js:373:11) | |
| Mar 21 15:48:11 bmhv4 node[149964]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:11 bmhv4 node[149964]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:11 bmhv4 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE | |
| Mar 21 15:48:11 bmhv4 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'. | |
| Mar 21 15:48:12 bmhv4 etcd[144820]: failed to reach the peerURL(http://10.10.0.164:2380) of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:12 bmhv4 etcd[144820]: cannot get the version of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:12 bmhv4 etcd[144820]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader) | |
| Mar 21 15:48:12 bmhv4 etcd[144820]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.166:60366->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:12 bmhv4 etcd[144820]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:48:15 bmhv4 etcd[144820]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:48:18 bmhv4 etcd[144820]: failed to reach the peerURL(http://10.10.0.164:2380) of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:18 bmhv4 etcd[144820]: cannot get the version of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:21 bmhv4 systemd[1]: vitastor-mon.service: Scheduled restart job, restart counter is at 1. | |
| Mar 21 15:48:21 bmhv4 systemd[1]: Stopped Vitastor monitor. | |
| Mar 21 15:48:21 bmhv4 systemd[1]: Started Vitastor monitor. | |
| Mar 21 15:48:23 bmhv4 etcd[144820]: raft2023/03/21 15:48:23 WARN: af5ac16031361c6a stepped down to follower since quorum is not active | |
| Mar 21 15:48:23 bmhv4 etcd[144820]: raft2023/03/21 15:48:23 INFO: af5ac16031361c6a became follower at term 22 | |
| Mar 21 15:48:23 bmhv4 etcd[144820]: raft2023/03/21 15:48:23 INFO: raft.node: af5ac16031361c6a lost leader af5ac16031361c6a at term 22 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: failed to reach the peerURL(http://10.10.0.164:2380) of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: cannot get the version of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a is starting a new election at term 22 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a became candidate at term 23 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a received MsgVoteResp from af5ac16031361c6a at term 23 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732] sent MsgVote request to 46d9e5656b97850f at term 23 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732] sent MsgVote request to 607474ecd5112191 at term 23 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732] sent MsgVote request to cb7af46e9829ddc8 at term 23 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732] sent MsgVote request to cfefa00537b3d384 at term 23 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a received MsgVoteResp rejection from cb7af46e9829ddc8 at term 23 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a has received 1 MsgVoteResp votes and 1 vote rejections | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [term: 23] received a MsgVote message with higher term from cb7af46e9829ddc8 [term: 24] | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a became follower at term 24 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732, vote: 0] cast MsgVote for cb7af46e9829ddc8 [logterm: 22, index: 2732] at term 24 | |
| Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: raft.node: af5ac16031361c6a elected leader cb7af46e9829ddc8 at term 24 | |
| Mar 21 15:48:25 bmhv4 node[168679]: failed to query etcd: {"error":"etcdserver: leader changed","message":"etcdserver: leader changed","code":14} | |
| Mar 21 15:48:26 bmhv4 etcd[144820]: lost the TCP streaming connection with peer 607474ecd5112191 (stream Message reader) | |
| Mar 21 15:48:26 bmhv4 etcd[144820]: failed to read 607474ecd5112191 on stream Message (read tcp 10.10.0.166:37116->10.10.0.171:2380: i/o timeout) | |
| Mar 21 15:48:26 bmhv4 etcd[144820]: peer 607474ecd5112191 became inactive (message send to peer failed) | |
| Mar 21 15:48:26 bmhv4 etcd[144820]: peer 607474ecd5112191 became active | |
| Mar 21 15:48:26 bmhv4 etcd[144820]: established a TCP streaming connection with peer 607474ecd5112191 (stream Message reader) | |
| Mar 21 15:48:29 bmhv4 etcd[144820]: raft2023/03/21 15:48:29 INFO: af5ac16031361c6a [term: 24] received a MsgVote message with higher term from cb7af46e9829ddc8 [term: 25] | |
| Mar 21 15:48:29 bmhv4 etcd[144820]: raft2023/03/21 15:48:29 INFO: af5ac16031361c6a became follower at term 25 | |
| Mar 21 15:48:29 bmhv4 etcd[144820]: raft2023/03/21 15:48:29 INFO: af5ac16031361c6a [logterm: 24, index: 2736, vote: 0] cast MsgVote for cb7af46e9829ddc8 [logterm: 24, index: 2736] at term 25 | |
| Mar 21 15:48:29 bmhv4 etcd[144820]: raft2023/03/21 15:48:29 INFO: raft.node: af5ac16031361c6a lost leader cb7af46e9829ddc8 at term 25 | |
| Mar 21 15:48:30 bmhv4 node[168679]: failed to query etcd: timeout | |
| Mar 21 15:48:30 bmhv4 node[168679]: events.js:174 | |
| Mar 21 15:48:30 bmhv4 node[168679]: throw er; // Unhandled 'error' event | |
| Mar 21 15:48:30 bmhv4 node[168679]: ^ | |
| Mar 21 15:48:30 bmhv4 node[168679]: Error: socket hang up | |
| Mar 21 15:48:30 bmhv4 node[168679]: at createHangUpError (_http_client.js:332:15) | |
| Mar 21 15:48:30 bmhv4 node[168679]: at Socket.socketCloseListener (_http_client.js:373:25) | |
| Mar 21 15:48:30 bmhv4 node[168679]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:30 bmhv4 node[168679]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:30 bmhv4 node[168679]: Emitted 'error' event at: | |
| Mar 21 15:48:30 bmhv4 node[168679]: at Socket.socketCloseListener (_http_client.js:373:11) | |
| Mar 21 15:48:30 bmhv4 node[168679]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:30 bmhv4 node[168679]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:30 bmhv4 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE | |
| Mar 21 15:48:30 bmhv4 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- Logs begin at Tue 2023-03-21 13:08:47 UTC. -- | |
| Mar 21 15:43:12 bmhv7 etcd[43768]: store.index: compact 1022 | |
| Mar 21 15:43:12 bmhv7 etcd[43768]: finished scheduled compaction at 1022 (took 392.040541ms) | |
| Mar 21 15:43:36 bmhv7 systemd[1]: Created slice libcontainer container kubepods-burstable-pod9ac5c3e2_566f_48fd_9332_9c845c7e25d1.slice. | |
| Mar 21 15:43:37 bmhv7 systemd-udevd[52478]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. | |
| Mar 21 15:43:37 bmhv7 systemd-udevd[52477]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. | |
| Mar 21 15:43:37 bmhv7 networkd-dispatcher[987]: WARNING:Unknown index 33 seen, reloading interface list | |
| Mar 21 15:43:37 bmhv7 systemd-udevd[52478]: Using default interface naming scheme 'v245'. | |
| Mar 21 15:43:37 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Link UP | |
| Mar 21 15:43:37 bmhv7 systemd-udevd[52477]: Using default interface naming scheme 'v245'. Mar 21 15:43:37 bmhv7 kernel: eth0: renamed from tmp2e607 Mar 21 15:43:37 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Gained carrier Mar 21 15:43:37 bmhv7 networkd-dispatcher[987]: ERROR:Unknown interface index 33 seen even after reload Mar 21 15:43:37 bmhv7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready | |
| Mar 21 15:43:37 bmhv7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): lxc2de1f96c479d: link becomes ready Mar 21 15:43:37 bmhv7 networkd-dispatcher[987]: WARNING:Unknown index 33 seen, reloading interface list | |
| Mar 21 15:43:37 bmhv7 networkd-dispatcher[987]: ERROR:Unknown interface index 33 seen even after reload | |
| Mar 21 15:43:37 bmhv7 systemd[1]: Started libcontainer container 2e60701dea55ace882d0aaffffdb788ba1adcff3a209ed36572157773e090036. | |
| Mar 21 15:43:37 bmhv7 systemd[1]: Started libcontainer container 6469156091c79ef354fd42b729ad7935f1f5d7613b53d5f510eb6dd696ba6518. | |
| Mar 21 15:43:38 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Gained IPv6LL | |
| Mar 21 15:43:39 bmhv7 systemd[1]: Started libcontainer container a43c58055c9ff58e07cb3ba37b000582e90113e3d30f4a1e830ef1a3001f2e66. Mar 21 15:43:40 bmhv7 systemd[1]: Started libcontainer container e69e69fd116cc1680bd8e64baf347f11d5228740e5efb59ab3b3e7d2125d28b3. | |
| Mar 21 15:43:41 bmhv7 systemd[1]: Started libcontainer container 92ebe5561c73a539e8f4d86e2c35b1f2db17670ef92a703e113634082c92cf1d. | |
| Mar 21 15:43:41 bmhv7 systemd[1]: Started libcontainer container 4906f25538e1c29cb297716d18db0a36eac678107c033f4e2f74619a44fe2467. | |
| Mar 21 15:43:41 bmhv7 systemd[1]: Started libcontainer container fb9df170f0612b08de30cf1e4e9644a08f535b592a1071bc297740b7d1ecbc48. | |
| Mar 21 15:43:42 bmhv7 systemd[1]: Started libcontainer container be0851e05a65eefd3ae24938f6d67863334b234c1eb4e3510bd3545e6989fad0. | |
| Mar 21 15:43:42 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:42 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:44 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:44 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:45 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:45 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:46 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:46 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:47 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:47 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:44:42 bmhv7 systemd[1]: Started libcontainer container 93a962bc9dfcacba9b1e2c84a967c9ee6de4d7328974a881633cfa5b3d41ced8. | |
| Mar 21 15:44:59 bmhv7 systemd-logind[997]: New session 5 of user ubuntu. | |
| Mar 21 15:44:59 bmhv7 systemd[1]: Started Session 5 of user ubuntu. | |
| Mar 21 15:45:41 bmhv7 systemd[1]: Created slice libcontainer container kubepods-besteffort-podf7c4c46b_e818_4e46_ab66_ca8021decabc.slice. | |
| Mar 21 15:45:54 bmhv7 kernel: nbd0: detected capacity change from 0 to 20971520 | |
| Mar 21 15:45:54 bmhv7 kernel: EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. | |
| Mar 21 15:45:55 bmhv7 systemd-udevd[53894]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. | |
| Mar 21 15:45:55 bmhv7 systemd-udevd[53894]: Using default interface naming scheme 'v245'. | |
| Mar 21 15:45:55 bmhv7 networkd-dispatcher[987]: WARNING:Unknown index 35 seen, reloading interface list | |
| Mar 21 15:45:55 bmhv7 systemd-udevd[53895]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable. | |
| Mar 21 15:45:55 bmhv7 kernel: eth0: renamed from tmp7362b | |
| Mar 21 15:45:55 bmhv7 systemd-networkd[946]: lxcc5c9eeefa2ef: Link UP | |
| Mar 21 15:45:55 bmhv7 systemd-networkd[946]: tmp7362b: Failed to wait for the interface to be initialized: No such device | |
| Mar 21 15:45:55 bmhv7 systemd-udevd[53895]: Using default interface naming scheme 'v245'. | |
| Mar 21 15:45:55 bmhv7 systemd-networkd[946]: lxcc5c9eeefa2ef: Gained carrier | |
| Mar 21 15:45:55 bmhv7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready | |
| Mar 21 15:45:55 bmhv7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): lxcc5c9eeefa2ef: link becomes ready | |
| Mar 21 15:45:55 bmhv7 networkd-dispatcher[987]: ERROR:Unknown interface index 35 seen even after reload | |
| Mar 21 15:45:55 bmhv7 networkd-dispatcher[987]: WARNING:Unknown index 35 seen, reloading interface list | |
| Mar 21 15:45:55 bmhv7 networkd-dispatcher[987]: ERROR:Unknown interface index 35 seen even after reload | |
| Mar 21 15:45:55 bmhv7 systemd[1]: Started libcontainer container 7362b5c17fd6a71070a9eba241c980f3f6a170b591d1e993d245cd2562af8a0e. | |
| Mar 21 15:45:56 bmhv7 systemd[1]: Started libcontainer container 660f211d046a9cc1866bb33c409c3208ae4fc29067bd7c2893ea2a72cdc5f673. | |
| Mar 21 15:45:56 bmhv7 systemd-networkd[946]: lxcc5c9eeefa2ef: Gained IPv6LL | |
| Mar 21 15:46:01 bmhv7 systemd[1]: Started libcontainer container bc548033ca24f73d7f3178c5d054de083f6f64838c1878016b2a68021b9f6e84. | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 6 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 9 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 10 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 22 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 26 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 31 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 32 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 39 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 40 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 41 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 44 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 45 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 48 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 49 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 57 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 62 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 65 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 73 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 75 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 78 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 81 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 84 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 85 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 95 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 96 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 97 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 101 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 102 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 110 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 112 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 118 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 123 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 127 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 128 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 129 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 131 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 132 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 140 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 145 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 149 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 150 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 151 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 153 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 154 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 155 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 156 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 158 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 159 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 162 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 163 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 167 primary OSD from 2 to 1 | |
| ... | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 192 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 202 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 210 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 212 primary OSD from 2 to 4 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 217 primary OSD from 2 to 4 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 220 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 221 primary OSD from 2 to 4 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 228 primary OSD from 2 to 4 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 230 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 232 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 235 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 253 primary OSD from 2 to 4 | |
| Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 256 primary OSD from 2 to 1 | |
| Mar 21 15:46:46 bmhv7 node[43861]: PG configuration successfully changed | |
| Mar 21 15:47:02 bmhv7 kernel: vitastor-osd[45920]: segfault at 559dad0fc6a8 ip 00007fd29a6808ad sp 00007ffd06ae11c0 error 4 in libvitastor_blk.so.0.8.6[7fd29a664000+2d000] | |
| Mar 21 15:47:02 bmhv7 kernel: Code: 0f 1f 44 00 00 31 d2 48 f7 b7 c8 02 00 00 48 8d 42 01 49 39 c2 74 66 48 8b 8f c0 02 00 00 48 89 c2 48 c1 e2 05 48 8d 54 11 e0 <48> 83 7a 08 00 75 4c 4c 8b 87 80 02 00 00 48 | |
| 8b 0a 48 8b 97 90 02 | |
| Mar 21 15:47:02 bmhv7 etcd[43768]: time="2023-03-21T15:47:02Z" level=warning msg="error reading websocket message: websocket: close 1006 (abnormal closure): unexpected EOF" | |
| Mar 21 15:47:02 bmhv7 systemd[1]: vitastor-osd@1.service: Main process exited, code=killed, status=11/SEGV | |
| Mar 21 15:47:02 bmhv7 systemd[1]: vitastor-osd@1.service: Failed with result 'signal'. | |
| Mar 21 15:47:12 bmhv7 systemd[1]: vitastor-osd@1.service: Scheduled restart job, restart counter is at 1. | |
| Mar 21 15:47:12 bmhv7 systemd[1]: Stopped Vitastor object storage daemon osd.1. | |
| Mar 21 15:47:12 bmhv7 systemd[1]: Starting Vitastor object storage daemon osd.1... | |
| Mar 21 15:47:12 bmhv7 systemd[1]: Started Vitastor object storage daemon osd.1. | |
| Mar 21 15:47:12 bmhv7 systemd[1]: vitastor-osd@1.service: Succeeded. | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 1 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 2 primary OSD from 3 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 4 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 5 primary OSD from 4 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 6 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 7 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 8 primary OSD from 3 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 9 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 10 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 11 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 12 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 13 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 14 primary OSD from 3 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 15 primary OSD from 4 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 16 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 17 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 18 primary OSD from 1 to 0 | |
| ... | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 243 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 244 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 245 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 246 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 247 primary OSD from 3 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 248 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 249 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 250 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 251 primary OSD from 3 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 252 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 253 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 254 primary OSD from 4 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 255 primary OSD from 1 to 0 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 1 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 2 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 3 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 7 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 10 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 11 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 14 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 16 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 17 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 19 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 24 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 25 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 26 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 31 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 35 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 40 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 41 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 42 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 44 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 46 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 47 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 48 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 49 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 55 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 57 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 59 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 60 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 62 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 64 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 73 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 79 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 80 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 83 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 85 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 87 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 88 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 89 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 90 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 94 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 96 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 99 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 110 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 113 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 116 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 120 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 121 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 123 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 125 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 127 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 130 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 136 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 140 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 141 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 144 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 145 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 146 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 150 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 155 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 159 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 160 primary OSD from 1 to 4 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 162 primary OSD from 1 to 3 | |
| Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 163 primary OSD from 1 to 3 | |
| ... | |
| Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 243 primary OSD from 4 to 1 | |
| Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 245 primary OSD from 4 to 1 | |
| Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 246 primary OSD from 3 to 1 | |
| Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 249 primary OSD from 4 to 1 | |
| Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 250 primary OSD from 4 to 1 | |
| Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 255 primary OSD from 4 to 1 | |
| Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 256 primary OSD from 3 to 1 | |
| Mar 21 15:47:23 bmhv7 node[43861]: PG configuration successfully changed | |
| Mar 21 15:47:28 bmhv7 systemd[1]: Started libcontainer container b40aab9cc637dc0409f532b7557546278bfef400958bb7416e85aea28ca069f3. | |
| Mar 21 15:47:43 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Link DOWN | |
| Mar 21 15:47:43 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Lost carrier | |
| Mar 21 15:47:44 bmhv7 networkd-dispatcher[803668]: Interface "lxc2de1f96c479d" not found. | |
| Mar 21 15:47:44 bmhv7 systemd[1]: networkd-dispatcher.service: Got notification message from PID 803668, but reception only permitted for main PID 987 | |
| Mar 21 15:47:44 bmhv7 networkd-dispatcher[987]: ERROR:Failed to get interface "lxc2de1f96c479d" status: Command '['/usr/bin/networkctl', 'status', '--no-pager', '--no-legend', '--', 'lxc2de1f96c479d']' returne | |
| d non-zero exit status 1. | |
| Mar 21 15:47:44 bmhv7 systemd[1]: run-netns-cni\x2d0c18c1b2\x2de692\x2d3636\x2df712\x2dbfcd32ab2781.mount: Succeeded. | |
| Mar 21 15:47:44 bmhv7 systemd[18106]: run-netns-cni\x2d0c18c1b2\x2de692\x2d3636\x2df712\x2dbfcd32ab2781.mount: Succeeded. | |
| Mar 21 15:47:44 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:44 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:44 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:44 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:44 bmhv7 systemd[1]: Removed slice libcontainer container kubepods-burstable-pod9ac5c3e2_566f_48fd_9332_9c845c7e25d1.slice. | |
| Mar 21 15:47:45 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:45 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:45 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:45 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:46 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:46 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:46 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:46 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:47 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:47 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:47 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:47 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:48 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0 | |
| Mar 21 15:47:48 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:47:48 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0 | |
| Mar 21 15:47:48 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06 | |
| Mar 21 15:48:09 bmhv7 etcd[43768]: raft2023/03/21 15:48:09 INFO: 607474ecd5112191 [logterm: 21, index: 2714, vote: 0] ignored MsgVote from cfefa00537b3d384 [logterm: 21, index: 2714] at term 21: lease is not e | |
| xpired (remaining ticks: 10) | |
| Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: 607474ecd5112191 [term: 21] received a MsgVote message with higher term from af5ac16031361c6a [term: 22] | |
| Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: 607474ecd5112191 became follower at term 22 | |
| Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: 607474ecd5112191 [logterm: 21, index: 2715, vote: 0] cast MsgVote for af5ac16031361c6a [logterm: 21, index: 2715] at term 22 | |
| Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: raft.node: 607474ecd5112191 lost leader cb7af46e9829ddc8 at term 22 | |
| Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: raft.node: 607474ecd5112191 elected leader af5ac16031361c6a at term 22 | |
| Mar 21 15:48:10 bmhv7 node[43861]: failed to query etcd: timeout | |
| Mar 21 15:48:10 bmhv7 node[43861]: events.js:174 | |
| Mar 21 15:48:10 bmhv7 node[43861]: throw er; // Unhandled 'error' event | |
| Mar 21 15:48:10 bmhv7 node[43861]: ^ | |
| Mar 21 15:48:10 bmhv7 node[43861]: Error: socket hang up | |
| Message from syslogd@bmhv7 at Mar 21 15:48:29 ... | |
| kernel:[ 3446.142595] NMI watchdog: Watchdog detected hard LOCKUP on cpu 14 | |
| Message from syslogd@bmhv7 at Mar 21 15:48:29 ... | |
| kernel:[ 3454.797679] NMI watchdog: Watchdog detected hard LOCKUP on cpu 15 | |
| Message from syslogd@bmhv7 at Mar 21 15:48:29 ... | |
| kernel:[ 3457.801254] NMI watchdog: Watchdog detected hard LOCKUP on cpu 31 | |
| Message from syslogd@bmhv7 at Mar 21 15:48:29 ... | |
| kernel:[ 3460.575290] watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [kworker/17:2:49099] | |
| Mar 21 15:48:10 bmhv7 node[43861]: at createHangUpError (_http_client.js:332:15) | |
| Mar 21 15:48:10 bmhv7 node[43861]: at Socket.socketCloseListener (_http_client.js:373:25) | |
| Mar 21 15:48:10 bmhv7 node[43861]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:10 bmhv7 node[43861]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:10 bmhv7 node[43861]: Emitted 'error' event at: | |
| Mar 21 15:48:10 bmhv7 node[43861]: at Socket.socketCloseListener (_http_client.js:373:11) | |
| Mar 21 15:48:10 bmhv7 node[43861]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:10 bmhv7 node[43861]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:10 bmhv7 etcd[43768]: time="2023-03-21T15:48:10Z" level=warning msg="error reading websocket message: websocket: close 1006 (abnormal closure): unexpected EOF" | |
| Mar 21 15:48:10 bmhv7 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE | |
| Mar 21 15:48:10 bmhv7 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'. | |
| Mar 21 15:48:14 bmhv7 etcd[43768]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader) | |
| Mar 21 15:48:14 bmhv7 etcd[43768]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.171:46512->10.10.0.164:2380: i/o timeout) | |
| Mar 21 15:48:14 bmhv7 etcd[43768]: peer cfefa00537b3d384 became inactive (message send to peer failed) | |
| Mar 21 15:48:14 bmhv7 etcd[43768]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader) | |
| Mar 21 15:48:20 bmhv7 systemd[1]: vitastor-mon.service: Scheduled restart job, restart counter is at 1. | |
| Mar 21 15:48:20 bmhv7 systemd[1]: Stopped Vitastor monitor. | |
| Mar 21 15:48:20 bmhv7 systemd[1]: Started Vitastor monitor. | |
| Mar 21 15:48:21 bmhv7 node[1125603]: Waiting to become master | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 is starting a new election at term 22 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 became candidate at term 23 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 received MsgVoteResp from 607474ecd5112191 at term 23 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 [logterm: 22, index: 2732] sent MsgVote request to 46d9e5656b97850f at term 23 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 [logterm: 22, index: 2732] sent MsgVote request to af5ac16031361c6a at term 23 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 [logterm: 22, index: 2732] sent MsgVote request to cb7af46e9829ddc8 at term 23 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 [logterm: 22, index: 2732] sent MsgVote request to cfefa00537b3d384 at term 23 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: raft.node: 607474ecd5112191 lost leader af5ac16031361c6a at term 23 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 received MsgVoteResp from cb7af46e9829ddc8 at term 23 | |
| Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 has received 2 MsgVoteResp votes and 0 vote rejections | |
| Mar 21 15:48:25 bmhv7 node[1125603]: failed to query etcd: timeout | |
| Mar 21 15:48:25 bmhv7 node[1125603]: events.js:174 | |
| Mar 21 15:48:25 bmhv7 node[1125603]: throw er; // Unhandled 'error' event | |
| Mar 21 15:48:25 bmhv7 node[1125603]: ^ | |
| Mar 21 15:48:25 bmhv7 node[1125603]: Error: socket hang up | |
| Mar 21 15:48:25 bmhv7 node[1125603]: at createHangUpError (_http_client.js:332:15) | |
| Mar 21 15:48:25 bmhv7 node[1125603]: at Socket.socketCloseListener (_http_client.js:373:25) | |
| Mar 21 15:48:25 bmhv7 node[1125603]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:25 bmhv7 node[1125603]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:25 bmhv7 node[1125603]: Emitted 'error' event at: | |
| Mar 21 15:48:25 bmhv7 node[1125603]: at Socket.socketCloseListener (_http_client.js:373:11) | |
| Mar 21 15:48:25 bmhv7 node[1125603]: at Socket.emit (events.js:203:15) | |
| Mar 21 15:48:25 bmhv7 node[1125603]: at TCP._handle.close (net.js:607:12) | |
| Mar 21 15:48:25 bmhv7 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE | |
| Mar 21 15:48:25 bmhv7 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'. | |
| Mar 21 15:48:25 bmhv7 etcd[43768]: raft2023/03/21 15:48:25 INFO: 607474ecd5112191 [logterm: 22, index: 2732, vote: 607474ecd5112191] rejected MsgVote from af5ac16031361c6a [logterm: 22, index: 2732] at term 23 | |
| Mar 21 15:48:25 bmhv7 etcd[43768]: raft2023/03/21 15:48:25 INFO: 607474ecd5112191 [term: 23] received a MsgVote message with higher term from cb7af46e9829ddc8 [term: 24] | |
| Mar 21 15:48:25 bmhv7 etcd[43768]: raft2023/03/21 15:48:25 INFO: 607474ecd5112191 became follower at term 24 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Mar 21 15:42:52 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:42:57 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:02 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:07 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:12 bmhv9 etcd[1775114]: store.index: compact 1022 | |
| Mar 21 15:43:12 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:12 bmhv9 etcd[1775114]: finished scheduled compaction at 1022 (took 394.570871ms) | |
| Mar 21 15:43:17 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:22 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:27 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:32 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:37 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:42 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:42 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:42 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:44 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:44 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:45 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:45 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:46 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:46 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:47 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0 | |
| Mar 21 15:43:47 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06 | |
| Mar 21 15:43:47 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:52 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:43:57 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:02 bmhv9 systemd-logind[1052]: New session 25 of user ubuntu. | |
| Mar 21 15:44:02 bmhv9 systemd[1]: Started Session 25 of user ubuntu. | |
| Mar 21 15:44:02 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:07 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:12 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:17 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:22 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:27 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:29 bmhv9 systemd[1]: Started libcontainer container 7a789cf3d4a7555ff33c58ae0a8ef52fe660d8e5f77b32950e9be086d8f17ab7. | |
| Mar 21 15:44:30 bmhv9 systemd[1]: Started libcontainer container 65817557137d7633369240e89e947d42aab40395d74c5c688de4f4534a4a8e79. | |
| Mar 21 15:44:30 bmhv9 systemd[1]: Started libcontainer container a1a121ec70a2485f960acf4c8d34ebf1c2665ca6c86452b2e25fe9fba82b5764. | |
| Mar 21 15:44:31 bmhv9 systemd[1]: Started libcontainer container d3b79c6588f80b976ed68b1fc21424ee1459e0273678034e5ee5bf2298b35680. | |
| Mar 21 15:44:32 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:37 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:42 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:44 bmhv9 systemd[1]: Started libcontainer container 94708abb99d4d1299dee031d3134f81130fcfb602bb18d2c5db839da69cd0eac. | |
| Mar 21 15:44:44 bmhv9 systemd[1]: Started libcontainer container 4586913ccc859a4f8ae495f6e23fcb1594e83d9923d3ab2ef33a7ac41c6fd723. | |
| Mar 21 15:44:45 bmhv9 systemd[1]: Started libcontainer container ad8efd7bdc27bf272a06f5c11cce2bbbef8b1abb373d1481e8aa07c4ee297819. | |
| Mar 21 15:44:47 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:44:52 bmhv9 node[1780565]: Waiting to become master | |
| Message from syslogd@bmhv9 at Mar 21 15:47:00 ... | |
| kernel:[73646.799077] NMI watchdog: Watchdog detected hard LOCKUP on cpu 26 | |
| Message from syslogd@bmhv9 at Mar 21 15:47:00 ... | |
| kernel:[73647.598937] NMI watchdog: Watchdog detected hard LOCKUP on cpu 27 | |
| Message from syslogd@bmhv9 at Mar 21 15:47:00 ... | |
| kernel:[73659.545002] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kworker/10:3:1797192] | |
| Message from syslogd@bmhv9 at Mar 21 15:47:00 ... | |
| kernel:[73659.557003] watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [migration/14:103] | |
| Message from syslogd@bmhv9 at Mar 21 15:47:00 ... | |
| kernel:[73659.561002] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [runc:1799309] | |
| Mar 21 15:44:57 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:02 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:07 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:12 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:16 bmhv9 systemd[1]: Started libcontainer container 61aac3dabe59401178cb9e1b7df09f8a46e14ef055e4fd10a720cb0a269e7da3. | |
| Mar 21 15:45:17 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:22 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:27 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:32 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:38 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:43 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:48 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:53 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:45:58 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:03 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:08 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:13 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:18 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:18 bmhv9 systemd[1]: Started libcontainer container a49a9db20d4aed7f8e06282534dd5c4ea6fec9fd5fc762e2a22faa23dc3e7455. | |
| Mar 21 15:46:23 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:24 bmhv9 systemd[1]: Started libcontainer container 9c87575152ce6f679ea02b2cc6a2331e7be100d93995d35889b40c1783a71a3c. | |
| Mar 21 15:46:28 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:33 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:38 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:43 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:48 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:53 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:46:58 bmhv9 node[1780565]: Waiting to become master | |
| Mar 21 15:47:00 bmhv9 kernel: NMI watchdog: Watchdog detected hard LOCKUP on cpu 26 | |
| Mar 21 15:47:00 bmhv9 kernel: Modules linked in: msr xfs xt_recent vhost_net vhost vhost_iotlb tap xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 ip_set xt_CT cls_bpf algif_hash af_alg sch_ingress vxlan ip6_udp_tunne | |
| l udp_tunnel veth xfrm_user xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6table_raw iptable_raw tls nf_conntrack_netlink xt_statistic xt_nat xt_addrtype ipt_REJECT nf_reject_ipv4 xt_tcpudp nf_tables nfnetlink ip_ | |
| vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6table_filter ip6table_nat ip6_tables iptable_mangle xt_MASQUERADE xt_conntrack xt_comment iptable_filter xt_mark iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 | |
| nf_defrag_ipv4 libcrc32c bpfilter ib_core aufs binfmt_misc ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd ast kvm_amd drm_vram_helper kvm joydev input_leds crct10dif_pclmul drm_ttm_helper | |
| nls_iso8859_1 ghash_clmulni_intel ttm aesni_intel drm_kms_helper crypto_simd cryptd cls_u32 cec rc_core rapl sch_prio fb_sys_fops syscopyarea sysfillrect sysimgblt ccp k10temp | |
| Mar 21 15:47:00 bmhv9 kernel: acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mac_hid tcp_bbr sch_fq_codel br_netfilter bridge stp llc overlay ramoops reed_solomon pstore_blk pstore_zone efi_pstore drm ip_tabl | |
| es x_tables autofs4 hid_generic usbhid hid ixgbe igb nvme xfrm_algo i2c_algo_bit ahci nvme_core mdio crc32_pclmul xhci_pci libahci i2c_piix4 xhci_pci_renesas dca | |
| Mar 21 15:47:00 bmhv9 kernel: CPU: 26 PID: 550435 Comm: virt-handler Not tainted 5.15.0-67-generic #74~20.04.1-Ubuntu | |
| Mar 21 15:47:00 bmhv9 kernel: Hardware name: Supermicro Super Server/H11SSL-i, BIOS 1.3 06/25/2019 | |
| Mar 21 15:47:00 bmhv9 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x79/0x230 | |
| Mar 21 15:47:00 bmhv9 kernel: Code: 7c f0 0f ba 2b 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 03 30 e4 09 d0 a9 00 01 ff ff 75 56 85 c0 74 0e 8b 03 84 c0 74 08 f3 90 <8b> 03 84 c0 75 f8 b8 01 00 00 00 66 89 03 48 | |
| 83 c4 08 5b 41 5c 41 | |
| Mar 21 15:47:00 bmhv9 kernel: RSP: 0018:ffffb2b046950c20 EFLAGS: 00000002 | |
| Mar 21 15:47:00 bmhv9 kernel: RAX: 0000000000700101 RBX: ffff94d8ccefc7d8 RCX: ffff94dee18e7d28 | |
| Mar 21 15:47:00 bmhv9 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff94d8ccefc7d8 | |
| Mar 21 15:47:00 bmhv9 kernel: RBP: ffffb2b046950c48 R08: 0000000000000002 R09: 0000000000000064 | |
| Mar 21 15:47:00 bmhv9 kernel: R10: ffff94dee1c00600 R11: 0000000000000002 R12: ffffb2b046950cb0 | |
| Mar 21 15:47:00 bmhv9 kernel: R13: ffff94d8ccefc7d8 R14: ffff94ef7bc48710 R15: 0000000000000001 | |
| Mar 21 15:47:00 bmhv9 kernel: FS: 00007f8b78ff9700(0000) GS:ffff94eebfd80000(0000) knlGS:0000000000000000 | |
| Mar 21 15:47:00 bmhv9 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 | |
| Mar 21 15:47:00 bmhv9 kernel: CR2: 00007fed55bcc620 CR3: 0000001048fda000 CR4: 00000000003506e0 | |
| Mar 21 15:47:00 bmhv9 kernel: Call Trace: | |
| Mar 21 15:47:00 bmhv9 kernel: <IRQ> | |
| Mar 21 15:47:00 bmhv9 kernel: _raw_spin_lock+0x22/0x30 | |
| Mar 21 15:47:00 bmhv9 kernel: take_dentry_name_snapshot+0x20/0x80 | |
| Mar 21 15:47:00 bmhv9 kernel: __fsnotify_parent+0x1d6/0x310 | |
| Mar 21 15:47:00 bmhv9 kernel: ? native_smp_send_reschedule+0x2a/0x50 | |
| Mar 21 15:47:00 bmhv9 kernel: ? mempool_free_slab+0x17/0x20 | |
| Mar 21 15:47:00 bmhv9 kernel: __io_complete_rw_common+0x132/0x180 | |
| Mar 21 15:47:00 bmhv9 kernel: ? __io_complete_rw_common+0x132/0x180 | |
| Mar 21 15:47:00 bmhv9 kernel: io_complete_rw+0x17/0x50 | |
| Mar 21 15:47:00 bmhv9 kernel: blkdev_bio_end_io+0x8c/0x110 | |
| Mar 21 15:47:00 bmhv9 kernel: bio_endio+0x106/0x170 | |
| Mar 21 15:47:00 bmhv9 kernel: blk_update_request+0x249/0x4d0 | |
| Mar 21 15:47:00 bmhv9 kernel: blk_mq_end_request+0x21/0x140 | |
| Mar 21 15:47:00 bmhv9 kernel: nvme_complete_rq+0x80/0x260 [nvme_core] | |
| Mar 21 15:47:00 bmhv9 kernel: nvme_pci_complete_rq+0x82/0xa0 [nvme] | |
| Mar 21 15:47:00 bmhv9 kernel: nvme_irq+0x15f/0x250 [nvme] | |
| Mar 21 15:47:00 bmhv9 kernel: __handle_irq_event_percpu+0x42/0x170 | |
| Mar 21 15:47:00 bmhv9 kernel: handle_irq_event_percpu+0x33/0x80 | |
| Mar 21 15:47:00 bmhv9 kernel: handle_irq_event+0x3b/0x60 | |
| Mar 21 15:47:00 bmhv9 kernel: handle_edge_irq+0x93/0x1c0 | |
| Mar 21 15:47:00 bmhv9 kernel: __common_interrupt+0x52/0xe0 | |
| Mar 21 15:47:00 bmhv9 kernel: common_interrupt+0x89/0xa0 | |
| Mar 21 15:47:00 bmhv9 kernel: </IRQ> | |
| Mar 21 15:47:00 bmhv9 kernel: <TASK> | |
| Mar 21 15:47:00 bmhv9 kernel: asm_common_interrupt+0x27/0x40 | |
| Mar 21 15:47:00 bmhv9 kernel: RIP: 0010:_raw_spin_lock+0x10/0x30 | |
| Mar 21 15:47:00 bmhv9 kernel: Code: 02 00 00 e8 f2 d0 34 ff 5d e9 0c bc 28 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 e9 e9 bb 28 00 55 89 c6 48 89 e5 e8 2e | |
| 16 3c ff 66 90 5d e9 | |
| Mar 21 15:47:00 bmhv9 kernel: RSP: 0018:ffffb2b066a3fb58 EFLAGS: 00000246 | |
| Mar 21 15:47:00 bmhv9 kernel: RAX: 0000000000000000 RBX: ffff94d8ccefc788 RCX: 0000000000000009 | |
| Mar 21 15:47:00 bmhv9 kernel: RDX: 0000000000000001 RSI: ffffb2b066a3fc90 RDI: ffff94d8ccefc7d8 | |
| Mar 21 15:47:00 bmhv9 kernel: RBP: ffffb2b066a3fb98 R08: 0000000000000000 R09: 61c8864680b583eb | |
| Mar 21 15:47:00 bmhv9 kernel: R10: 8080808080807fff R11: 8080808080808080 R12: ffff94d8ccefc7d8 | |
| Mar 21 15:47:00 bmhv9 kernel: R13: ffff94dee1c00600 R14: 0000000014ce31d5 R15: ffffb2b066a3fdd8 | |
| Mar 21 15:47:00 bmhv9 kernel: ? __d_lookup+0x64/0x150 | |
| Mar 21 15:47:00 bmhv9 kernel: lookup_fast+0xbb/0x160 | |
| Mar 21 15:47:00 bmhv9 kernel: walk_component+0x43/0x1c0 | |
| Mar 21 15:47:00 bmhv9 kernel: ? path_init+0x2b2/0x390 | |
| Mar 21 15:47:00 bmhv9 kernel: path_lookupat.isra.0+0x6e/0x150 | |
| Mar 21 15:47:00 bmhv9 kernel: filename_lookup+0xcf/0x1a0 | |
| Mar 21 15:47:00 bmhv9 kernel: ? path_get+0x27/0x30 | |
| Mar 21 15:47:00 bmhv9 kernel: user_path_at_empty+0x3f/0x60 | |
| Mar 21 15:47:00 bmhv9 kernel: vfs_statx+0x73/0x110 | |
| Mar 21 15:47:00 bmhv9 kernel: __do_sys_newfstatat+0x36/0x70 | |
| Mar 21 15:47:00 bmhv9 kernel: ? syscall_exit_to_user_mode+0x27/0x50 | |
| Mar 21 15:47:00 bmhv9 kernel: ? syscall_trace_enter.isra.0+0x14e/0x1c0 | |
| Mar 21 15:47:00 bmhv9 kernel: __x64_sys_newfstatat+0x1e/0x30 | |
| Mar 21 15:47:00 bmhv9 kernel: do_syscall_64+0x5c/0xc0 | |
| Mar 21 15:47:00 bmhv9 kernel: ? do_syscall_64+0x69/0xc0 | |
| Mar 21 15:47:00 bmhv9 kernel: ? do_syscall_64+0x69/0xc0 | |
| Mar 21 15:47:00 bmhv9 kernel: ? irqentry_exit_to_user_mode+0x9/0x20 | |
| Mar 21 15:47:00 bmhv9 kernel: ? irqentry_exit+0x1d/0x30 | |
| Mar 21 15:47:00 bmhv9 kernel: ? common_interrupt+0x55/0xa0 | |
| Mar 21 15:47:00 bmhv9 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb | |
| Mar 21 15:47:00 bmhv9 kernel: RIP: 0033:0x47ff2a | |
| Mar 21 15:47:00 bmhv9 kernel: Code: e8 fb b6 fe ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 4c 8b 54 24 28 4c 8b 44 24 30 4c 8b 4c 24 38 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 40 ff ff | |
| ff ff 48 c7 44 24 48 | |
| Mar 21 15:47:00 bmhv9 kernel: RSP: 002b:000000c00298db10 EFLAGS: 00000202 ORIG_RAX: 0000000000000106 | |
| Mar 21 15:47:00 bmhv9 kernel: RAX: ffffffffffffffda RBX: 000000c000067800 RCX: 000000000047ff2a | |
| Mar 21 15:47:00 bmhv9 kernel: RDX: 000000c003c209f8 RSI: 000000c003c192a0 RDI: ffffffffffffff9c | |
| Mar 21 15:47:00 bmhv9 kernel: RBP: 000000c00298dba0 R08: 0000000000000000 R09: 0000000000000000 | |
| Mar 21 15:47:00 bmhv9 kernel: R10: 0000000000000100 R11: 0000000000000202 R12: 000000c003c192a0 | |
| Mar 21 15:47:00 bmhv9 kernel: R13: 0000000000000001 R14: 000000c001df11e0 R15: ffffffffffffffff | |
| Mar 21 15:47:00 bmhv9 kernel: </TASK> | |
| Mar 21 15:47:00 bmhv9 kernel: NMI watchdog: Watchdog detected hard LOCKUP on cpu 27 | |
| Mar 21 15:47:00 bmhv9 kernel: Modules linked in: msr xfs xt_recent vhost_net vhost vhost_iotlb tap xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 ip_set xt_CT cls_bpf algif_hash af_alg sch_ingress vxlan ip6_udp_tunne | |
| l udp_tunnel veth xfrm_user xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6table_raw iptable_raw tls nf_conntrack_netlink xt_statistic xt_nat xt_addrtype ipt_REJECT nf_reject_ipv4 xt_tcpudp nf_tables nfnetlink ip_ | |
| vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6table_filter ip6table_nat ip6_tables iptable_mangle xt_MASQUERADE xt_conntrack xt_comment iptable_filter xt_mark iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 | |
| nf_defrag_ipv4 libcrc32c bpfilter ib_core aufs binfmt_misc ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd ast kvm_amd drm_vram_helper kvm joydev input_leds crct10dif_pclmul drm_ttm_helper | |
| nls_iso8859_1 ghash_clmulni_intel ttm aesni_intel drm_kms_helper crypto_simd cryptd cls_u32 cec rc_core rapl sch_prio fb_sys_fops syscopyarea sysfillrect sysimgblt ccp k10temp | |
| Mar 21 15:47:00 bmhv9 kernel: acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mac_hid tcp_bbr sch_fq_codel br_netfilter bridge stp llc overlay ramoops reed_solomon pstore_blk pstore_zone efi_pstore drm ip_tabl | |
| es x_tables autofs4 hid_generic usbhid hid ixgbe igb nvme xfrm_algo i2c_algo_bit ahci nvme_core mdio crc32_pclmul xhci_pci libahci i2c_piix4 xhci_pci_renesas dca | |
| Mar 21 15:47:00 bmhv9 kernel: CPU: 27 PID: 0 Comm: swapper/27 Not tainted 5.15.0-67-generic #74~20.04.1-Ubuntu | |
| Mar 21 15:47:00 bmhv9 kernel: Hardware name: Supermicro Super Server/H11SSL-i, BIOS 1.3 06/25/2019 | |
| Mar 21 15:47:00 bmhv9 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x159/0x230 | |
| Mar 21 15:47:00 bmhv9 kernel: Code: 5d 41 5e 5d e9 78 a4 ec 00 83 c6 01 c1 e2 10 c1 e6 12 09 f2 89 d0 c1 e8 10 66 87 43 02 89 c1 c1 e1 10 75 53 31 c9 eb 02 f3 90 <8b> 03 66 85 c0 75 f7 89 c6 66 31 f6 39 f2 0f | |
| 84 93 00 00 00 c6 03 | |
| Mar 21 15:47:00 bmhv9 kernel: RSP: 0018:ffffb2b046984c20 EFLAGS: 00000002 | |
| Mar 21 15:47:00 bmhv9 kernel: RAX: 0000000000700101 RBX: ffff94d8ccefc7d8 RCX: 0000000000000000 | |
| Mar 21 15:47:00 bmhv9 kernel: RDX: 0000000000700000 RSI: 0000000000700000 RDI: ffff94d8ccefc7d8 | |
| Mar 21 15:47:00 bmhv9 kernel: RBP: ffffb2b046984c48 R08: 0000000000000002 R09: 0000000000000064 | |
| Mar 21 15:47:00 bmhv9 kernel: R10: ffff94dee1c00600 R11: 0000000000000002 R12: ffff94eebfdf1900 | |
| Mar 21 15:47:00 bmhv9 kernel: R13: ffff94d8ccefc7d8 R14: ffff94ef7bc48710 R15: 0000000000000001 | |
| Mar 21 15:47:00 bmhv9 kernel: FS: 0000000000000000(0000) GS:ffff94eebfdc0000(0000) knlGS:0000000000000000 | |
| Mar 21 15:47:00 bmhv9 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 | |
| Mar 21 15:47:00 bmhv9 kernel: CR2: 000055cd2403b0e0 CR3: 000000183196e000 CR4: 00000000003506e0 | |
| Mar 21 15:47:00 bmhv9 kernel: Call Trace: | |
| Mar 21 15:47:00 bmhv9 kernel: <IRQ> | |
| Mar 21 15:47:00 bmhv9 kernel: _raw_spin_lock+0x22/0x30 | |
| Mar 21 15:47:00 bmhv9 kernel: take_dentry_name_snapshot+0x20/0x80 | |
| Mar 21 15:47:00 bmhv9 kernel: __fsnotify_parent+0x1d6/0x310 | |
| Mar 21 15:47:00 bmhv9 kernel: ? trace_event_raw_event_block_rq_remap+0xb1/0x120 | |
| Mar 21 15:47:00 bmhv9 kernel: ? mempool_free_slab+0x17/0x20 | |
| Mar 21 15:47:00 bmhv9 kernel: __io_complete_rw_common+0x132/0x180 | |
| Mar 21 15:47:00 bmhv9 kernel: ? __io_complete_rw_common+0x132/0x180 | |
| Mar 21 15:47:00 bmhv9 kernel: io_complete_rw+0x17/0x50 | |
| Mar 21 15:47:00 bmhv9 kernel: blkdev_bio_end_io+0x8c/0x110 | |
| Mar 21 15:47:00 bmhv9 kernel: bio_endio+0x106/0x170 | |
| Mar 21 15:47:00 bmhv9 kernel: blk_update_request+0x249/0x4d0 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment