Skip to content

Instantly share code, notes, and snippets.

@Gabelbombe
Last active April 25, 2019 19:24
Show Gist options
  • Save Gabelbombe/6c3c7239673481a398ac10a91c5e018a to your computer and use it in GitHub Desktop.
Save Gabelbombe/6c3c7239673481a398ac10a91c5e018a to your computer and use it in GitHub Desktop.
Stale Nomad job that got stuck because of... yeah, thanks thobe....
ehimeken in ~/nomad-orchestrator/examples/root-example
○ → nomad job status
ID Type Priority Status Submit Date
fabio system 50 running 2019-04-25T11:15:22-07:00
ping_service service 50 running 2019-04-25T11:20:09-07:00
ehimeken in ~/nomad-orchestrator/examples/root-example
○ → nomad job status ping_service
ID = ping_service
Name = ping_service
Submit Date = 2019-04-25T11:20:09-07:00
Type = service
Priority = 50
Datacenters = public-services,private-services,content-connector,backoffice
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
ping_service_group 0 3 1 0 0 0
Allocations
ID Node ID Task Group Version Desired Status Created Modified
3de3b2b3 032217e1 ping_service_group 0 run pending 48m38s ago 33s ago <- bleh
4c02852d 23b79904 ping_service_group 0 run pending 48m38s ago 37s ago <- bleh
4f545b91 1601d6b1 ping_service_group 0 run pending 48m38s ago 31s ago <- bleh
ca50d521 2260997a ping_service_group 0 run running 48m38s ago 48m5s ago
ehimeken in ~/nomad-orchestrator/examples/root-example
○ → nomad alloc status 3de3b2b3
ID = 3de3b2b3
Eval ID = d0fe6487
Name = ping_service.ping_service_group[0]
Node ID = 032217e1
Job ID = ping_service
Job Version = 0
Client Status = pending
Client Description = <none>
Desired Status = run
Desired Description = <none>
Created = 48m58s ago
Modified = 13s ago
Task "ping_service_task" is "pending"
Task Resources
CPU Memory Disk IOPS Addresses
100 MHz 20 MiB 50 MiB 0 http: 10.xxx.xxx.xxx:30474
Task Events:
Started At = N/A
Finished At = N/A
Total Restarts = 43
Last Restart = 2019-04-25T19:08:54Z
Recent Events:
Time Type Description
2019-04-25T12:08:54-07:00 Restarting Task restarting in 31.064415928s
2019-04-25T12:08:54-07:00 Driver Failure failed to initialize task "ping_service_task" for alloc "3de3b2b3-7aab-5730-a51f-7ba72c9534b6": Failed to pull `thobe/ping_service:0.0.9`: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/cf/cfaa80d7f11f028474f755c007960a0b219c90e1edc45d94039a987c46d7ca32/data?verify=1556222304-dDhHjBAQAk8hLtWcIIj%2F2C%2B6XKM%3D: dial tcp 104.18.125.25:443: i/o timeout
2019-04-25T12:08:15-07:00 Driver Downloading image thobe/ping_service:0.0.9
2019-04-25T12:07:46-07:00 Restarting Task restarting in 29.023874526s
2019-04-25T12:07:46-07:00 Driver Failure failed to initialize task "ping_service_task" for alloc "3de3b2b3-7aab-5730-a51f-7ba72c9534b6": Failed to pull `thobe/ping_service:0.0.9`: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/cf/cfaa80d7f11f028474f755c007960a0b219c90e1edc45d94039a987c46d7ca32/data?verify=1556222236-08cgPJypjc3kH0NGg9z3tNtXHsM%3D: dial tcp 104.18.124.25:443: i/o timeout
2019-04-25T12:07:03-07:00 Driver Downloading image thobe/ping_service:0.0.9
2019-04-25T12:06:36-07:00 Restarting Task restarting in 26.828974441s
2019-04-25T12:06:36-07:00 Driver Failure failed to initialize task "ping_service_task" for alloc "3de3b2b3-7aab-5730-a51f-7ba72c9534b6": Failed to pull `thobe/ping_service:0.0.9`: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/cf/cfaa80d7f11f028474f755c007960a0b219c90e1edc45d94039a987c46d7ca32/data?verify=1556222166-VzGSfwFW4kBmBF1q%2F%2FFXIefLMXg%3D: dial tcp 104.18.125.25:443: i/o timeout
2019-04-25T12:05:57-07:00 Driver Downloading image thobe/ping_service:0.0.9
2019-04-25T12:05:32-07:00 Restarting Task restarting in 25.277064566s
=========
But .....
=========
ehimeken in ~/nomad-orchestrator/examples/root-example
○ → nomad alloc status ca50d521
ID = ca50d521
Eval ID = d0fe6487
Name = ping_service.ping_service_group[3]
Node ID = 2260997a
Job ID = ping_service
Job Version = 0
Client Status = running
Client Description = <none>
Desired Status = run
Desired Description = <none>
Created = 58m35s ago
Modified = 58m2s ago
Task "ping_service_task" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
0/100 MHz 5.6 MiB/20 MiB 50 MiB 0 http: 10.128.132.254:27623
Task Events:
Started At = 2019-04-25T18:20:24Z
Finished At = N/A
Total Restarts = 0
Last Restart = N/A
Recent Events:
Time Type Description
2019-04-25T11:20:24-07:00 Started Task started by client
2019-04-25T11:20:09-07:00 Driver Downloading image thobe/ping_service:0.0.9
2019-04-25T11:20:09-07:00 Task Setup Building Task Directory
2019-04-25T11:20:09-07:00 Received Task received by client
¯\_(ツ)_/¯
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment