Created
April 11, 2023 21:18
-
-
Save bnaul/f40c47e42f4964d1adaaa9872e0fc539 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/usr/local/lib/python3.10/site-packages/distributed/cli/dask_worker.py:264: FutureWarning: dask-worker is deprecated and will be removed in a future release; use `dask worker` instead | |
warnings.warn( | |
2023-04-09 15:43:19,990 - distributed.nanny - INFO - Start Nanny at: 'tcp://10.76.9.27:43473' | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - Start worker at: tcp://10.76.9.27:46429 | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - Listening to: tcp://10.76.9.27:46429 | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - Worker name: brett-1f0ecf6d-daskworkers-large-865bf95dbc-5vqnd | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - dashboard at: 10.76.9.27:8787 | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - Waiting to connect to: tcp://brett-1f0ecf6d-daskscheduler:8786 | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - ------------------------------------------------- | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - Threads: 4 | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - Memory: 89.41 GiB | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-3id3r54g | |
2023-04-09 15:43:20,788 - distributed.worker - INFO - ------------------------------------------------- | |
2023-04-09 15:43:21,005 - distributed.worker - INFO - Registered to: tcp://brett-1f0ecf6d-daskscheduler:8786 | |
2023-04-09 15:43:21,005 - distributed.worker - INFO - ------------------------------------------------- | |
2023-04-09 15:43:21,006 - distributed.core - INFO - Starting established connection to tcp://brett-1f0ecf6d-daskscheduler:8786 | |
[2023-04-09 15:43:26+0000] INFO - prefect.CloudTaskRunner | Task 'on_demand_auto_calibration-places/configs/south_central/2022_Q4/saturday.yaml': Starting task run... | |
[2023-04-09 15:43:38+0000] INFO - root | Loading on demand auto data... | |
[2023-04-09 15:43:47+0000] INFO - root | Loading trips | |
[2023-04-09 15:43:48+0000] INFO - root | Saving query result to temporary table temp_0330502b7ac44d6da5bddbab79b875c1 | |
[2023-04-09 15:43:49+0000] INFO - root | Running query: https://console.cloud.google.com/bigquery?project=model-159019&j=bq:US:9a48b9bf-7fc7-4c3c-9928-b0493a6fce44&page=queryresults | |
[2023-04-09 15:44:18+0000] INFO - root | Query was billed for 5.989GB (~$0.03) | |
[2023-04-09 15:44:20+0000] INFO - root | Computing hourly on-demand coefficients... | |
2023-04-09 15:45:45,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.47:44359 | |
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.47:44359 | |
2023-04-09 15:45:45,233 - distributed.core - INFO - Removing comms to tcp://10.76.9.45:39307 | |
2023-04-09 15:45:45,233 - distributed.core - INFO - Removing comms to tcp://10.76.9.43:38263 | |
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.45:39307 | |
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.43:38263 | |
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.44:38033 | |
2023-04-09 15:45:45,233 - distributed.core - INFO - Removing comms to tcp://10.76.9.44:38033 | |
2023-04-09 15:45:45,234 - distributed.core - INFO - Removing comms to tcp://10.76.9.46:43001 | |
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.46:43001 | |
[2023-04-09 15:46:08+0000] CRITICAL - root | Web-PDB: starting web-server on brett-1f0ecf6d-daskworkers-large-865bf95dbc-5vqnd:5555... | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.40:46305 | |
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.40:46305 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.56:39929 | |
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.56:39929 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:38839 | |
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:38839 | |
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.55:42775 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.55:42775 | |
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.52:38589 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.52:38589 | |
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.34:38143 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.34:38143 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.32:41163 | |
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.32:41163 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.37:42627 | |
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.37:42627 | |
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.36:43351 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.36:43351 | |
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.42:42791 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.42:42791 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.54:46409 | |
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.54:46409 | |
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.49:33475 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.49:33475 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.30:40417 | |
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.30:40417 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.39:44649 | |
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.39:44649 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.33:42703 | |
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.33:42703 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.38:40847 | |
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.38:40847 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.53:46049 | |
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.53:46049 | |
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.41:44511 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.41:44511 | |
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.48:45847 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.48:45847 | |
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.51:38981 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.51:38981 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:39323 | |
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:39323 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.31:34457 | |
2023-04-09 15:46:25,248 - distributed.core - INFO - Removing comms to tcp://10.76.9.31:34457 | |
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.35:33371 | |
2023-04-09 15:46:25,248 - distributed.core - INFO - Removing comms to tcp://10.76.9.35:33371 | |
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.60:41331 | |
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.60:41331 | |
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.64:39303 | |
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.64:39303 | |
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:35323 | |
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:35323 | |
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:33561 | |
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:33561 | |
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.50:35227 | |
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.50:35227 | |
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.57:38951 | |
2023-04-09 15:47:45,234 - distributed.core - INFO - Removing comms to tcp://10.76.9.57:38951 | |
2023-04-09 15:47:45,234 - distributed.core - INFO - Removing comms to tcp://10.76.9.59:34401 | |
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.59:34401 | |
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.62:37533 | |
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.62:37533 | |
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.61:43181 | |
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.61:43181 | |
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.58:38979 | |
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.58:38979 | |
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:41347 | |
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:41347 | |
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.63:35491 | |
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.63:35491 | |
<stdin>:1: UserWarning: `meta` is not specified, inferred from partial data. Please provide `meta` if the result is unexpected. | |
Before: .apply(func) | |
After: .apply(func, meta={'x': 'f8', 'y': 'f8'}) for dataframe result | |
or: .apply(func, meta=('x', 'f8')) for series result | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:34413 | |
2023-04-09 15:49:25,238 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:34413 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.75:37509 | |
2023-04-09 15:49:25,238 - distributed.core - INFO - Removing comms to tcp://10.76.9.75:37509 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.72:33713 | |
2023-04-09 15:49:25,238 - distributed.core - INFO - Removing comms to tcp://10.76.9.72:33713 | |
2023-04-09 15:49:25,238 - distributed.core - INFO - Removing comms to tcp://10.76.9.76:43395 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.76:43395 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.71:43447 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.71:43447 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.68:45359 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.68:45359 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:39177 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:39177 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.69:34013 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.69:34013 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.70:37279 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.70:37279 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.73:45365 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.73:45365 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:33075 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:33075 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.74:42141 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.74:42141 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.77:42095 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.77:42095 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.67:42745 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.67:42745 | |
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.65:36627 | |
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.65:36627 | |
<stdin>:1: UserWarning: `meta` is not specified, inferred from partial data. Please provide `meta` if the result is unexpected. | |
Before: .apply(func) | |
After: .apply(func, meta={'x': 'f8', 'y': 'f8'}) for dataframe result | |
or: .apply(func, meta=('x', 'f8')) for series result | |
[2023-04-09 15:49:49+0000] INFO - distributed.utils_perf | full garbage collection released 22.26 MiB from 510 reference cycles (threshold: 9.54 MiB) | |
2023-04-09 15:49:49,530 - distributed.utils_perf - INFO - full garbage collection released 22.26 MiB from 510 reference cycles (threshold: 9.54 MiB) | |
[2023-04-09 15:50:09+0000] INFO - distributed.utils_perf | full garbage collection released 51.62 MiB from 570 reference cycles (threshold: 9.54 MiB) | |
2023-04-09 15:50:09,226 - distributed.utils_perf - INFO - full garbage collection released 51.62 MiB from 570 reference cycles (threshold: 9.54 MiB) | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.82:34499 | |
2023-04-09 15:50:35,240 - distributed.core - INFO - Removing comms to tcp://10.76.9.82:34499 | |
2023-04-09 15:50:35,240 - distributed.core - INFO - Removing comms to tcp://10.76.9.89:38245 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.89:38245 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:35487 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:35487 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.78:39053 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.78:39053 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:38073 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:38073 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.86:34919 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.86:34919 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.80:34689 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.80:34689 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.81:43023 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.81:43023 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.88:38229 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.88:38229 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.84:32885 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.84:32885 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.85:40007 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.85:40007 | |
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.87:38171 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.87:38171 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.83:42677 | |
2023-04-09 15:50:35,242 - distributed.core - INFO - Removing comms to tcp://10.76.9.83:42677 | |
2023-04-09 15:50:35,242 - distributed.core - INFO - Removing comms to tcp://10.76.9.79:37789 | |
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.79:37789 | |
/usr/local/lib/python3.10/site-packages/dask/dataframe/core.py:6984: FutureWarning: Meta is not valid, `map_partitions` and `map_overlap` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns. In the future the meta you passed will not work. | |
warnings.warn( | |
[2023-04-09 15:50:48+0000] INFO - distributed.utils_perf | full garbage collection released 21.95 MiB from 433 reference cycles (threshold: 9.54 MiB) | |
2023-04-09 15:50:48,801 - distributed.utils_perf - INFO - full garbage collection released 21.95 MiB from 433 reference cycles (threshold: 9.54 MiB) | |
2023-04-09 15:52:19,721 - distributed.comm.tcp - INFO - Connection from tcp://10.76.9.108:50670 closed before handshake completed | |
[2023-04-09 15:52:19+0000] INFO - distributed.comm.tcp | Connection from tcp://10.76.9.108:50670 closed before handshake completed | |
2023-04-09 15:52:35,506 - distributed.core - INFO - Removing comms to tcp://10.76.117.70:40427 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.70:40427 | |
2023-04-09 15:52:35,508 - distributed.core - INFO - Removing comms to tcp://10.76.9.118:34731 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.118:34731 | |
2023-04-09 15:52:35,541 - distributed.core - INFO - Removing comms to tcp://10.76.9.123:41057 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.123:41057 | |
2023-04-09 15:52:35,544 - distributed.core - INFO - Removing comms to tcp://10.76.117.62:35147 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.62:35147 | |
2023-04-09 15:52:35,612 - distributed.core - INFO - Removing comms to tcp://10.76.9.116:35619 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.116:35619 | |
2023-04-09 15:52:35,656 - distributed.core - INFO - Removing comms to tcp://10.76.9.90:43623 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.90:43623 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.120:45829 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.66:43527 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.121:34003 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.106:33747 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.68:37137 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.124:34529 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.71:36633 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.104:35301 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.64:37585 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.109:41427 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.117:35185 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.103:38401 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.72:37413 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.110:44621 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.122:44313 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.67:33027 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.111:35995 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.91:45783 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.119:33227 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:34593 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.97:39951 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.65:46807 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.73:43233 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.95:40549 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.114:37299 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.98:46447 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:46773 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.63:36409 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.69:45159 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.102:37163 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.115:36053 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.107:42381 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.101:46881 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.112:40069 | |
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.100:37729 | |
2023-04-09 15:52:35,657 - distributed.core - INFO - Removing comms to tcp://10.76.9.120:45829 | |
2023-04-09 15:52:35,657 - distributed.core - INFO - Removing comms to tcp://10.76.117.66:43527 | |
2023-04-09 15:52:35,657 - distributed.core - INFO - Removing comms to tcp://10.76.9.121:34003 | |
2023-04-09 15:52:35,657 - distributed.core - INFO - Removing comms to tcp://10.76.9.106:33747 | |
2023-04-09 15:52:35,658 - distributed.core - INFO - Removing comms to tcp://10.76.117.68:37137 | |
2023-04-09 15:52:35,658 - distributed.core - INFO - Removing comms to tcp://10.76.9.124:34529 | |
2023-04-09 15:52:35,658 - distributed.core - INFO - Removing comms to tcp://10.76.117.71:36633 | |
2023-04-09 15:52:35,658 - distributed.core - INFO - Removing comms to tcp://10.76.9.104:35301 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.117.64:37585 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.109:41427 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.117:35185 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.103:38401 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.117.72:37413 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.110:44621 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.122:44313 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.117.67:33027 | |
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.111:35995 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.91:45783 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.119:33227 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:34593 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.97:39951 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.117.65:46807 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.117.73:43233 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.95:40549 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.114:37299 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.98:46447 | |
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:46773 | |
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.117.63:36409 | |
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.117.69:45159 | |
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.102:37163 | |
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.115:36053 | |
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.107:42381 | |
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.101:46881 | |
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.112:40069 | |
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.100:37729 | |
2023-04-09 15:53:25,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.96:40683 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.96:40683 | |
2023-04-09 15:53:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.93:46657 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.93:46657 | |
2023-04-09 15:53:25,248 - distributed.core - INFO - Removing comms to tcp://10.76.9.99:38421 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.99:38421 | |
2023-04-09 15:53:25,249 - distributed.core - INFO - Removing comms to tcp://10.76.117.76:44435 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.76:44435 | |
2023-04-09 15:53:25,249 - distributed.core - INFO - Removing comms to tcp://10.76.9.105:33501 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.105:33501 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.108:43421 | |
2023-04-09 15:53:25,249 - distributed.core - INFO - Removing comms to tcp://10.76.9.108:43421 | |
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.9.113:43369 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.113:43369 | |
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.9.94:38567 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.94:38567 | |
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.117.74:46113 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.74:46113 | |
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.117.78:33169 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.78:33169 | |
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.117.77:38099 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.77:38099 | |
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:43851 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:43851 | |
2023-04-09 15:53:25,251 - distributed.core - INFO - Removing comms to tcp://10.76.9.92:38357 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.92:38357 | |
2023-04-09 15:53:25,251 - distributed.core - INFO - Removing comms to tcp://10.76.117.75:43379 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.75:43379 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.79:40527 | |
2023-04-09 15:53:25,251 - distributed.core - INFO - Removing comms to tcp://10.76.117.79:40527 | |
2023-04-09 15:53:25,251 - distributed.core - INFO - Removing comms to tcp://10.76.9.66:35095 | |
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.66:35095 | |
/usr/local/lib/python3.10/site-packages/dask/dataframe/core.py:6984: FutureWarning: Meta is not valid, `map_partitions` and `map_overlap` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns. In the future the meta you passed will not work. | |
warnings.warn( | |
[2023-04-09 15:54:18+0000] INFO - google.cloud.bigquery_storage_v1.writer | RPC termination has signaled streaming pull manager shutdown. | |
[2023-04-09 15:54:18+0000] INFO - google.api_core.bidi | Thread-ConsumeBidirectionalStream exiting | |
2023-04-09 15:55:30,089 - distributed.worker - ERROR - failed during get data with tcp://10.76.9.27:46429 -> tcp://10.76.117.100:34073 | |
[2023-04-09 15:55:30+0000] ERROR - distributed.worker | failed during get data with tcp://10.76.9.27:46429 -> tcp://10.76.117.100:34073 | |
tornado.iostream.StreamClosedError: Stream is closed | |
frames_nbytes = await stream.read_bytes(fmt_size) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read | |
Traceback (most recent call last): | |
null | |
The above exception was the direct cause of the following exception: | |
null | |
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://10.76.9.27:46429 remote=tcp://10.76.117.100:44916>: Stream is closed | |
raise CommClosedError(f"in {obj}: {exc}") from exc | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error | |
convert_stream_closed_error(self, e) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read | |
response = await comm.read(deserializers=serializers) | |
File "/usr/local/lib/python3.10/site-packages/distributed/worker.py", line 1793, in get_data | |
Traceback (most recent call last): | |
2023-04-09 15:55:30,287 - distributed.core - INFO - Lost connection to 'tcp://10.76.117.100:44916' | |
tornado.iostream.StreamClosedError: Stream is closed | |
frames_nbytes = await stream.read_bytes(fmt_size) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read | |
Traceback (most recent call last): | |
null | |
The above exception was the direct cause of the following exception: | |
null | |
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://10.76.9.27:46429 remote=tcp://10.76.117.100:44916>: Stream is closed | |
raise CommClosedError(f"in {obj}: {exc}") from exc | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error | |
convert_stream_closed_error(self, e) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read | |
response = await comm.read(deserializers=serializers) | |
File "/usr/local/lib/python3.10/site-packages/distributed/worker.py", line 1793, in get_data | |
return await func(self, *args, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 987, in wrapper | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
tornado.iostream.StreamClosedError: Stream is closed | |
frames_nbytes = await stream.read_bytes(fmt_size) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read | |
Traceback (most recent call last): | |
null | |
The above exception was the direct cause of the following exception: | |
null | |
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://10.76.9.27:46429 remote=tcp://10.76.117.100:44916>: Stream is closed | |
raise CommClosedError(f"in {obj}: {exc}") from exc | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error | |
convert_stream_closed_error(self, e) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read | |
response = await comm.read(deserializers=serializers) | |
File "/usr/local/lib/python3.10/site-packages/distributed/worker.py", line 1793, in get_data | |
Traceback (most recent call last): | |
[2023-04-09 15:55:30+0000] INFO - distributed.core | Lost connection to 'tcp://10.76.117.100:44916' | |
tornado.iostream.StreamClosedError: Stream is closed | |
frames_nbytes = await stream.read_bytes(fmt_size) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read | |
Traceback (most recent call last): | |
null | |
The above exception was the direct cause of the following exception: | |
null | |
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://10.76.9.27:46429 remote=tcp://10.76.117.100:44916>: Stream is closed | |
raise CommClosedError(f"in {obj}: {exc}") from exc | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error | |
convert_stream_closed_error(self, e) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read | |
response = await comm.read(deserializers=serializers) | |
File "/usr/local/lib/python3.10/site-packages/distributed/worker.py", line 1793, in get_data | |
return await func(self, *args, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 987, in wrapper | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
2023-04-09 15:55:30,717 - distributed.core - INFO - Removing comms to tcp://10.76.117.98:45247 | |
[2023-04-09 15:55:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.98:45247 | |
2023-04-09 15:55:30,722 - distributed.core - INFO - Removing comms to tcp://10.76.117.97:39537 | |
[2023-04-09 15:55:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.97:39537 | |
2023-04-09 15:55:30,722 - distributed.core - INFO - Removing comms to tcp://10.76.117.99:42495 | |
2023-04-09 15:55:30,724 - distributed.core - INFO - Removing comms to tcp://10.76.117.100:34073 | |
[2023-04-09 15:55:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.99:42495 | |
[2023-04-09 15:55:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.100:34073 | |
2023-04-09 15:55:33,310 - distributed.core - INFO - Removing comms to tcp://10.76.9.136:45665 | |
[2023-04-09 15:55:33+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.136:45665 | |
2023-04-09 15:56:00,992 - distributed.core - INFO - Removing comms to tcp://10.76.9.138:40117 | |
[2023-04-09 15:56:00+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.138:40117 | |
2023-04-09 15:56:01,196 - distributed.core - INFO - Removing comms to tcp://10.76.9.137:46167 | |
[2023-04-09 15:56:01+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.137:46167 | |
2023-04-09 15:56:07,179 - distributed.client - WARNING - Couldn't gather 34 keys, rescheduling {"('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 151)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 273)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 114)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 270)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 315)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 89)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 369)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 274)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 266)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 238)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 283)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 243)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 284)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 282)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 254)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 245)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 20)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 153)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 262)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 269)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 286)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 276)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 72)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 167)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 250)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 67)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 224)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 158)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 278)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 257)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 235)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 331)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 325)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 96)": ()} | |
[2023-04-09 15:56:07+0000] WARNING - distributed.client | Couldn't gather 34 keys, rescheduling {"('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 151)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 273)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 114)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 270)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 315)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 89)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 369)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 274)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 266)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 238)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 283)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 243)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 284)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 282)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 254)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 245)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 20)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 153)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 262)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 269)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 286)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 276)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 72)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 167)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 250)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 67)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 224)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 158)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 278)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 257)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 235)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 331)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 325)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 96)": ()} | |
2023-04-09 15:56:35,269 - distributed.core - INFO - Removing comms to tcp://10.76.9.169:42489 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.169:42489 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.170:42145 | |
2023-04-09 15:56:35,269 - distributed.core - INFO - Removing comms to tcp://10.76.9.170:42145 | |
2023-04-09 15:56:35,269 - distributed.core - INFO - Removing comms to tcp://10.76.9.154:42649 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.154:42649 | |
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.117.89:45579 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.89:45579 | |
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.9.156:38383 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.156:38383 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.80:44355 | |
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.117.80:44355 | |
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.9.165:40627 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.165:40627 | |
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.9.166:42423 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.166:42423 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.87:41569 | |
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.117.87:41569 | |
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.65.77:35861 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.65.77:35861 | |
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.9.168:37409 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.168:37409 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.157:32993 | |
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.9.157:32993 | |
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.9.171:46407 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.171:46407 | |
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.117.86:38891 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.86:38891 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.83:43401 | |
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.117.83:43401 | |
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.117.82:43551 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.82:43551 | |
2023-04-09 15:56:35,272 - distributed.core - INFO - Removing comms to tcp://10.76.9.167:38061 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.167:38061 | |
2023-04-09 15:56:35,272 - distributed.core - INFO - Removing comms to tcp://10.76.9.153:45781 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.153:45781 | |
2023-04-09 15:56:35,272 - distributed.core - INFO - Removing comms to tcp://10.76.117.94:40615 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.94:40615 | |
[2023-04-09 15:56:35+0000] ERROR - distributed.shuffle | shuffle_barrier failed during shuffle ca23d72b3d695085e9da34db46f8233c | |
2023-04-09 15:56:35,279 - distributed.shuffle - ERROR - shuffle_barrier failed during shuffle ca23d72b3d695085e9da34db46f8233c | |
tornado.iostream.StreamClosedError: Stream is closed | |
frames_nbytes = await stream.read_bytes(fmt_size) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read | |
Traceback (most recent call last): | |
tornado.iostream.StreamClosedError: Stream is closed | |
frames_nbytes = await stream.read_bytes(fmt_size) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read | |
Traceback (most recent call last): | |
null | |
The above exception was the direct cause of the following exception: | |
null | |
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.shuffle_barrier local=tcp://10.76.9.27:34274 remote=tcp://brett-1f0ecf6d-daskscheduler:8786>: Stream is closed | |
raise CommClosedError(f"in {obj}: {exc}") from exc | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error | |
convert_stream_closed_error(self, e) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read | |
response = await comm.read(deserializers=deserializers) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
await self.scheduler.shuffle_barrier(id=self.id, run_id=self.run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 117, in barrier | |
await shuffle.barrier() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 652, in _barrier | |
value = future.result() | |
File "/usr/local/lib/python3.10/site-packages/tornado/gen.py", line 762, in run | |
result = yield future | |
File "/usr/local/lib/python3.10/site-packages/distributed/utils.py", line 389, in f | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/utils.py", line 416, in sync | |
result = sync(self.worker.loop, self._barrier, shuffle_id, run_ids) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 865, in barrier | |
return _get_worker_extension().barrier(id, run_ids) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_shuffle.py", line 91, in shuffle_barrier | |
Traceback (most recent call last): | |
null | |
The above exception was the direct cause of the following exception: | |
null | |
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.shuffle_barrier local=tcp://10.76.9.27:34274 remote=tcp://brett-1f0ecf6d-daskscheduler:8786>: Stream is closed | |
raise CommClosedError(f"in {obj}: {exc}") from exc | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error | |
convert_stream_closed_error(self, e) | |
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read | |
response = await comm.read(deserializers=deserializers) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
await self.scheduler.shuffle_barrier(id=self.id, run_id=self.run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 117, in barrier | |
await shuffle.barrier() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 652, in _barrier | |
value = future.result() | |
File "/usr/local/lib/python3.10/site-packages/tornado/gen.py", line 762, in run | |
result = yield future | |
File "/usr/local/lib/python3.10/site-packages/distributed/utils.py", line 389, in f | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/utils.py", line 416, in sync | |
result = sync(self.worker.loop, self._barrier, shuffle_id, run_ids) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 865, in barrier | |
return _get_worker_extension().barrier(id, run_ids) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_shuffle.py", line 91, in shuffle_barrier | |
Traceback (most recent call last): | |
2023-04-09 15:56:35,283 - distributed.worker - WARNING - Compute Failed | |
[2023-04-09 15:56:35+0000] WARNING - distributed.worker | Compute Failed | |
Key: shuffle-barrier-ca23d72b3d695085e9da34db46f8233c | |
Function: shuffle_barrier | |
args: ('ca23d72b3d695085e9da34db46f8233c', [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, | |
Key: shuffle-barrier-ca23d72b3d695085e9da34db46f8233c | |
Function: shuffle_barrier | |
args: ('ca23d72b3d695085e9da34db46f8233c', [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, | |
kwargs: {} | |
Exception: "RuntimeError('shuffle_barrier failed during shuffle ca23d72b3d695085e9da34db46f8233c')" | |
kwargs: {} | |
Exception: "RuntimeError('shuffle_barrier failed during shuffle ca23d72b3d695085e9da34db46f8233c')" | |
2023-04-09 15:56:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.127:38379 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.127:38379 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:36685 | |
2023-04-09 15:56:35,712 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:36685 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.84:36653 | |
2023-04-09 15:56:35,719 - distributed.core - INFO - Removing comms to tcp://10.76.117.84:36653 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.159:36777 | |
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.9.159:36777 | |
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.9.144:38619 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.144:38619 | |
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.9.148:36611 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.148:36611 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.102:46079 | |
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.117.102:46079 | |
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.9.158:43367 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.158:43367 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.145:44429 | |
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.9.145:44429 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.126:35341 | |
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.9.126:35341 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.101:39155 | |
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.117.101:39155 | |
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.9.134:35609 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.134:35609 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.65.79:40717 | |
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.65.79:40717 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.152:33859 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.152:33859 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.155:34533 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.155:34533 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.117.85:46813 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.85:46813 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.81:44087 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.117.81:44087 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.142:46145 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.142:46145 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.141:34333 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.141:34333 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.139:46663 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.139:46663 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.162:36021 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.162:36021 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.140:35963 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.140:35963 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.160:36573 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.160:36573 | |
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.150:42363 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.150:42363 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.146:36013 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.146:36013 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.164:42399 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.164:42399 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.149:38583 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.149:38583 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.143:43253 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.143:43253 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.151:38761 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.151:38761 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.163:34535 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.163:34535 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.161:40815 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.161:40815 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:45303 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:45303 | |
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.135:39905 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.135:39905 | |
2023-04-09 15:56:35,724 - distributed.core - INFO - Removing comms to tcp://10.76.9.132:36293 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.132:36293 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.129:38587 | |
2023-04-09 15:56:35,724 - distributed.core - INFO - Removing comms to tcp://10.76.9.129:38587 | |
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.130:39149 | |
2023-04-09 15:56:35,724 - distributed.core - INFO - Removing comms to tcp://10.76.9.130:39149 | |
2023-04-09 15:56:45,485 - distributed.core - INFO - Removing comms to tcp://10.76.117.88:36365 | |
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.88:36365 | |
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.65.80:42509 | |
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.65.80:42509 | |
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:38943 | |
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:38943 | |
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.117.91:38851 | |
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.91:38851 | |
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.9.133:46883 | |
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.133:46883 | |
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.93:44355 | |
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.117.93:44355 | |
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.96:37269 | |
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.117.96:37269 | |
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.131:44185 | |
2023-04-09 15:56:45,487 - distributed.core - INFO - Removing comms to tcp://10.76.9.131:44185 | |
2023-04-09 15:56:48,307 - distributed.shuffle._comms - ERROR - 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
Traceback (most recent call last): | |
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._comms | 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
Traceback (most recent call last): | |
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._comms | 'ca23d72b3d695085e9da34db46f8233c' | |
2023-04-09 15:56:48,320 - distributed.shuffle._comms - ERROR - 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
Traceback (most recent call last): | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
Traceback (most recent call last): | |
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._comms | 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
Traceback (most recent call last): | |
2023-04-09 15:56:48,331 - distributed.shuffle._comms - ERROR - 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
Traceback (most recent call last): | |
2023-04-09 15:56:48,342 - distributed.shuffle._comms - ERROR - 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
Traceback (most recent call last): | |
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._comms | 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
Traceback (most recent call last): | |
2023-04-09 15:56:48,352 - distributed.shuffle._worker_extension - ERROR - 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
await self._process(id, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 107, in process | |
raise self._exception | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 222, in raise_on_exception | |
self._comm_buffer.raise_on_exception() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 173, in inputs_done | |
await shuffle.inputs_done() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 597, in shuffle_inputs_done | |
Traceback (most recent call last): | |
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._worker_extension | 'ca23d72b3d695085e9da34db46f8233c' | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
await self._process(id, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 107, in process | |
raise self._exception | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 222, in raise_on_exception | |
self._comm_buffer.raise_on_exception() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 173, in inputs_done | |
await shuffle.inputs_done() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 597, in shuffle_inputs_done | |
Traceback (most recent call last): | |
2023-04-09 15:56:48,352 - distributed.core - ERROR - Exception while handling op shuffle_inputs_done | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
await self._process(id, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 107, in process | |
raise self._exception | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 222, in raise_on_exception | |
self._comm_buffer.raise_on_exception() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 173, in inputs_done | |
await shuffle.inputs_done() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 597, in shuffle_inputs_done | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
[2023-04-09 15:56:48+0000] ERROR - distributed.core | Exception while handling op shuffle_inputs_done | |
KeyError: 'ca23d72b3d695085e9da34db46f8233c' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
return await self.rpc(address).shuffle_receive( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send | |
await self.send(address, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process | |
await self._process(id, shards) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 107, in process | |
raise self._exception | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 222, in raise_on_exception | |
self._comm_buffer.raise_on_exception() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 173, in inputs_done | |
await shuffle.inputs_done() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 597, in shuffle_inputs_done | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
/usr/local/lib/python3.10/site-packages/dask/dataframe/core.py:6984: FutureWarning: Meta is not valid, `map_partitions` and `map_overlap` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns. In the future the meta you passed will not work. | |
warnings.warn( | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.104:37161 | |
2023-04-09 15:57:51,606 - distributed.core - INFO - Removing comms to tcp://10.76.117.104:37161 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.125:39587 | |
2023-04-09 15:57:51,606 - distributed.core - INFO - Removing comms to tcp://10.76.9.125:39587 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.90:40771 | |
2023-04-09 15:57:51,606 - distributed.core - INFO - Removing comms to tcp://10.76.117.90:40771 | |
2023-04-09 15:57:51,609 - distributed.core - INFO - Event loop was unresponsive in Worker for 63.60s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Event loop was unresponsive in Worker for 63.60s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. | |
2023-04-09 15:57:51,623 - distributed.core - INFO - Removing comms to tcp://10.76.17.3:33923 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.3:33923 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.147:38979 | |
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.9.147:38979 | |
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.9.128:34777 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.128:34777 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.92:38415 | |
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.117.92:38415 | |
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.7.6:40431 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.6:40431 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.3:36247 | |
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.7.3:36247 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.65.78:43525 | |
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.65.78:43525 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.4:42465 | |
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.7.4:42465 | |
2023-04-09 15:57:51,628 - distributed.core - INFO - Removing comms to tcp://10.76.117.103:41355 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.103:41355 | |
2023-04-09 15:57:51,628 - distributed.core - INFO - Removing comms to tcp://10.76.7.5:33549 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.5:33549 | |
2023-04-09 15:57:51,628 - distributed.core - INFO - Removing comms to tcp://10.76.117.95:32937 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.95:32937 | |
2023-04-09 15:57:51,628 - distributed.core - INFO - Removing comms to tcp://10.76.19.3:37473 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.3:37473 | |
2023-04-09 15:57:51,630 - distributed.core - INFO - Removing comms to tcp://10.76.28.3:45377 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.3:45377 | |
2023-04-09 15:57:51,630 - distributed.core - INFO - Removing comms to tcp://10.76.17.4:34897 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.4:34897 | |
2023-04-09 15:57:51,630 - distributed.core - INFO - Removing comms to tcp://10.76.28.5:40339 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.5:40339 | |
2023-04-09 15:57:51,630 - distributed.core - INFO - Removing comms to tcp://10.76.28.4:38529 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.4:38529 | |
2023-04-09 15:57:51,631 - distributed.core - INFO - Removing comms to tcp://10.76.19.4:36373 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.4:36373 | |
2023-04-09 15:57:51,631 - distributed.core - INFO - Removing comms to tcp://10.76.7.7:46001 | |
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.7:46001 | |
[2023-04-09 15:57:54+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.11:37173 | |
2023-04-09 15:57:54,416 - distributed.core - INFO - Removing comms to tcp://10.76.7.11:37173 | |
[2023-04-09 15:57:54+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.8:36697 | |
2023-04-09 15:57:54,425 - distributed.core - INFO - Removing comms to tcp://10.76.7.8:36697 | |
[2023-04-09 15:57:54+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.10:39997 | |
2023-04-09 15:57:54,435 - distributed.core - INFO - Removing comms to tcp://10.76.7.10:39997 | |
[2023-04-09 15:57:54+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.9:46433 | |
2023-04-09 15:57:54,435 - distributed.core - INFO - Removing comms to tcp://10.76.7.9:46433 | |
[2023-04-09 15:58:00+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.12:38649 | |
2023-04-09 15:58:00,159 - distributed.core - INFO - Removing comms to tcp://10.76.7.12:38649 | |
[2023-04-09 15:58:00+0000] ERROR - distributed.core | Exception while handling op shuffle_receive | |
RuntimeError: Worker tcp://10.76.7.12:38649 left during active shuffle a4d98659bee6af96be24ed925c9a884f | |
raise self._exception | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 163, in raise_if_closed | |
self.raise_if_closed() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 155, in _write_to_disk | |
await self._write_to_disk(groups) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 480, in _receive | |
await self._receive(data) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 463, in receive | |
await shuffle.receive(data) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 588, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
2023-04-09 15:58:00,236 - distributed.core - ERROR - Exception while handling op shuffle_receive | |
RuntimeError: Worker tcp://10.76.7.12:38649 left during active shuffle a4d98659bee6af96be24ed925c9a884f | |
raise self._exception | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 163, in raise_if_closed | |
self.raise_if_closed() | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 155, in _write_to_disk | |
await self._write_to_disk(groups) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 480, in _receive | |
await self._receive(data) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 463, in receive | |
await shuffle.receive(data) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 588, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
2023-04-09 15:58:00,249 - distributed.core - ERROR - Exception while handling op shuffle_receive | |
[2023-04-09 15:58:00+0000] ERROR - distributed.core | Exception while handling op shuffle_receive | |
RuntimeError: Stale shuffle | |
raise RuntimeError("Stale shuffle") | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 682, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
[2023-04-09 15:58:00+0000] ERROR - distributed.core | Exception while handling op shuffle_receive | |
KeyError: 'a4d98659bee6af96be24ed925c9a884f' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
RuntimeError: Stale shuffle | |
raise RuntimeError("Stale shuffle") | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 682, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
[2023-04-09 15:58:00+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.14:45377 | |
2023-04-09 15:58:00,311 - distributed.core - ERROR - Exception while handling op shuffle_receive | |
KeyError: 'a4d98659bee6af96be24ed925c9a884f' | |
state = self.states[id] | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get | |
result = handler(**msg) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm | |
raise exc.with_traceback(tb) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv | |
return await send_recv(comm=comm, op=key, **kwargs) | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc | |
result = await self.worker.scheduler.shuffle_get( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle | |
shuffle = await self._refresh_shuffle( | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run | |
shuffle = await self._get_shuffle_run(shuffle_id, run_id) | |
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive | |
result = await result | |
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm | |
Traceback (most recent call last): | |
2023-04-09 15:58:00,645 - distributed.core - INFO - Removing comms to tcp://10.76.7.14:45377 | |
/usr/local/lib/python3.10/site-packages/dask/dataframe/core.py:6984: FutureWarning: Meta is not valid, `map_partitions` and `map_overlap` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns. In the future the meta you passed will not work. | |
[2023-04-09 15:58:02+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.13:36351 | |
warnings.warn( | |
2023-04-09 15:58:02,976 - distributed.core - INFO - Removing comms to tcp://10.76.7.13:36351 | |
2023-04-09 15:58:03,985 - distributed.core - INFO - Removing comms to tcp://10.76.7.15:45883 | |
[2023-04-09 15:58:03+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.15:45883 | |
2023-04-09 15:58:06,430 - distributed.core - INFO - Removing comms to tcp://10.76.7.16:43913 | |
[2023-04-09 15:58:06+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.16:43913 | |
2023-04-09 15:58:07,441 - distributed.core - INFO - Removing comms to tcp://10.76.7.17:33859 | |
[2023-04-09 15:58:07+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.17:33859 | |
2023-04-09 15:58:09,160 - distributed.core - INFO - Removing comms to tcp://10.76.7.18:38483 | |
[2023-04-09 15:58:09+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.18:38483 | |
[2023-04-09 15:58:10+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.19:39703 | |
2023-04-09 15:58:10,302 - distributed.core - INFO - Removing comms to tcp://10.76.7.19:39703 | |
[2023-04-09 15:58:12+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.20:37461 | |
2023-04-09 15:58:12,022 - distributed.core - INFO - Removing comms to tcp://10.76.7.20:37461 | |
[2023-04-09 15:58:13+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.21:42639 | |
2023-04-09 15:58:13,031 - distributed.core - INFO - Removing comms to tcp://10.76.7.21:42639 | |
2023-04-09 15:58:15,056 - distributed.core - INFO - Removing comms to tcp://10.76.7.22:37661 | |
[2023-04-09 15:58:15+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.22:37661 | |
[2023-04-09 15:58:17+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.23:39713 | |
2023-04-09 15:58:17,212 - distributed.core - INFO - Removing comms to tcp://10.76.7.23:39713 | |
2023-04-09 15:58:19,193 - distributed.core - INFO - Removing comms to tcp://10.76.7.24:41493 | |
[2023-04-09 15:58:19+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.24:41493 | |
[2023-04-09 15:58:20+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.25:45635 | |
2023-04-09 15:58:20,074 - distributed.core - INFO - Removing comms to tcp://10.76.7.25:45635 | |
2023-04-09 15:58:22,083 - distributed.core - INFO - Removing comms to tcp://10.76.7.26:43345 | |
[2023-04-09 15:58:22+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.26:43345 | |
[2023-04-09 15:58:24+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.27:43925 | |
2023-04-09 15:58:24,092 - distributed.core - INFO - Removing comms to tcp://10.76.7.27:43925 | |
[2023-04-09 15:58:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.28:36047 | |
2023-04-09 15:58:25,098 - distributed.core - INFO - Removing comms to tcp://10.76.7.28:36047 | |
2023-04-09 15:58:27,118 - distributed.core - INFO - Removing comms to tcp://10.76.7.29:36805 | |
[2023-04-09 15:58:27+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.29:36805 | |
[2023-04-09 15:58:29+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.31:45077 | |
2023-04-09 15:58:29,124 - distributed.core - INFO - Removing comms to tcp://10.76.7.31:45077 | |
[2023-04-09 15:58:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.32:39645 | |
2023-04-09 15:58:30,128 - distributed.core - INFO - Removing comms to tcp://10.76.7.32:39645 | |
2023-04-09 15:58:32,142 - distributed.core - INFO - Removing comms to tcp://10.76.7.30:39415 | |
[2023-04-09 15:58:32+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.30:39415 | |
2023-04-09 15:58:34,151 - distributed.core - INFO - Removing comms to tcp://10.76.7.33:43941 | |
[2023-04-09 15:58:34+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.33:43941 | |
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.28.14:38443 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.14:38443 | |
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.19.16:34879 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.16:34879 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.15:33929 | |
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.17.15:33929 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.39:35533 | |
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.7.39:35533 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.18:43819 | |
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.28.18:43819 | |
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.7.49:41661 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.49:41661 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.45:39845 | |
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.7.45:39845 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.16:43729 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.28.16:43729 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.7.48:40677 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.48:40677 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.7.34:33577 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.34:33577 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.17.22:38489 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.22:38489 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.17:33719 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.17.17:33719 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.19.11:37067 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.11:37067 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.7.46:39677 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.46:39677 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.17.6:46027 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.6:46027 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.19.17:37697 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.17:37697 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.19.15:32769 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.15:32769 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.28.19:41825 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.19:41825 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.28.15:37167 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.15:37167 | |
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.17.20:35041 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.20:35041 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.17:46221 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.17:46221 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.20:36963 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.20:36963 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.19:33049 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.19.19:33049 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.7.37:36807 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.37:36807 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.19.18:43969 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.18:43969 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.36:38533 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.7.36:38533 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.17.11:46091 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.11:46091 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.41:38825 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.7.41:38825 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.7.50:44429 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.50:44429 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.12:40011 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.12:40011 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.6:42893 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.6:42893 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.19.7:44433 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.7:44433 | |
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.9:45017 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.9:45017 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.19.5:38911 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.5:38911 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.9:35877 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.19.9:35877 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.28.7:42501 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.7:42501 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:43793 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:43793 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.7.35:39413 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.35:39413 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.19.21:40021 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.21:40021 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.17.14:42727 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.14:42727 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.7.42:34953 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.42:34953 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.17.16:37455 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.16:37455 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.17.21:35309 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.21:35309 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.13:39117 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.19.13:39117 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.17.9:35497 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.9:35497 | |
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.28.11:33319 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.11:33319 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.18:44755 | |
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.17.18:44755 | |
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.17.8:36251 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.8:36251 | |
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.19.10:36831 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.10:36831 | |
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.19.20:37825 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.20:37825 | |
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.7.43:38071 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.43:38071 | |
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.7.40:46691 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.40:46691 | |
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.17.13:33681 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.13:33681 | |
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.19.6:34521 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.6:34521 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.12:40935 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.17.12:40935 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.19:34327 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.17.19:34327 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.7.47:45667 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.47:45667 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.28.8:45155 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.8:45155 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.5:40253 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.17.5:40253 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.17.7:46267 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.7:46267 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.8:41017 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.19.8:41017 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.7.38:45473 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.38:45473 | |
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.19.14:34803 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.14:34803 | |
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.7.44:38215 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.44:38215 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.10:40415 | |
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.17.10:40415 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.13:42999 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.10:42223 | |
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.12:45525 | |
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.28.13:42999 | |
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.28.10:42223 | |
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.19.12:45525 | |
[2023-04-09 16:01:03+0000] ERROR - asyncore_wsgi.SimpleWebSocketServer | Exception in <web_pdb.web_console.WebConsoleSocket connected 127.0.0.1:39698 at 0x7f768dbcba90>! | |
asyncore_wsgi.SimpleWebSocketServer.WebSocketError: remote socket closed | |
raise WebSocketError("remote socket closed") | |
File "/usr/local/lib/python3.10/site-packages/asyncore_wsgi/SimpleWebSocketServer.py", line 290, in _handleData | |
self._handleData() | |
File "/usr/local/lib/python3.10/site-packages/asyncore_wsgi/SimpleWebSocketServer.py", line 596, in handle_read | |
Traceback (most recent call last): | |
2023-04-09 16:11:20,280 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:37003 | |
[2023-04-09 16:11:20+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:37003 | |
2023-04-09 16:36:16,462 - distributed._signals - INFO - Received signal SIGTERM (15) | |
2023-04-09 16:36:16,463 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.76.9.27:43473'. Reason: nanny-close | |
2023-04-09 16:36:16,463 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close | |
2023-04-09 16:36:16,464 - distributed.worker - INFO - Stopping worker at tcp://10.76.9.27:46429. Reason: nanny-close | |
[2023-04-09 16:36:16+0000] INFO - distributed.worker | Stopping worker at tcp://10.76.9.27:46429. Reason: nanny-close | |
2023-04-09 16:36:16,472 - distributed.core - INFO - Connection to tcp://brett-1f0ecf6d-daskscheduler:8786 has been closed. | |
[2023-04-09 16:36:16+0000] INFO - distributed.core | Connection to tcp://brett-1f0ecf6d-daskscheduler:8786 has been closed. | |
/usr/local/lib/python3.10/site-packages/distributed/cli/dask_worker.py:264: FutureWarning: dask-worker is deprecated and will be removed in a future release; use `dask worker` instead | |
warnings.warn( | |
2023-04-09 16:36:52,371 - distributed.nanny - INFO - Start Nanny at: 'tcp://10.76.34.20:35471' | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - Start worker at: tcp://10.76.34.20:45213 | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - Listening to: tcp://10.76.34.20:45213 | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - Worker name: brett-1f0ecf6d-daskworkers-large-865bf95dbc-fbk8j | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - dashboard at: 10.76.34.20:8787 | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - Waiting to connect to: tcp://brett-1f0ecf6d-daskscheduler:8786 | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - ------------------------------------------------- | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - Threads: 4 | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - Memory: 89.41 GiB | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-v6e12mq5 | |
2023-04-09 16:36:53,088 - distributed.worker - INFO - ------------------------------------------------- | |
2023-04-09 16:36:53,316 - distributed.worker - INFO - Registered to: tcp://brett-1f0ecf6d-daskscheduler:8786 | |
2023-04-09 16:36:53,316 - distributed.worker - INFO - ------------------------------------------------- | |
2023-04-09 16:36:53,317 - distributed.core - INFO - Starting established connection to tcp://brett-1f0ecf6d-daskscheduler:8786 | |
[2023-04-09 16:36:56+0000] INFO - prefect.CloudTaskRunner | Task 'on_demand_auto_calibration-places/configs/south_central/2022_Q4/saturday.yaml': Starting task run... | |
[2023-04-09 16:36:56+0000] ERROR - prefect | Worker died while running task; retrying (attempt 2 of 3) | |
[2023-04-09 16:37:07+0000] INFO - root | Sources already exist, skipping DataManager(name='on_demand_auto_calibration', data_config='places/configs/south_central/2022_Q4/saturday.yaml') | |
[2023-04-09 16:37:07+0000] INFO - prefect.CloudTaskRunner | SKIP signal raised: SKIP(None) | |
[2023-04-09 16:37:07+0000] INFO - prefect.CloudTaskRunner | Task 'on_demand_auto_calibration-places/configs/south_central/2022_Q4/saturday.yaml': Finished task run for task with final state: 'Skipped' | |
2023-04-09 16:37:09,386 - distributed._signals - INFO - Received signal SIGTERM (15) | |
2023-04-09 16:37:09,386 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.76.34.20:35471'. Reason: nanny-close | |
2023-04-09 16:37:09,386 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close | |
2023-04-09 16:37:09,387 - distributed.worker - INFO - Stopping worker at tcp://10.76.34.20:45213. Reason: nanny-close | |
[2023-04-09 16:37:09+0000] INFO - distributed.worker | Stopping worker at tcp://10.76.34.20:45213. Reason: nanny-close | |
[2023-04-09 16:37:09+0000] INFO - distributed.core | Connection to tcp://brett-1f0ecf6d-daskscheduler:8786 has been closed. | |
2023-04-09 16:37:09,388 - distributed.core - INFO - Connection to tcp://brett-1f0ecf6d-daskscheduler:8786 has been closed. | |
[2023-04-09 16:37:09+0000] INFO - distributed.nanny | Worker closed | |
2023-04-09 16:37:09,389 - distributed.nanny - INFO - Worker closed | |
2023-04-09 16:37:10,664 - distributed.dask_worker - INFO - End worker |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment