--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Action | Mean duration (s) | Num calls | Total time (s) | Percentage % |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Total | - | 14775 | 295.54 | 100 % |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| run_training_epoch | 144.66 | 2 | 289.32 | 97.895 |
| run_training_batch | 0.68959 | 337 | 232.39 | 78.634 |
| [Strategy]SingleDeviceStrategy.training_step | 0.68952 | 337 | 232.37 | 78.626 |
| optimizer_step | 0.14866 | 674 | 100.2 | 33.903 |
| [LightningModule]OptiSpeech.on_before_optimizer_step | 0.14329 | 674 | 96.578 | 32.679 |
| [Strategy]SingleDeviceStrategy.validation_step | 0.63157 | 38 | 23.999 | 8.1207 |
| [LightningModule]OptiSpeech.on_validation_end | 3.703 | 1 | 3.703 | 1.253 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_train_epoch_end | 1.3929 | 2 | 2.7859 | 0.94264 |
| [Callback]TQDMProgressBar.on_train_batch_end | 0.0041187 | 337 | 1.388 | 0.46965 |
| [_TrainingEpochLoop].train_dataloader_next | 0.0011521 | 337 | 0.38826 | 0.13137 |
| [Strategy]SingleDeviceStrategy.batch_to_device | 0.00059782 | 375 | 0.22418 | 0.075857 |
| [LightningModule]OptiSpeech.transfer_batch_to_device | 0.0005096 | 375 | 0.1911 | 0.064662 |
| [_EvaluationLoop].val_next | 0.0025568 | 38 | 0.09716 | 0.032876 |
| [Callback]RichModelSummary.on_fit_start | 0.065201 | 1 | 0.065201 | 0.022062 |
| [Callback]TQDMProgressBar.on_validation_batch_end | 0.00032541 | 38 | 0.012366 | 0.0041842 |
| [Callback]TQDMProgressBar.on_train_epoch_end | 0.0051144 | 2 | 0.010229 | 0.0034611 |
| [LightningDataModule]TextWavDataModule.setup | 0.0077251 | 1 | 0.0077251 | 0.0026139 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_train_batch_end | 2.1625e-05 | 337 | 0.0072876 | 0.0024659 |
| [LightningModule]OptiSpeech.on_validation_model_eval | 0.0051472 | 1 | 0.0051472 | 0.0017417 |
| [Callback]RichModelSummary.on_after_backward | 5.9571e-06 | 674 | 0.0040151 | 0.0013586 |
| [Callback]RichModelSummary.on_before_backward | 5.0997e-06 | 674 | 0.0034372 | 0.001163 |
| [LightningModule]OptiSpeech.configure_optimizers | 0.00326 | 1 | 0.00326 | 0.0011031 |
| [Callback]TQDMProgressBar.on_train_start | 0.0028371 | 1 | 0.0028371 | 0.00095998 |
| [Callback]RichModelSummary.on_before_optimizer_step | 4.1129e-06 | 674 | 0.0027721 | 0.00093798 |
| [LightningModule]OptiSpeech.on_validation_model_zero_grad | 0.0024871 | 1 | 0.0024871 | 0.00084154 |
| [Callback]TQDMProgressBar.on_validation_end | 0.0015967 | 1 | 0.0015967 | 0.00054028 |
| [Callback]TQDMProgressBar.on_validation_batch_start | 4.0845e-05 | 38 | 0.0015521 | 0.00052519 |
| [Callback]TQDMProgressBar.on_before_optimizer_step | 2.2109e-06 | 674 | 0.0014901 | 0.00050421 |
| [LightningModule]OptiSpeech.on_after_backward | 2.123e-06 | 674 | 0.0014309 | 0.00048416 |
| [Callback]TQDMProgressBar.on_after_backward | 2.0308e-06 | 674 | 0.0013688 | 0.00046314 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_before_optimizer_step | 1.972e-06 | 674 | 0.0013291 | 0.00044973 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_after_backward | 1.8714e-06 | 674 | 0.0012613 | 0.00042679 |
| [Callback]RichModelSummary.on_train_batch_start | 3.6811e-06 | 337 | 0.0012405 | 0.00041976 |
| [LightningModule]OptiSpeech.on_before_backward | 1.7355e-06 | 674 | 0.0011697 | 0.00039579 |
| [Callback]TQDMProgressBar.on_train_epoch_start | 0.00056056 | 2 | 0.0011211 | 0.00037935 |
| [Callback]RichModelSummary.on_train_batch_end | 3.2838e-06 | 337 | 0.0011066 | 0.00037445 |
| [Callback]TQDMProgressBar.on_before_backward | 1.6098e-06 | 674 | 0.001085 | 0.00036713 |
| [Callback]TQDMProgressBar.on_validation_start | 0.0010137 | 1 | 0.0010137 | 0.00034301 |
| [LightningModule]OptiSpeech.on_before_batch_transfer | 2.7024e-06 | 375 | 0.0010134 | 0.0003429 |
| [Callback]TQDMProgressBar.on_train_end | 0.00099926 | 1 | 0.00099926 | 0.00033812 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_before_backward | 1.3884e-06 | 674 | 0.00093577 | 0.00031663 |
| [LightningModule]OptiSpeech.on_after_batch_transfer | 2.2515e-06 | 375 | 0.0008443 | 0.00028569 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_train_batch_start | 2.3026e-06 | 337 | 0.00077599 | 0.00026257 |
| [Callback]TQDMProgressBar.on_train_batch_start | 2.239e-06 | 337 | 0.00075456 | 0.00025532 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.setup | 0.00069392 | 1 | 0.00069392 | 0.0002348 |
| [LightningModule]OptiSpeech.on_train_batch_start | 2.0016e-06 | 337 | 0.00067454 | 0.00022824 |
| [Strategy]SingleDeviceStrategy.on_train_batch_start | 1.7455e-06 | 337 | 0.00058825 | 0.00019905 |
| [LightningModule]OptiSpeech.on_train_batch_end | 1.6503e-06 | 337 | 0.00055617 | 0.00018819 |
| [LightningDataModule]TextWavDataModule.val_dataloader | 0.00048264 | 1 | 0.00048264 | 0.00016331 |
| [LightningModule]OptiSpeech.on_validation_epoch_start | 0.00038351 | 1 | 0.00038351 | 0.00012977 |
| [LightningDataModule]TextWavDataModule.train_dataloader | 0.00035959 | 1 | 0.00035959 | 0.00012167 |
| [Callback]RichModelSummary.on_validation_batch_end | 3.3289e-06 | 38 | 0.0001265 | 4.2803e-05 |
| [Callback]RichModelSummary.on_validation_batch_start | 2.6035e-06 | 38 | 9.8932e-05 | 3.3476e-05 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_validation_batch_end | 2.5614e-06 | 38 | 9.7333e-05 | 3.2934e-05 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_validation_batch_start | 1.6966e-06 | 38 | 6.4471e-05 | 2.1815e-05 |
| [LightningModule]OptiSpeech.on_validation_batch_end | 1.6545e-06 | 38 | 6.2872e-05 | 2.1274e-05 |
| [LightningModule]OptiSpeech.on_validation_batch_start | 1.3753e-06 | 38 | 5.2262e-05 | 1.7684e-05 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_validation_end | 5.1722e-05 | 1 | 5.1722e-05 | 1.7501e-05 |
| [Callback]RichModelSummary.on_train_start | 3.7191e-05 | 1 | 3.7191e-05 | 1.2584e-05 |
| [LightningModule]OptiSpeech.on_train_epoch_start | 1.566e-05 | 2 | 3.132e-05 | 1.0598e-05 |
| [Callback]RichModelSummary.on_train_epoch_end | 7.1451e-06 | 2 | 1.429e-05 | 4.8354e-06 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_train_start | 1.1461e-05 | 1 | 1.1461e-05 | 3.878e-06 |
| [LightningDataModule]TextWavDataModule.state_dict | 3.11e-06 | 3 | 9.33e-06 | 3.157e-06 |
| [LightningModule]OptiSpeech.on_load_checkpoint | 7.8501e-06 | 1 | 7.8501e-06 | 2.6562e-06 |
| [Callback]RichModelSummary.on_validation_start | 7.1498e-06 | 1 | 7.1498e-06 | 2.4193e-06 |
| [Strategy]SingleDeviceStrategy.on_train_start | 6.9803e-06 | 1 | 6.9803e-06 | 2.3619e-06 |
| [Callback]RichModelSummary.on_save_checkpoint | 2.2131e-06 | 3 | 6.6394e-06 | 2.2466e-06 |
| [Callback]RichModelSummary.on_train_epoch_start | 3.0354e-06 | 2 | 6.0708e-06 | 2.0542e-06 |
| [LightningModule]OptiSpeech.on_train_start | 6.0303e-06 | 1 | 6.0303e-06 | 2.0405e-06 |
| [LightningModule]OptiSpeech.on_validation_start | 5.94e-06 | 1 | 5.94e-06 | 2.0099e-06 |
| [Strategy]SingleDeviceStrategy.on_validation_start | 5.5204e-06 | 1 | 5.5204e-06 | 1.8679e-06 |
| [Callback]TQDMProgressBar.setup | 5.5097e-06 | 1 | 5.5097e-06 | 1.8643e-06 |
| [Strategy]SingleDeviceStrategy.on_validation_end | 5.4799e-06 | 1 | 5.4799e-06 | 1.8542e-06 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_train_epoch_start | 2.6252e-06 | 2 | 5.2503e-06 | 1.7765e-06 |
| [Callback]RichModelSummary.on_train_end | 5.1502e-06 | 1 | 5.1502e-06 | 1.7427e-06 |
| [Callback]RichModelSummary.setup | 5.0701e-06 | 1 | 5.0701e-06 | 1.7156e-06 |
| [Callback]RichModelSummary.on_fit_end | 4.8997e-06 | 1 | 4.8997e-06 | 1.6579e-06 |
| [Callback]RichModelSummary.on_load_checkpoint | 4.5402e-06 | 1 | 4.5402e-06 | 1.5363e-06 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_save_checkpoint | 1.4931e-06 | 3 | 4.4792e-06 | 1.5156e-06 |
| [Callback]TQDMProgressBar.on_save_checkpoint | 1.3902e-06 | 3 | 4.1705e-06 | 1.4112e-06 |
| [LightningModule]OptiSpeech.on_train_epoch_end | 1.9001e-06 | 2 | 3.8003e-06 | 1.2859e-06 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_validation_start | 3.5609e-06 | 1 | 3.5609e-06 | 1.2049e-06 |
| [Callback]TQDMProgressBar.on_fit_start | 3.4901e-06 | 1 | 3.4901e-06 | 1.181e-06 |
| [LightningModule]OptiSpeech.setup | 3.4301e-06 | 1 | 3.4301e-06 | 1.1606e-06 |
| [LightningDataModule]TextWavDataModule.teardown | 3.281e-06 | 1 | 3.281e-06 | 1.1102e-06 |
| [Callback]RichModelSummary.on_validation_end | 3.1698e-06 | 1 | 3.1698e-06 | 1.0725e-06 |
| [LightningModule]OptiSpeech.on_save_checkpoint | 9.8689e-07 | 3 | 2.9607e-06 | 1.0018e-06 |
| [LightningModule]OptiSpeech.on_train_end | 2.8699e-06 | 1 | 2.8699e-06 | 9.7107e-07 |
| [Callback]RichModelSummary.on_validation_epoch_end | 2.7502e-06 | 1 | 2.7502e-06 | 9.3058e-07 |
| [Callback]RichModelSummary.on_validation_epoch_start | 2.7004e-06 | 1 | 2.7004e-06 | 9.1372e-07 |
| [LightningModule]OptiSpeech.configure_callbacks | 2.59e-06 | 1 | 2.59e-06 | 8.7638e-07 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_train_end | 2.4401e-06 | 1 | 2.4401e-06 | 8.2564e-07 |
| [Callback]RichModelSummary.teardown | 2.0801e-06 | 1 | 2.0801e-06 | 7.0384e-07 |
| [Callback]TQDMProgressBar.on_load_checkpoint | 2.0401e-06 | 1 | 2.0401e-06 | 6.9029e-07 |
| [Callback]TQDMProgressBar.on_validation_epoch_start | 1.83e-06 | 1 | 1.83e-06 | 6.1923e-07 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_validation_epoch_start | 1.8203e-06 | 1 | 1.8203e-06 | 6.1592e-07 |
| [Callback]TQDMProgressBar.on_fit_end | 1.7998e-06 | 1 | 1.7998e-06 | 6.0899e-07 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_fit_start | 1.7099e-06 | 1 | 1.7099e-06 | 5.7858e-07 |
| [LightningModule]OptiSpeech.on_fit_start | 1.7001e-06 | 1 | 1.7001e-06 | 5.7527e-07 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_load_checkpoint | 1.6699e-06 | 1 | 1.6699e-06 | 5.6503e-07 |
| [Callback]TQDMProgressBar.teardown | 1.5302e-06 | 1 | 1.5302e-06 | 5.1776e-07 |
| [LightningModule]OptiSpeech.on_fit_end | 1.5101e-06 | 1 | 1.5101e-06 | 5.1098e-07 |
| [Callback]TQDMProgressBar.on_validation_epoch_end | 1.4897e-06 | 1 | 1.4897e-06 | 5.0405e-07 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.teardown | 1.4398e-06 | 1 | 1.4398e-06 | 4.8719e-07 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_fit_end | 1.43e-06 | 1 | 1.43e-06 | 4.8388e-07 |
| [LightningModule]OptiSpeech.teardown | 1.4203e-06 | 1 | 1.4203e-06 | 4.8057e-07 |
| [Strategy]SingleDeviceStrategy.on_train_end | 1.37e-06 | 1 | 1.37e-06 | 4.6356e-07 |
| [Callback]ModelCheckpoint{'monitor': 'epoch', 'mode': 'max', 'every_n_train_steps': 0, 'every_n_epochs': 5, 'train_time_interval': None}.on_validation_epoch_end | 1.3602e-06 | 1 | 1.3602e-06 | 4.6025e-07 |
| [LightningModule]OptiSpeech.on_validation_epoch_end | 1.2801e-06 | 1 | 1.2801e-06 | 4.3315e-07 |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[2024-09-12 16:25:04,266][optispeech.utils.generic][INFO] - Output dir: /root/home/optispeech/data/saspeech/logs/train/saspeech_he/runs/2024-09-12_16-20-05
[2024-09-12 16:25:04,267][optispeech.utils.generic][INFO] - Metric name is None! Skipping metric value retrieval...
root@C.12435007:~/home/optispeech$ tmux capture-pane -S - -E - -p > output.txt
Created
September 12, 2024 16:29
-
-
Save thewh1teagle/5ac171bc73b8215c1e13a6f4c9c4c652 to your computer and use it in GitHub Desktop.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment