-
Notifications
You must be signed in to change notification settings - Fork 12
Description
[2023-08-26 09:23:35,166 main.py:283 INFO pretrain-language-model] Construct dataset.
/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
[2023-08-26 09:23:36,980 main.py:84 INFO pretrain-language-model] 102085 training items found.
[2023-08-26 09:23:36,980 main.py:86 INFO pretrain-language-model] 50000 valid items found.
[2023-08-26 09:23:36,980 main.py:289 INFO pretrain-language-model] Construct model.
[2023-08-26 09:23:41,078 main.py:292 INFO pretrain-language-model] Construct learner.
[2023-08-26 09:23:46,050 main.py:296 INFO pretrain-language-model] Start training.
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
[2023-08-26 09:23:46,569 callbacks.py:179 INFO pretrain-language-model] Train ended
[2023-08-26 09:23:46,569 callbacks.py:143 INFO pretrain-language-model] average data time = 0.0000s, average running time = 0.0000s
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastai/basic_train.py", line 99, in fit
for xb,yb in progress_bar(learn.data.train_dl, parent=pbar):
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 39, in iter
if self.total != 0: self.update(0)
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 59, in update
self.update_bar(0)
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 81, in update_bar
self.on_update(val, f'{pct}[{val}/{tot} {elapsed_t}{self.lt}{remaining_t}{end}]')
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 134, in on_update
elif self.parent is not None: self.parent.show()
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 177, in show
self.out.update(HTML(self.html_code))
AttributeError: 'NoneType' object has no attribute 'update'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/content/drive/.shortcut-targets-by-id/12Fpyqv7ac7VkBnm5mHDZrlJ-pRtYx53q/AIO/Competition_Module/BKAI/semimtr-text-recognition/main.py", line 306, in
main()
File "/content/drive/.shortcut-targets-by-id/12Fpyqv7ac7VkBnm5mHDZrlJ-pRtYx53q/AIO/Competition_Module/BKAI/semimtr-text-recognition/main.py", line 297, in main
learner.fit(epochs=config.training_epochs,
File "/usr/local/lib/python3.10/dist-packages/fastai/basic_train.py", line 200, in fit
fit(epochs, self, metrics=self.metrics, callbacks=self.callbacks+callbacks)
File "/usr/local/lib/python3.10/dist-packages/fastai/basic_train.py", line 112, in fit
finally: cb_handler.on_train_end(exception)
File "/usr/local/lib/python3.10/dist-packages/fastai/callback.py", line 323, in on_train_end
self('train_end', exception=exception)
File "/usr/local/lib/python3.10/dist-packages/fastai/callback.py", line 251, in call
for cb in self.callbacks: self._call_and_update(cb, cb_name, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fastai/callback.py", line 241, in call_and_update
new = ifnone(getattr(cb, f'on{cb_name}')(**self.state_dict, **kwargs), dict())
File "/content/drive/.shortcut-targets-by-id/12Fpyqv7ac7VkBnm5mHDZrlJ-pRtYx53q/AIO/Competition_Module/BKAI/semimtr-text-recognition/semimtr/callbacks/callbacks.py", line 180, in on_train_end
self._eval_model()
File "/content/drive/.shortcut-targets-by-id/12Fpyqv7ac7VkBnm5mHDZrlJ-pRtYx53q/AIO/Competition_Module/BKAI/semimtr-text-recognition/semimtr/callbacks/callbacks.py", line 146, in _eval_model
last_metrics = self._validate()
File "/content/drive/.shortcut-targets-by-id/12Fpyqv7ac7VkBnm5mHDZrlJ-pRtYx53q/AIO/Competition_Module/BKAI/semimtr-text-recognition/semimtr/callbacks/callbacks.py", line 62, in _validate
val_metrics = validate(self.learn.model, dl, self.loss_func, cb_handler)
File "/usr/local/lib/python3.10/dist-packages/fastai/basic_train.py", line 57, in validate
for xb,yb in progress_bar(dl, parent=pbar, leave=(pbar is not None)):
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 39, in iter
if self.total != 0: self.update(0)
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 59, in update
self.update_bar(0)
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 81, in update_bar
self.on_update(val, f'{pct}[{val}/{tot} {elapsed_t}{self.lt}{remaining_t}{end}]')
File "/usr/local/lib/python3.10/dist-packages/fastprogress/fastprogress.py", line 133, in on_update
if self.display: self.out.update(HTML(self.progress))
AttributeError: 'NoneType' object has no attribute 'update'