TFLite on GPUs

[duongkstn]

Hi,
your notebook TFLiteExperiments.ipynb are awesome, I can run it in my CPU (tensorflow 2.3.1 or 2.4.1 is ok) without any bugs.
But have you tried:

• run that notebook on a single GPU ? I tried but it failed as following:

INFO: TfLiteFlexDelegate delegate: 517 nodes delegated out of 517 nodes with 1 partitions.

Segmentation fault (core dumped)

• run on CPU and GPU with input is a batch ?
• run on CPU and GPU with input is a batch but with dynamic sequence length ?

[bhadreshpsavani]

Hi,

Thanks for letting me know,

Yesterday only I come across this thing!
Earlier when I run it on GPU, it was working fine. Because of some recent changes in Transformers I think this bug is introduced.

I will fix the transformer version in the colab notebook and it will be resolve.

[duongkstn]

By the way, I am using transformers==3.5.1 in CPU and GPU. Your code works with tensorflow 2.3.1 and transformers 3.5.1 in CPU so I don't think version is an issue

[bhadreshpsavani]

Ya, you are right!

I just tried in the colab notebook recently but it was crushing at inference, I will check and let you know if I found some solutions for running inference on GPU issue. Please also let me know If you discover any solution