Support SpeechT5 text-to-speech pipeline by OpenVINO

[rkazants]

What does this PR do?

This PR introduces support of SpeechT5 text-to-speech pipeline using OpenVINO. Here is a demo code:

import soundfile as sf
import torch
from datasets import load_dataset
from optimum.intel import OVModelForTextToSpeechSeq2Seq
from transformers import SpeechT5Processor

model_id = "microsoft/speecht5_tts"
vocoder_id = "microsoft/speecht5_hifigan"

ov_pipe = OVModelForTextToSpeechSeq2Seq.from_pretrained(model_id, export=True, vocoder=vocoder_id)
ov_pipe.save_pretrained("speecht5_tts")
ov_pipe = OVModelForTextToSpeechSeq2Seq.from_pretrained("speecht5_tts")

processor = SpeechT5Processor.from_pretrained(model_id)

inputs = processor(text="Hello, this PR introduces support of SpeechT5 text-to-speech pipeline using OpenVINO.",
                   return_tensors="pt")

# load vector containing speaker's voice characteristics from a dataset
embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")
speaker_embeddings = torch.tensor(embeddings_dataset[7306]["xvector"]).unsqueeze(0)

speech = ov_pipe.generate(input_ids=inputs["input_ids"],
                          speaker_embeddings=speaker_embeddings)

sf.write("speech.wav", speech.numpy()[0], samplerate=16000)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[eaidova]

@rkazants could you please provide tests?

[eaidova]

@rkazants please update also import structure for added model classes:
https://github.com/huggingface/optimum-intel/blob/main/optimum/intel/__init__.py#L163
https://github.com/huggingface/optimum-intel/blob/main/optimum/intel/utils/dummy_openvino_objects.py

[rkazants]

@rkazants please update also import structure for added model classes: https://github.com/huggingface/optimum-intel/blob/main/optimum/intel/__init__.py#L163 https://github.com/huggingface/optimum-intel/blob/main/optimum/intel/utils/dummy_openvino_objects.py

Updated

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[rkazants]

@rkazants could you please provide tests?

Done

[rkazants]

@eaidova, @IlyasMoutawwakil, @echarlaix, could you please review PR?

Thanks,
Roman

[rkazants]

CI failures does not relate to my changes in PR. For example, I see issue with whisper model. Please correct me if I am wrong.

[eaidova]

@IlyasMoutawwakil could you please rerun ci, thanks

[IlyasMoutawwakil]

@eaidova done, was this issue fixed ?

FAILED tests/openvino/test_modeling.py::OVModelForTextToSpeechSeq2SeqIntegrationTest::test_compare_to_transformers_0_speecht5 - RuntimeError: The size of tensor a (512) must match the size of tensor b (32) at non-singleton dimension 1

[eaidova]

@eaidova done, was this issue fixed ?

FAILED tests/openvino/test_modeling.py::OVModelForTextToSpeechSeq2SeqIntegrationTest::test_compare_to_transformers_0_speecht5 - RuntimeError: The size of tensor a (512) must match the size of tensor b (32) at non-singleton dimension 1

@IlyasMoutawwakil thanks, yes @rkazants is working on the fix

[IlyasMoutawwakil]

LGTM, great addition !
I think there's still some redundancy / room to make the implementation leaner, for example with Whisper which has a custom generate method, we only make sure our class is compliant with its behavior and use the method directly from transformers.

[rkazants]

LGTM, great addition ! I think there's still some redundancy / room to make the implementation leaner, for example with Whisper which has a custom generate method, we only make sure our class is compliant with its behavior and use the method directly from transformers.

Responded here #1230 (comment)
Thanks

[echarlaix]

left couple of minor comments, good to merge once resolved