Added partial visualisation for Audio Datasets under tfds #1683

harshitadd · 2020-03-20T06:52:32Z

Added partial visualization support for audio datasets under tfds: Tested on LJspeech, Groove
Issue Link: 1528

Procedure:

Pick random samples from argument dataset
Trim the picked audio samples to include only the first 6 seconds of the audio
Use IPython to display the trimmed audio sample
Plot the corresponding audio waveforms.

Notebook demonstration: Link

indentation error

Bracket missing after raise value error

Eshan-Agarwal

@harshitadd Please have a look on some of the changes I requested if I am mistaken anywhere let me know.

Eshan-Agarwal · 2020-03-21T17:46:21Z

tensorflow_datasets/core/visualization.py

 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function


Please remove them as we drop use of python 2, so we don't need these codes

Internally, we are still supporting Python2, so it shouldn't be removed yet.

@Conchylicultor thanks for correction

Eshan-Agarwal · 2020-03-21T17:51:14Z

tensorflow_datasets/core/visualization.py

+
+

@harshitadd Please remove one line and check your code style by running ./oss_scripts/lint.sh tensorflow_datasets/image

Eshan-Agarwal · 2020-03-21T18:01:24Z

tensorflow_datasets/core/visualization.py

-    if not image_keys:
-      raise ValueError(
-          "Visualisation not supported for dataset `{}`. Was not able to "
-          "auto-infer image.".format(ds_info.name))
-


Why removing these, I think we should print a message if provided data is video or text. Maybe you can add param like audio key and use if not image_keys and not audio_keys then again check for if not image_keys so print this message else print shows error like visualization is supports only for audio and mage dataset

Eshan-Agarwal · 2020-03-21T18:04:14Z

tensorflow_datasets/core/visualization.py

@@ -66,17 +70,13 @@ def show_examples(ds_info, ds, rows=3, cols=3, plot_scale=3., image_key=None):
  plt = lazy_imports_lib.lazy_imports.matplotlib.pyplot

  if not image_key:


Also use if not image_key and not audio_key after adding audio key param and generate audio_keys using features_lib.Audio so that if any other dataset like video or text is passed it can print error

I tried this - feature_lib.Audio was returning None for both the tested datasets. Since that instantiation check was not working - I had to resort to this implementation. []Here) is the commit with that code - Perhaps you could point out the mistake?

Don't know why its not working , dataset you are trying for audio_keys have how many number of channels ? , Audio feature supports only 1-D values see

Just rechecked - I think I was missing something trying it earlier - the instantiation check is working (at least with the feature_lib.Audio) so I will add that. My bad.
Thanks for the code style check reminder - have done that as well and will make the required changes! 👍

Eshan-Agarwal · 2020-03-21T18:12:55Z

tensorflow_datasets/core/visualization.py

@@ -85,46 +85,82 @@ def show_examples(ds_info, ds, rows=3, cols=3, plot_scale=3., image_key=None):

    image_key = image_keys[0]


I think check first like if image_keys then only set image_key as if data is not image dataset image_key set to empty

Refer to comment 78 and code 79 - The check to run this code has been added ( only when the image instances are populated )

Ohh sorry its my mistake

Eshan-Agarwal · 2020-03-21T18:15:07Z

tensorflow_datasets/core/visualization.py

+    if not label_key:
+      logging.info("Was not able to auto-infer label.")
+
+    num_examples = rows * cols


Add a check here if image_key so that below code runs only if data is image dataset

Refer to code line 78 - This is all under if image keys: indicating that the code runs only when the image keys have been successfully generated.

Eshan-Agarwal · 2020-03-21T18:17:35Z

tensorflow_datasets/core/visualization.py

+    return fig
+
+  # if not image item instances 
+  if not image_key:


What if dataset is like video dataset then it tries to run this command also so change it to if audio_key after passing audio_key param

Agreed - As mentioned - The audio key instantiation check wasn't working ( commit linked above ) - I am working on resolving that, which should allow introducing video/text/audio based instance checks.

harshitadd · 2020-03-27T10:42:21Z

@Conchylicultor - I tried refactoring the code to plug it in the visualizer codebase but the audio_key extraction seems to be returning a null as a result of which the match() method fails to execute.
I have removed the write on disk dependency ( audio now displays directly from the np array ) and added the missing imports.

Conchylicultor

To check if a dataset has an audio feature, you can see on our catalog: For instance https://www.tensorflow.org/datasets/catalog/crema_d

It seems Groove is using tfds.features.Tensor instead of tfds.features.Audio, which sounds like a bug to me, we should upgrade groove to use audio feature. I'll open a bug for this:
https://www.tensorflow.org/datasets/catalog/groove#groovefull-16000hz

Conchylicultor · 2020-03-27T16:56:25Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+from tensorflow_datasets.core import features as features_lib
+from tensorflow_datasets.core import lazy_imports_lib
+from tensorflow_datasets.core.visualization import visualizer
+plt = lazy_imports_lib.lazy_imports.matplotlib.pyplot


This will raise ImportError for users not having installed matplotlib.
The goal of lazy import is to avoid non-essencial dependencies by importing within specific function instead of in global scope.

To check if a dataset has an audio feature, you can see on our catalog: For instance https://www.tensorflow.org/datasets/catalog/crema_d

It seems Groove is using tfds.features.Tensor instead of tfds.features.Audio, which sounds like a bug to me, we should upgrade groove to use audio feature. I'll open a bug for this:
https://www.tensorflow.org/datasets/catalog/groove#groovefull-16000hz

Edit - Thank you for your comments - A bug for the Groove 'Tensor' attribute will be helpful - #1741 : Just realized that you have generated the issue

The bug has been fixed so it should works if you are rebasing from master.

It does, Thanks! The auto inferring works now ( Link . I have tested it with Groove, crema_d, ljspeech.

Conchylicultor · 2020-03-27T16:57:34Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+from tensorflow_datasets.core.visualization import visualizer
+plt = lazy_imports_lib.lazy_imports.matplotlib.pyplot
+
+class AudioGridVisualizer(visualizer.Visualizer):


Could you format your code with https://www.tensorflow.org/datasets/add_dataset#5_check_your_code_style

(add docstring, new line before method declaration, correct docstring...)

Conchylicultor · 2020-03-27T17:03:28Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+
+
+import random
+import IPython.display


This will crash for all user not running in a non-ipython environement. This should not be imported in the global scope

Conchylicultor · 2020-03-27T17:06:19Z

tensorflow_datasets/core/visualization/audio_visualizer.py

@@ -0,0 +1,68 @@
+""" Audio Visualization


Could you add a test in show_example_test.py to make sure this works ? You can use https://docs.python.org/3/library/unittest.mock.html to make sure the AudioVisualizer is chosen.

I added a mock patch for the AudioVisualizer class and get the following type error :
TypeError: test_show_examples() takes 2 positional arguments but 3 were given

Ran 2 tests in 0.557s

FAILED (errors=1, skipped=1)

Conchylicultor · 2020-03-27T17:23:28Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+  ):
+    """Display the dataset.
+
+      Args:


Arguments are not matching

Conchylicultor · 2020-03-27T17:24:00Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+        image_key: `string`, name of the feature that contains the image. If not
+          set, the system will try to auto-detect it.
+      """
+    key = audio_keys[0]


Where is it declared ?

Conchylicultor · 2020-03-28T18:53:04Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+        image_key: `string`, name of the feature that contains the image. If not
+          set, the system will try to auto-detect it.
+      """
+    import random


Random is a standard Python module, so installed in every users, so it is safe to keep it in the global scope

Conchylicultor

Thanks for the update. This looks better.

Conchylicultor · 2020-04-03T18:08:49Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+    key = audio_keys[0]
+    audio_samples = []
+
+    samplerate = 16000


Now you can use ds_info.features[key].sample_rate when defined (and use default value if sample_rate is None)

Conchylicultor · 2020-04-03T18:13:10Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+      audio = audio[t1:t2]
+      ipd.display(ipd.Audio(audio, rate=samplerate))
+    plt = lazy_imports_lib.lazy_imports.matplotlib.pyplot
+    fig, a = plt.subplots(2, 2)


To make the code more generic, you could refactor the code to use rows and cols, similarly to https://github.com/tensorflow/datasets/blob/b5d7ec65b84aa95e7f5e78f01f7698958498f65c/tensorflow_datasets/core/visualization/image_visualizer.py

And ideally, you could try to reuse the make_grid function

datasets/tensorflow_datasets/core/visualization/image_visualizer.py

Line 31 in b5d7ec6

def _make_grid(plot_single_ex_fn, ds, rows, cols, plot_scale):

Or make a similar util function.

All Right! I made some edits accordingly

Conchylicultor · 2020-04-03T18:13:38Z

tensorflow_datasets/core/visualization/audio_visualizer.py

+      ds,
+      rows=2,
+      cols=2,
+      plot_scale=3.,


Some of the arguments here are unused.

Conchylicultor · 2020-04-03T19:27:11Z

tensorflow_datasets/core/visualization/show_examples_test.py

  @mock.patch('matplotlib.pyplot.figure')
-  def test_show_examples(self, mock_fig):
+  @mock.patch('audio_visualizer.AudioGridVisualizer')


The goal is to test AudioGridVisualizer to make sure it don't crash so it shouldn't be patched. Only the low level functions, like IPython.display might be patched.

harshitadd added 23 commits March 18, 2020 17:51

Checking for audio key generation

313b571

1

cb417fc

2

5489a5f

3

f70cf46

5

ebf78db

Update visualization.py

8075721

6

8fc8ddf

7

02bc9e2

8

5242458

9

f801b41

10

04af5b2

10

d84455d

Update visualization.py

581620d

1.1

8791219

1.2

17b9188

indentation error

1.2

a5dc660

Bracket missing after raise value error

adding null check on image key

df53df0

unexpected indent

15550dc

audio key instancing not working

cfb259b

imported some headers

f01622e

taking the 1st 20 ms of the audio

0378ca4

Update visualization.py

e95ded9

Update visualization.py

42df323

googlebot added the cla: yes Author has signed CLA label Mar 20, 2020

tfds-bot added the community:please_review Community - We need your help to review this PR. label Mar 20, 2020

Eshan-Agarwal suggested changes Mar 21, 2020

View reviewed changes

tfds-bot added author:please_respond Author - please respond to the recent comments. tfds:is_reviewing TFDS team: PTAL and removed community:please_review Community - We need your help to review this PR. author:please_respond Author - please respond to the recent comments. labels Mar 21, 2020

harshitadd added 7 commits March 27, 2020 14:40

bug: audio_keys returning NULL

314ee0c

_

344f8d3

Updated init.py to include AudioGridVisualizer

8df4e9d

_

1e12f8d

Added missing imports

bb07fa2

Fixed formatting

817dcfa

Fixed formatting

7af71b3

Conchylicultor requested changes Mar 27, 2020

View reviewed changes

tfds-bot added author:please_respond Author - please respond to the recent comments. and removed tfds:is_reviewing TFDS team: PTAL labels Mar 27, 2020

Local scope, declaration of audio_keys

26fd653

Conchylicultor requested changes Mar 28, 2020

View reviewed changes

tfds-bot added tfds:is_reviewing TFDS team: PTAL author:please_respond Author - please respond to the recent comments. and removed author:please_respond Author - please respond to the recent comments. tfds:is_reviewing TFDS team: PTAL labels Apr 1, 2020

harshitadd and others added 2 commits April 3, 2020 15:17

Merge branch 'master' of https://github.com/tensorflow/datasets

3e99ff0

Class docstrings and code style

ac73594

tfds-bot added tfds:is_reviewing TFDS team: PTAL and removed author:please_respond Author - please respond to the recent comments. labels Apr 3, 2020

Adding mock patch for AudioVisualizer

b5d7ec6

Conchylicultor requested changes Apr 3, 2020

View reviewed changes

tfds-bot added author:please_respond Author - please respond to the recent comments. and removed tfds:is_reviewing TFDS team: PTAL labels Apr 3, 2020

Conchylicultor requested changes Apr 3, 2020

View reviewed changes

harshitadd added 2 commits April 8, 2020 12:56

Supporting generic inputs

cc487e9

Removed unused imports

801987f

tfds-bot added tfds:is_reviewing TFDS team: PTAL and removed author:please_respond Author - please respond to the recent comments. labels Apr 8, 2020

		@@ -66,17 +70,13 @@ def show_examples(ds_info, ds, rows=3, cols=3, plot_scale=3., image_key=None):
		plt = lazy_imports_lib.lazy_imports.matplotlib.pyplot

		if not image_key:

		@@ -85,46 +85,82 @@ def show_examples(ds_info, ds, rows=3, cols=3, plot_scale=3., image_key=None):

		image_key = image_keys[0]

Added partial visualisation for Audio Datasets under tfds #1683

Are you sure you want to change the base?

Added partial visualisation for Audio Datasets under tfds #1683

Uh oh!

Conversation

harshitadd commented Mar 20, 2020

Uh oh!

Eshan-Agarwal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eshan-Agarwal Mar 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshitadd commented Mar 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Conchylicultor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshitadd Apr 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshitadd Apr 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eshan-Agarwal Mar 22, 2020 •

edited

Loading

harshitadd commented Mar 27, 2020 •

edited

Loading

harshitadd Apr 1, 2020 •

edited

Loading

harshitadd Apr 3, 2020 •

edited

Loading