Add malaria bbbc dataset #1025

powergkrry · 2019-09-24T23:54:00Z

No description provided.

googlebot · 2019-09-24T23:54:04Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

powergkrry · 2019-09-24T23:57:27Z

@googlebot I signed it!

…

On Tue, Sep 24, 2019 at 7:54 PM googlebot ***@***.***> wrote: Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 *Please visit https://cla.developers.google.com/ <https://cla.developers.google.com/> to sign.* Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it. ------------------------------ What to do if you already signed the CLA Individual signers - It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data <https://cla.developers.google.com/clas> and verify that your email is set on your git commits <https://help.github.com/articles/setting-your-email-in-git/>. Corporate signers - Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version <https://opensource.google.com/docs/cla/#troubleshoot>). - The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data <https://cla.developers.google.com/clas> and verify that your email is set on your git commits <https://help.github.com/articles/setting-your-email-in-git/>. - The email used to register you as an authorized contributor must also be attached to your GitHub account <https://github.com/settings/emails>. ℹ️ *Googlers: Go here <https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2Ftensorflow%2Fdatasets%2Fpull%2F1025> for more info*. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1025?email_source=notifications&email_token=AH5W2TZMQ7RGDMMBPVZGBRDQLKSCDA5CNFSM4I2GBS7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QEO5I#issuecomment-534792053>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH5W2T3DHDRXTORQLDAMTG3QLKSCDANCNFSM4I2GBS7A> .

googlebot · 2019-09-24T23:57:30Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

Ouwen

Overall looks good.

tensorflow_datasets/image/malaria_bbbc.py

us

The test data should not contain any copyrighted material.

tensorflow_datasets/image/__init__.py

us · 2019-09-27T07:49:02Z

tensorflow_datasets/image/malaria_bbbc.py

+            "label": tfds.features.ClassLabel(names=_NAMES),
+        }),
+        supervised_keys=("image", "label"),
+        urls=[_URL],


this url should be dataset home page link not download link.

@us The dataset has the following copyright access. Free to redistribute.
https://creativecommons.org/licenses/by-nc-sa/3.0/

tensorflow_datasets/image/malaria_bbbc_test.py

Ouwen

Tests pass, but dataset server is very slow... would be ideal to move into gcp. @powergkrry can you give this a sanity check via the colab notebook below?

https://colab.research.google.com/drive/1ozOwyvehz-XUeu9JLxhZLUYVstKuxylR

tensorflow_datasets/image/malaria_bbbc.py

Ouwen · 2019-09-27T20:52:28Z

tensorflow_datasets/image/malaria_bbbc.py

+import tensorflow as tf
+import tensorflow_datasets.public_api as tfds
+
+_URL = "http://people.duke.edu/~kk349/bbbc.zip"


@us @Conchylicultor what is the process for adding datasets like this into the tfds bucket? The download from these personal duke servers is around 3MB/s unfortunately.

Unfortunately, for legal reason, we are not allowed to host the datasets ourselves. Only the original author of the datasets can host it on a faster server.

@powergkrry you are the original author of this dataset correct?

Hi Ouwen,
We are not the original authors of the dataset. The data is obtained from https://data.broadinstitute.org/bbbc/BBBC041/
We are free to copy and redistribute the data though.

Ouwen · 2019-10-02T17:03:01Z

tensorflow_datasets/image/malaria_bbbc.py

+consists of two classes of uninfected cells(red blood cells and leukocytes) and
+four classes of infected cells(gametocytes, rings, schizonts, and trophozoites).
+The Malaria dataset contains a total of 74,238 cell images and we have removed 441 cell images
+because annotators marked the cells as difficult as it was not clearly in one of the cell classes.


max lines should be PEP8 compliant 72 preferred 79 max chars

Ouwen

https://colab.research.google.com/drive/1ozOwyvehz-XUeu9JLxhZLUYVstKuxylR

Tests pass dataset loads. Unfortunate that the dataset download is slow @powergkrry. Consider using Duke's data hosting service: https://research.repository.duke.edu/

Make the minor documentation changes.

Conchylicultor · 2019-11-04T18:14:36Z

Hi @powergkrry, this doesn't seems to be an official dataset.
Otherwise, could you provide the official homepage of this dataset ?

If the data is from https://data.broadinstitute.org/bbbc/index.html, I'm not sure I understand why you're not using the official data from the website, which contains 2GB of data (https://data.broadinstitute.org/bbbc/BBBC041/malaria.zip), vs 600MB for this one.
Similarly, the original data provide both train and test set, while this data only provide test set.

TFDS should redistribute the datasets as the original dataset was, otherwise users may be misslead and train their model on wrong data.

version_1

7d0dfff

googlebot added the cla: no Author has not signed CLA label Sep 24, 2019

googlebot added cla: yes Author has signed CLA and removed cla: no Author has not signed CLA labels Sep 24, 2019

powergkrry changed the title ~~version_1~~ Add malaria bbbc dataset Sep 25, 2019

Ouwen reviewed Sep 25, 2019

View reviewed changes

tensorflow_datasets/image/malaria_bbbc.py Outdated Show resolved Hide resolved

tensorflow_datasets/image/malaria_bbbc.py Outdated Show resolved Hide resolved

powergkrry added 2 commits September 24, 2019 20:42

version_1.1

0c337ba

version_1.1

66d1a2b

us suggested changes Sep 27, 2019

View reviewed changes

tensorflow_datasets/image/malaria_bbbc_test.py Outdated Show resolved Hide resolved

Ouwen suggested changes Sep 27, 2019

View reviewed changes

version_1.2

d476a72

Ouwen reviewed Oct 2, 2019

View reviewed changes

Ouwen suggested changes Oct 2, 2019

View reviewed changes

Conchylicultor added the dataset request Request for a new dataset to be added label Oct 15, 2019

cyfra added the being_merged label Nov 4, 2019

cyfra added cannot_merge:under review and removed being_merged labels Nov 4, 2019

cyfra added the icebox No response from the author for at least 1 month. label Jan 3, 2020

cyfra added author:please_respond Author - please respond to the recent comments. and removed cannot_merge:under review labels Feb 8, 2020

Add malaria bbbc dataset #1025

Are you sure you want to change the base?

Add malaria bbbc dataset #1025

Uh oh!

Conversation

powergkrry commented Sep 24, 2019

Uh oh!

googlebot commented Sep 24, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

powergkrry commented Sep 24, 2019 via email

Uh oh!

googlebot commented Sep 24, 2019

Uh oh!

Ouwen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

us left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

us Sep 27, 2019

Choose a reason for hiding this comment

Uh oh!

Dhanasekar-S Oct 3, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Ouwen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Ouwen Sep 27, 2019

Choose a reason for hiding this comment

Uh oh!

Conchylicultor Sep 30, 2019

Choose a reason for hiding this comment

Uh oh!

Ouwen Oct 2, 2019

Choose a reason for hiding this comment

Uh oh!

Dhanasekar-S Oct 3, 2019

Choose a reason for hiding this comment

Uh oh!

Ouwen Oct 2, 2019

Choose a reason for hiding this comment

Uh oh!

Ouwen left a comment

Choose a reason for hiding this comment

Uh oh!

Conchylicultor commented Nov 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants