Skip to content

Add dummy beam wrapper #1951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

Eshan-Agarwal
Copy link
Contributor

@Eshan-Agarwal Eshan-Agarwal commented Apr 25, 2020

Fix #1946

@googlebot googlebot added the cla: yes Author has signed CLA label Apr 25, 2020
@tfds-bot tfds-bot added the community:please_review Community - We need your help to review this PR. label Apr 25, 2020
@tfds-bot tfds-bot added author:please_respond Author - please respond to the recent comments. and removed community:please_review Community - We need your help to review this PR. labels Apr 25, 2020
@tfds-bot tfds-bot added tfds:is_reviewing TFDS team: PTAL and removed author:please_respond Author - please respond to the recent comments. labels Apr 26, 2020
@Eshan-Agarwal
Copy link
Contributor Author

@Conchylicultor Please have a look on changes.

Copy link
Contributor

@vijayphoenix vijayphoenix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can consider making these changes just for now.
However, after #1947 is merged, I think it would be best to inherit FakeModule rather than ModuleType.
It will avoid duplication of code

Comment on lines +40 to +44
def __getattribute__(self, _):
if getattr(DummyBeam, _, None) is Empty:
_name = super().__getattribute__('module_name')
err_msg = _MSG.format(name=_name)
raise ImportError(err_msg)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define __getattr__ instead of __getattribute__.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

__getattr__ make some trouble here because we want DoFn to call with DummyBeam so if use getattr here this will executing without any errors here, the reason why I am using __getattribute__ is __getattribute__ calls no matter what unlike getattr

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should throw an error when DoFn is accessed.

If beam isn't found, then the imports works, but calling SomeFn(), some_ptransform_fn(),... raise the lazy import error ('Please install appache_beam')

We don't want this to happen. So, we should not throw any errors when members of DummyBeam are accessed.

However, proceed only after consulting @Conchylicultor.
(Just in case I am wrong 😅)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beam API have DoFn and pipiline but it dont have SomeFn its just name i think thats why it inherit beam.DoFn so i think it is good to get error when invoking beam.DoFn 🙂

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vijayphoenix is right. The all point of this module is to allow using beam without raising error when declaring the beam functions. However error should be raised when the functions are called.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks but this won't works 🙂

Copy link
Contributor Author

@Eshan-Agarwal Eshan-Agarwal Apr 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However I found a way to do this using __new__ in Empty class and its working, when I call SomeFn() it raises error, is it necessary to use types.ModuleType ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't have to use __new__. __init__ should be enough.

What is the issue with types.ModuleType ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modules should inherit from ModuleType, but not other Class or functions.

class EmptyClass(object):
  def __init__(self):
    raise ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay then doing in this way ?

class Empty(object):
   
    def __init__(self, *args, **kwargs):
        err_msg = _MSG.format(name="apache_beam")
        raise ImportError(err_msg)
        
    
class DummyBeam(types.ModuleType):

    DoFn = Empty
    Pipeline = Empty
    ptransform = Empty

    def __init__(self):
        return None

what is use of types.ModuleType here ?

import apache_beam as beam
except ImportError:
beam = dummy_beam.DummyBeam()
with self.assertRaisesWithPredicateMatch(ImportError, _err_msg):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only add the with self.assertRaisesWithPredicateMatch(ImportError, _err_msg): around the line which actually raise the exception.
Otherwise I don't know where the exception is coming from.

with self.assertRaisesWithPredicateMatch(ImportError, _err_msg):
beam = dummy_beam.DummyBeam()

class SomeFn(beam.DoFn): # pylint: disable=unused-variable, too-few-public-methods
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creating the class should works, which is the entire point.
but instantiated the class should fail.:

class SomeDoFn(beam.DoFn):  # Works
  pass

with self.assertRaisesRegex(...):
  SomeDoFn()  # Raise error

Comment on lines +40 to +44
def __getattribute__(self, _):
if getattr(DummyBeam, _, None) is Empty:
_name = super().__getattribute__('module_name')
err_msg = _MSG.format(name=_name)
raise ImportError(err_msg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vijayphoenix is right. The all point of this module is to allow using beam without raising error when declaring the beam functions. However error should be raised when the functions are called.

@tfds-bot tfds-bot added author:please_respond Author - please respond to the recent comments. tfds:is_reviewing TFDS team: PTAL and removed tfds:is_reviewing TFDS team: PTAL author:please_respond Author - please respond to the recent comments. labels Apr 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes Author has signed CLA tfds:is_reviewing TFDS team: PTAL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[GSoC] Add a dummy beam wrapper
5 participants