-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Add dummy beam wrapper #1951
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Add dummy beam wrapper #1951
Conversation
@Conchylicultor Please have a look on changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can consider making these changes just for now.
However, after #1947 is merged, I think it would be best to inherit FakeModule
rather than ModuleType
.
It will avoid duplication of code
def __getattribute__(self, _): | ||
if getattr(DummyBeam, _, None) is Empty: | ||
_name = super().__getattribute__('module_name') | ||
err_msg = _MSG.format(name=_name) | ||
raise ImportError(err_msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Define __getattr__
instead of __getattribute__
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
__getattr__
make some trouble here because we want DoFn
to call with DummyBeam so if use getattr
here this will executing without any errors here, the reason why I am using __getattribute__
is __getattribute__
calls no matter what unlike getattr
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should throw an error when DoFn
is accessed.
If beam isn't found, then the imports works, but calling SomeFn(), some_ptransform_fn(),... raise the lazy import error ('Please install appache_beam')
We don't want this to happen. So, we should not throw any errors when members of DummyBeam
are accessed.
However, proceed only after consulting @Conchylicultor.
(Just in case I am wrong 😅)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Beam API have DoFn and pipiline but it dont have SomeFn its just name i think thats why it inherit beam.DoFn so i think it is good to get error when invoking beam.DoFn 🙂
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vijayphoenix is right. The all point of this module is to allow using beam without raising error when declaring the beam functions. However error should be raised when the functions are called.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks but this won't works 🙂
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
However I found a way to do this using __new__
in Empty class and its working, when I call SomeFn() it raises error, is it necessary to use types.ModuleType
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You shouldn't have to use __new__
. __init__
should be enough.
What is the issue with types.ModuleType
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Modules should inherit from ModuleType
, but not other Class or functions.
class EmptyClass(object):
def __init__(self):
raise ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay then doing in this way ?
class Empty(object):
def __init__(self, *args, **kwargs):
err_msg = _MSG.format(name="apache_beam")
raise ImportError(err_msg)
class DummyBeam(types.ModuleType):
DoFn = Empty
Pipeline = Empty
ptransform = Empty
def __init__(self):
return None
what is use of types.ModuleType
here ?
import apache_beam as beam | ||
except ImportError: | ||
beam = dummy_beam.DummyBeam() | ||
with self.assertRaisesWithPredicateMatch(ImportError, _err_msg): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only add the with self.assertRaisesWithPredicateMatch(ImportError, _err_msg):
around the line which actually raise the exception.
Otherwise I don't know where the exception is coming from.
with self.assertRaisesWithPredicateMatch(ImportError, _err_msg): | ||
beam = dummy_beam.DummyBeam() | ||
|
||
class SomeFn(beam.DoFn): # pylint: disable=unused-variable, too-few-public-methods |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Creating the class should works, which is the entire point.
but instantiated the class should fail.:
class SomeDoFn(beam.DoFn): # Works
pass
with self.assertRaisesRegex(...):
SomeDoFn() # Raise error
def __getattribute__(self, _): | ||
if getattr(DummyBeam, _, None) is Empty: | ||
_name = super().__getattribute__('module_name') | ||
err_msg = _MSG.format(name=_name) | ||
raise ImportError(err_msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vijayphoenix is right. The all point of this module is to allow using beam without raising error when declaring the beam functions. However error should be raised when the functions are called.
Fix #1946