Parsing bytes doesn't work

[gjask]

In [1]: from tatsu import compile
   ...: 
   ...: parser = compile(r"""
   ...:     start       = text ;
   ...:     text        = { sentence }* ;
   ...:     sentence    = { word }+ { punctuation }* ;
   ...:     word        = /\w+/ ;
   ...:     punctuation = /[?!.,;\"']/ ;
   ...: """)
   ...: 

In [2]: parser.parse("Hello world!")
Out[2]: [(['Hello', 'world'], ['!'])]

In [3]: parser.parse(b"Hello world!")
Out[3]: [(['b'], ["'"]), (['Hello', 'world'], ['!', "'"])]

I hoped to be able to parse bytes instead of str because I am trying to use TatSu to parse file format that uses ASCII for data structure but also can contain strings in other encoding. The other encoding is specified in content of the file. So I should try to bytes.decode() these parts only after parsing what encoding is used.

I kinda expected to get TypeError or similar on this. But actual error behaviour is more interesting. It seem like TatSu parser tries to cast input data to str which in turn gets bytes.__repr__() return value instead of contents.

Would it be possible to implement accepting bytes as data for parsing? If not I guess there should be some kind of type check on input instead of blind cast to str which in turn leads to parsing python representation of incompatible types instead of data itself.

[apalala]

Let me think about this...

It may be solved by adding str() somewhere, but that would be a patch.

If someone could provide a pull request to deal with this, it would be ideal.

[apalala]

@gjask, a pull request to solve this issue would be welcome.

The introduction of Tokenizer during the development of the Python PEG parser was intended to make TatSu not asume anything about the input.

If writing your own Tokenizer solves the issue, then there would be nothing to change in Buffering for now.

[gjask]

@apalala I didn't look into this deep enough yet to estimate if that is easy change or not.

I work on SGF editor in my spare time (meaning I don't move very fast) and I use TatSu to parse an SGF files. For time being I just assume all files are an UTF-8 encoded. But I will likely need to face this sooner or later. The problem with SGF format is that while it appears to be a plain text format it actually needs to be parsed as binary format because it is defined to be ASCII with possibility of embedded strings in different encodings.

So when I need it I may revisit this and see if I am able to adapt TatSu or I need to switch to different parser or parser generator. But I am afraid it won't be as easy because python re requires bytes pattern to match bytes type so you also need to be able distinguish between str and bytes patterns in grammar.

[gjask]

It may be solved by adding str() somewhere, but that would be a patch.

It most certainly cannot be. You cannot cast bytes to str without bytes.decode(). That means you have to know encoding of such string if it even has any valid encoding. And not everything is a text anyway. If it was so easy then there would be no need to parse bytes in the first place.

[apalala]

@gjask, I you can't provide a pull request to fix this issue, please proved a pull request with unit tests that prove that it is present?