Labeling Function still got access to calculated attribute after it has been deleted

[FelixKirsch]

From #1:

When I add a attribute that is basically a copy of a given text (let's call that attribute text_copy), and i then write a lf using text_copy, e.g.:

def my_lf(record):
    if "x" in record["text_copy"].text:
        return "label_a"

that works. But now, when I delete text_copy, and run that function again - or even modify it to label_b, and then run it again - it still works. How do we store the records exactly? might be that there is some place left that is not touched by the deletion of an attribute?

From the user's perspective, this behavior seems unexpected. For the moment, I decided to keep calculated attributes in the docbin and record_tokenized, until the tokenization is restarted. Otherwise, the tokenization would have to be restarted and the user would have to wait until it's finished.

At the point in time, when attributes can be tokenized by themselves without tokenizing the whole project, it will most probably also be possible to delete only one attribute. Then this issue can be fixed.

[FelixKirsch]

Issue in tokenizer:
code-kern-ai/refinery-tokenizer#7

[JWittmeyer]

@FelixKirsch Can this easily be done by repushing the s3 data with the new endpoint introduced for the attribute calculation?

[FelixKirsch]

Yes, that should work. Triggering the endpoint after removing a user-created attribute should fix the issue.

[FelixKirsch]

The issue is fixed for non-text attributes. Text attributes still remain in the docbins and are accessible after deletion. This will be fixed in a later version.