-
Notifications
You must be signed in to change notification settings - Fork 21
Description
Some time ago an alternative approach to rank
ing operation was introduced:
involving custom rank operation written in rank.py
script introduced in 318d88c commit.
An alternative oozie execution path could be selected by enabling load_filterTerms_calcTfidf_filter_ship_ranked
flag.
This was a solution to memory related issues related to PIG embedded rank
operation. In fact this may have been caused by the very same reason as the one causing #425.
The thing is as soon as #425 is fixed and PIG embedded rank
operator works properly we can get rid of this alternative path.
It is useless anyway because it causes failure at later docsim stage. Probably both ranking related PIG scripts diverged at some point and an alternative one is not fully compliant with main one.