_score values for predicting similar news articles

[sudo2012]

Let's assume I got a set of news articles in my ES store. Is there a way to use MinHash score value to check a new article if it fits to any article in ES. So what I want to acchieve is the following: let's assume there are 2 articles on the same subject. One is from MSNBC and the other from TheGuardian. How can I recognize by the score value that they represent the same subject?

[marevol]

Using "copy_bits_to" parameter, you can copy minhash value as visible bits string(ex. 01011...), and then I think that similar documents are found with term or fuzzy query.

https://github.com/codelibs/elasticsearch-minhash/blob/master/src/test/java/org/codelibs/elasticsearch/minhash/MinHashPluginTest.java#L99

[sudo2012]

Thank you for your answer. I'll try that asap and get back to u. Best

[sudo2012]

Is it correct that I would create the mapping like that:

$ curl -XPUT "localhost:9200/my_index/my_type/_mapping" -d '{
"my_type":{
"properties":{
"message":{
"type":"string",
"copy_to":"minhash_value"
},
"minhash_value":{
"type":"minhash",
"minhash_analyzer":"minhash_analyzer",
"copy_bits_to" : "minhash_bits"
},
"minhash_bits":{
"type":"string"
}
}
}
}'

[marevol]

If you want to make "minhash_bits" value visible, it's better to add "store":true.

[Fred12]

Hello,
I tried copy_bits_to operator, but it does not show me the created field:
PUT /test_minhash_test
{
"index":{
"analysis":{
"analyzer":{
"minhash_analyzer":{
"type":"custom",
"tokenizer":"standard",
"filter":["minhash"]
}
}
}
}
}

PUT /test_minhash_test/_doc/_mapping
{
"_doc":{
"properties":{
"message": {
"type":"text",
"copy_to":"minhash_value"
},
"minhash_value":{
"type":"minhash",
"minhash_analyzer":"minhash_analyzer",
"store":true,
"copy_bits_to": "content_minhash_bits"
},
"content_minhash_bits": {
"type": "keyword",
"store":true
}
}
}
}

GET /test_minhash_test/_doc/_search/?pretty&stored_fields=*