Fixed problem with indexing a multilanguage field larger than 32k

Added property mapping option to specify the mapping of the 'default' field in multilanguage properties

Author: pmishev

Annotation/Property.php

@@ -8,6 +8,7 @@
  * Annotation used to check mapping type during the parsing process.
  *
  * @Annotation
+ *
  * @Target("PROPERTY")
  */
 final class Property implements DumperInterface
@@ -35,6 +36,13 @@ final class Property implements DumperInterface
      */
     public $multilanguage;
 
+    /**
+     * Override mapping for the 'default' language field of multilanguage properties
+     *
+     * @var array
+     */
+    public $multilanguageDefaultOptions;
+
     /**
      * The object name must be defined, if type is 'object' or 'nested'
      *

Mapping/DocumentParser.php

@@ -130,7 +130,7 @@ public function getPropertiesMetadata(\ReflectionClass $documentReflection)
             $propertyAnnotation = $propertyAnnotation ?: $this->reader->getPropertyAnnotation($property, Score::class);
 
             // Ignore class properties without any recognized annotation
-            if ($propertyAnnotation === null) {
+            if (null === $propertyAnnotation) {
                 continue;
             }
 
@@ -231,10 +231,11 @@ private function getDocumentType(\ReflectionClass $documentReflection)
      *
      * @param \ReflectionProperty $property
      *
-     * @return Property
+     * @return Property|null
      */
     private function getPropertyAnnotationData($property)
     {
+        /** @var Property $annotation */
         $annotation = $this->reader->getPropertyAnnotation($property, Property::class);
 
         return $annotation;
@@ -332,10 +333,11 @@ private function getProperties(\ReflectionClass $documentReflection, array $inde
                 foreach ($this->languageProvider->getLanguages() as $language) {
                     $mapping[$propertyAnnotation->name.$this->languageSeparator.$language] = $this->getPropertyMapping($propertyAnnotation, $language, $indexAnalyzers);
                 }
-                // TODO: The application should decide whether it wants to use a default field at all and set its mapping on a global base (or per property?)
+                // TODO: The application should decide whether it wants to use a default field at all and set its mapping on a global base
                 // The custom mapping from the application should be set here, using perhaps some kind of decorator
-                $mapping[$propertyAnnotation->name.$this->languageSeparator.Property::DEFAULT_LANG_SUFFIX] = [
+                $mapping[$propertyAnnotation->name.$this->languageSeparator.Property::DEFAULT_LANG_SUFFIX] = $propertyAnnotation->multilanguageDefaultOptions ?: [
                     'type' => 'keyword',
+                    'ignore_above' => 256,
                 ];
             } else {
                 $mapping[$propertyAnnotation->name] = $this->getPropertyMapping($propertyAnnotation, null, $indexAnalyzers);

Resources/doc/i18n.md

@@ -37,6 +37,10 @@ That service must have a method returning an array of language codes that will b
      *  name="title",
      *  type="text",
      *  multilanguage=true,
+     *  multilanguageDefaultOptions={
+     *     "type":"text",
+     *     "index":false
+     *  },
      *  options={
      *      "analyzer":"{lang}_analyzer",
      *  }
@@ -46,6 +50,8 @@ That service must have a method returning an array of language codes that will b
 ```
 > Note the use of **{lang}** in the analyzer declaration. It is going to be replaced by a respective language code at runtime.
 
+Specifying `multilanguageDefaultOptions` is entirely optional. If it is specified, it will override the default mapping for the `title-default` field.
+
 * Then, you must make sure you have all language-specific analyzers defined in your index config:
 ```
     indices:

Resources/doc/mapping.md

@@ -83,6 +83,10 @@ Sometimes, you may have a field that is available in more than one language. Thi
      *  name="name",
      *  type="text",
      *  multilanguage=true,
+     *  multilanguageDefaultOptions={
+     *     "type":"text",
+     *     "index":false
+     *  },
      *  options={
      *      "analyzer":"{lang}_analyzer",
      *  }
@@ -92,7 +96,10 @@ Sometimes, you may have a field that is available in more than one language. Thi
 ```
 > Note the use of `{lang}` placeholder in the options.
 
-When you have a property definition like that, there will not be a field `name` in your index, but instead there will be `name-en`, `name-fr`, `name-de`, etc. where the suffixes are taken from the available languages in your application. You may also use the special `{lang}` placeholder in the options array, as often you would need to specify different analyzers, depending on the language. For more information on how that works, see [multilanguage support](i18n.md).
+When you have a property definition like that, there will not be a field `name` in your index, but instead there will be `name-en`, `name-fr`, `name-de`, etc. where the suffixes are taken from the available languages in your application. 
+There will also be a field `name-default`, whose default mapping of `type:keyword;ignore_above:256` you can optionally override by specifying alternative `multilanguageDefaultOptions`.
+
+You may also use the special `{lang}` placeholder in the options array, as often you would need to specify different analyzers, depending on the language. For more information on how that works, see [multilanguage support](i18n.md).
 
 ### Meta property annotations
 

Tests/Functional/Mapping/DocumentParserTest.php

@@ -166,6 +166,20 @@ public function testParse()
                     'ml_info-default' =>
                     [
                         'type' => 'keyword',
+                        'ignore_above' => 256,
+                    ],
+                    'ml_more_info-en' =>
+                    [
+                        'type' => 'text',
+                    ],
+                    'ml_more_info-fr' =>
+                    [
+                        'type' => 'text',
+                    ],
+                    'ml_more_info-default' =>
+                    [
+                        'type' => 'text',
+                        'index' => false,
                     ],
                     'pieces_count' =>
                     [
@@ -319,12 +333,19 @@ public function testParse()
                         'propertyAccess' => 1,
                     ],
                     'ml_info' =>
-                    [
-                        'propertyName' => 'mlInfo',
-                        'type' => 'text',
-                        'multilanguage' => true,
-                        'propertyAccess' => 1,
-                    ],
+                        [
+                            'propertyName' => 'mlInfo',
+                            'type' => 'text',
+                            'multilanguage' => true,
+                            'propertyAccess' => 1,
+                        ],
+                    'ml_more_info' =>
+                        [
+                            'propertyName' => 'mlMoreInfo',
+                            'type' => 'text',
+                            'multilanguage' => true,
+                            'propertyAccess' => 1,
+                        ],
                     'pieces_count' =>
                     [
                         'propertyName' => 'tokenPiecesCount',

Tests/app/fixture/Acme/BarBundle/Document/Product.php

@@ -96,6 +96,21 @@ class Product extends AbstractDocument
      */
     public $mlInfo;
 
+    /**
+     * @var MLProperty
+     *
+     * @ES\Property(
+     *  name="ml_more_info",
+     *  type="text",
+     *  multilanguage=true,
+     *  multilanguageDefaultOptions={
+     *     "type":"text",
+     *     "index":false,
+     *  }
+     * )
+     */
+    public $mlMoreInfo;
+
     /**
      * @var int
      *