Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
385 changes: 14 additions & 371 deletions install/elasticsearch.rst
Original file line number Diff line number Diff line change
@@ -1,381 +1,24 @@
Set Up Elasticsearch
====================

Zammad's search function can be powered by Elasticsearch (which is **highly
recommended**).
Elasticsearch
=============

.. include:: /install/includes/hosted-services.rst

.. note:: Some steps may be required depending on your Elasticsearch version and
configuration. See remarks in the configuration steps below.

Step 1: Installation
--------------------

Elasticsearch offers different versions. Currently, the versions 7, 8 and 9
are supported by Zammad. For installation instructions, you should first and
foremost follow
`Elastic's installation documentation <https://www.elastic.co/docs/deploy-manage/deploy/self-managed/installing-elasticsearch>`_.
Use the dropdown in the top right corner to choose which version you want to
install.

However, if you want go with Elasticsearch 7 (which is slightly easier to
install), you can find the consolidated installation steps below.
Be aware that the maintenance of version 7 might be stopped earlier than for
later versions, which also come with some additional security features.

.. hint:: If you are installing Elasticsearch 8 or newer and want to follow our
standard configuration in step 2, make sure to copy/save the password which
is shown while installing Elasticsearch.

.. tabs::

.. tab:: Ubuntu/Debian

.. code-block:: console

$ sudo apt install apt-transport-https sudo wget curl gnupg

.. code-block:: console

$ curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/elasticsearch.gpg> /dev/null

.. tabs::

.. tab:: Deb822 format

In this tab, the repository is added by using the
`deb822 format <https://repolib.readthedocs.io/en/latest/deb822-format.html>`_.
If you run a distribution which doesn't support it, use the legacy
format instead.

.. code-block:: console

$ printf "Types: deb
URIs: https://artifacts.elastic.co/packages/7.x/apt
Suites: stable
Components: main
Signed-By: /etc/apt/trusted.gpg.d/elasticsearch.gpg" | \
sudo tee /etc/apt/sources.list.d/elastic-7.x.sources > /dev/null

.. tab:: Legacy format

.. code-block:: console

$ echo "deb [signed-by=/etc/apt/trusted.gpg.d/elasticsearch.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main"| \
sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list > /dev/null

.. code-block:: console

$ sudo apt update

.. code-block:: console

$ sudo apt install elasticsearch

.. code-block:: console

$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

.. tab:: CentOS

.. code-block:: console

$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

.. code-block:: console

$ echo "[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md"| sudo tee /etc/yum.repos.d/elasticsearch-7.x.repo

.. code-block:: console

$ sudo yum install -y elasticsearch

.. code-block:: console

$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

.. tab:: OpenSUSE

.. code-block:: console

$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

.. code-block:: console

$ echo "[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md"| sudo tee /etc/zypp/repos.d/elasticsearch-7.x.repo

.. code-block:: console

$ sudo zypper install elasticsearch

.. code-block:: console

$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

.. tab:: Direct Download

Find the latest release on the
`downloads page <https://www.elastic.co/downloads/elasticsearch>`_,
or see the
`installation guide <https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_
for in-depth instructions. Ensure to also install the fitting
(and mandatory!) attachment plugin for Elasticsearch, if installing
version 7.

Install the attachment plugin:

.. code-block:: console

$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

Increase the virtual memory map limit:

.. code-block:: console

$ sudo sysctl -w vm.max_map_count=262144

After you installed Elasticsearch and its attachment plugin,
ensure to enable it by default and start it.

.. code-block:: console

$ sudo systemctl start elasticsearch

.. code-block:: console

$ sudo systemctl enable elasticsearch

Step 2: Configuration
---------------------

Install ingest-plugin (only for Elasticsearch <= 7)
.. code-block:: console

$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

Increase virtual memory map limit
.. code-block:: console

$ sudo sysctl -w vm.max_map_count=262144

Adjust ``/etc/elasticsearch/elasticsearch.yml``
We use the following settings to optimize the performance of our Elasticsearch
servers. You may want to append that to your ``elasticsearch.yml`` as a useful
basic configuration.

.. code-block:: text

# /etc/elasticsearch/elasticsearch.yml

# Tickets above this size (articles + attachments + metadata)
# may fail to be properly indexed (Default: 100mb).
#
# When Zammad sends tickets to Elasticsearch for indexing,
# it bundles together all the data on each individual ticket
# and issues a single HTTP request for it.
# Payloads exceeding this threshold will be truncated.
#
# Performance may suffer if it is set too high.
http.max_content_length: 400mb

# Allows the engine to generate larger (more complex) search queries.
# Elasticsearch will raise an error or deprecation notice if this value is too low,
# but setting it too high can overload system resources (Default: 1024).
#
# Available in version 6.6+ only.
indices.query.bool.max_clause_count: 2000

Enable and start Elasticsearch
.. code-block:: console

$ sudo systemctl start elasticsearch

.. code-block:: console

$ sudo systemctl enable elasticsearch

.. _configure_zammad_with_elasticsearch:

Step 3: Connecting Zammad with Elasticsearch
--------------------------------------------

Before proceeding here, make sure to install Zammad before running below
commands, as this will fail otherwise.

* Install from :doc:`package <package>`
* Install with :doc:`Docker <docker-compose>` (the default stack already includes Elasticsearch)

.. note::
This guide uses the ``zammad run`` command prefix in command line examples.
This prefix is only applicable to package installations.
Check the :doc:`console guide </admin/console>` for more information.

Elasticsearch URL
Set the Elasticsearch server address; adapt it to your scenario.

Elasticsearch 7:

.. code-block:: console

$ zammad run rails r "Setting.set('es_url', 'http://localhost:9200')"

Elasticsearch 8+:

.. code-block:: console

$ zammad run rails r "Setting.set('es_url', 'https://localhost:9200')"


Elasticsearch user and password (only for Elasticsearch >= 8)
Now you need your password which was shown during the installation.

Set Elasticsearch user:

.. code-block:: console

$ zammad run rails r "Setting.set('es_user', 'elastic')"

Set Elasticsearch password:

.. code-block:: console

$ zammad run rails r "Setting.set('es_password', '<password>')"

Add certificate to Zammad (only for Elasticsearch >= 8)
Show and copy the auto-generated certificate from Elasticsearch and add it
to Zammad. Make sure to copy/paste the delimiters
(e.g. ``-----BEGIN CERTIFICATE-----``) too.

.. code-block:: console

$ sudo cat /etc/elasticsearch/certs/http_ca.crt

Go to the admin panel of Zammad and add your copied certificate under
:admin-docs:`SSL Certificates </settings/security/ssl-certificates.html>`.

.. figure:: /images/install/elasticsearch/admin-certificate-management.png
:alt: Screenshot shows certificate management in Zammad's admin panel
:align: center

.. _es-rebuild-searchindex:

Build/rebuild the searchindex
.. hint::
- The rebuild may take many hours or even days, if a lot of data is
already present in a productive environment. However, you can safely
run this during operating times without the risk of loosing data. As a
downside, it could lead to reduced performance and that some data may
not be shown in search results.
- Consider specifying a number of CPU cores to be used for the rebuild
(see example below).

Without specifying CPU cores:

.. code-block:: console

$ zammad run rake zammad:searchindex:rebuild

Example with specifying 8 CPU cores:

.. code-block:: console

$ zammad run rake zammad:searchindex:rebuild[8]


Optional settings
-----------------

We collected some useful settings you may want to apply. For further
information please have a look at
`Elastic's documentation <https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_.

.. tabs::

.. tab:: Index namespacing

Useful when connecting multiple services or Zammad instances
to a single Elasticsearch server (to prevent name collisions during indexing).

.. code-block:: console

$ zammad run rails r "Setting.set('es_index', Socket.gethostname.downcase + '_zammad')"

.. tab:: File-attachment indexing rules

Zammad supports searching by the contents of file attachments, which means
Elasticsearch has to index those, too. Limiting such indexing can help
conserve system resources.

Files with these extensions will not be indexed:

.. code-block:: console

$ zammad run rails r "Setting.set('es_attachment_ignore',\
[ '.png', '.jpg', '.jpeg', '.mpeg', '.mpg', '.mov', '.bin', '.exe', '.box', '.mbox' ] )"

Files larger than this size (in MB) will not be indexed:

.. code-block:: console

$ zammad run rails r "Setting.set('es_attachment_max_size_in_mb', 50)"

.. tab:: Remote host

Change your Elasticsearch URL if you have a separate Elasticsearch server.
Default is ``localhost``.

.. code-block:: console

$ zammad run rails r "Setting.set('es_url', 'https://example.com')"

.. tab:: SSL Verification

You can disable SSL verification, which is not recommended. Default is
``true``.

.. code-block:: console

$ zammad run rails r "Setting.set('es_ssl_verify', false)"

If you want to use custom certificates, you can find information about
how to use them in Zammad
:admin-docs:`here </settings/security/ssl-certificates.html>`.

.. tab:: Asciifold

By default, the
`Asciifold <https://www.elastic.co/docs/reference/text-analysis/analysis-asciifolding-tokenfilter>`_
feature of Elasticsearch is enabled. This can be useful if you deal with
text which includes diacritics and/or umlauts.

In case you need a more exact search, you can turn it off:
Zammad's search function can be powered by Elasticsearch (which is **highly
recommended**).

.. code-block:: console
The order for a package installation of Zammad is like this:

$ zammad run rails r "Setting.set('es_asciifolding', false)"
1. :doc:`Install Elasticsearch (version 9 recommended) <elasticsearch/install-elasticsearch-9>`
2. :doc:`Install Zammad <./package>`
3. :doc:`Connect and configure Elasticsearch <elasticsearch/connect-configure-elasticsearch>`

Appendix
--------
.. hint:: For a transition period, there is also a
:doc:`installation guide for Elasticsearch 7 <elasticsearch/install-elasticsearch-7>` available.

.. toctree::
:maxdepth: 0
:titlesonly:
:hidden:

elasticsearch/indexed-attributes
elasticsearch/troubleshooting
elasticsearch/install-elasticsearch-7
elasticsearch/install-elasticsearch-9
elasticsearch/connect-configure-elasticsearch
Loading