elasticsearch.keystore generated at every run

[ml14tlc]

• 6.3.3:
• 4.10 and 5.5:
• RedHat 7.5 and CentOS 7.3:

Bug description

I'm using the secrets parameter to manage the file elasticsearch.keystore. Every time Puppet runs the file is recreated, even when the parameter purge_secrets is set to true. In particular, Puppet wants to remove keystore.seed. I suppose a new keystore is created each time, which implies the presence of keystore.seed, the entries in the config file are added and the temporary file is compared with the current one.

Besides that, would it be possible to 'unmanage' elasticsearch.keystore? Even if secrets is left to undefined and elasticsearch.keystore, Puppet replaces it with an "empty" one (except for the keystore.seed value). I'd prefer generate the file locally and the distribute it, rather than having passwords in the manifest.

[tylerjl]

Hmm, that behavior is definitely unexpected, the keystore provider should be intelligent enough to a) not try to create keystores and b) only detect values that you want in the keystore that it should change and leave others alone.

If you want to dig into the unexpected behavior a bit more to try and find the bug, I'd love to see an example manifest that demonstrates the behavior.

With regards to leaving the keystore unmanaged entirely, the module only manages the elasticsearch_keystore resource if you've set secrets (as you mention, if secrets is defined), so the file is coming back, it'd have to be happening from the elasticsearch executable or some other method since the module will only try to manage it if $elasticsearch::secrets or $secrets on the elasticsearch::instance defined type is set.

[ml14tlc]

Hi @tylerjl ,

thanks for the answer. It took me some time to reply since I tried to narrow it down but I couldn't.
I wrote a profile to configure my ES nodes, I'm attaching it together with the Hiera YAML configuration.
The ES node is configured correctly and starts correctly. The Puppet server is 4.10, the OS is RedHat 7.5. I restarted both the puppet agent and the server many times, but to no avail. At every run, Puppet wants to re-create elasticsearch.keystore

# puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for es02.my.domain
Info: Applying configuration version '6f922b05c65b0b247508445ba903230b45ce121e'
Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/File[/etc/elasticsearch/test-me/elasticsearch.keystore]/content:
Binary files /etc/elasticsearch/test-me/elasticsearch.keystore and /tmp/puppet-file20180917-3378-1ai65zg differ

Info: Computing checksum on file /etc/elasticsearch/test-me/elasticsearch.keystore
Info: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/File[/etc/elasticsearch/test-me/elasticsearch.keystore]: Filebucketed /etc/elasticsearch/test-me/elasticsearch.keystore to puppet with sum c2f3b3d894b6d72e9f132ee895379a44
Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/File[/etc/elasticsearch/test-me/elasticsearch.keystore]/content:

Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/File[/etc/elasticsearch/test-me/elasticsearch.keystore]/content: content changed '{md5}c2f3b3d894b6d72e9f132ee895379a44' to '{md5}8b9f898cba123c1cb27b390ff4754d7b'
Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/Elasticsearch_keystore[test-me]/settings: added: xpack.ssl.keystore.secure_password, xpack.ssl.truststore.secure_password removed: keystore.seedNotice: Applied catalog in 12.66 seconds

[tylerjl]

@ml14tlc thank you for the configuration examples, that's super helpful to look at the problem.

The configuration files coupled with your logs seem to make sense to me. Elasticsearch sets a seed value for the keystore, and because your configuration is set to purge secrets, since seed isn't present in your instance config's secrets parameter, the module tries to remove it.

Out of curiosity, does the module stop mutating the keystore if you change purge_secrets from true to false? I may just be misunderstanding something, since your latest comment suggests that you don't want the keystore to be created at all, but your configuration file is setting secrets (that require a keystore).

[ml14tlc]

Hi @tylerjl

• if I remove the option purge_secrets to false the keystore is still re-created at every run, of course the message is a bit different, but consistent (would have removed ...):

Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/Elasticsearch_keystore[test-me]/settings: added: xpack.ssl.keystore.secure_password, xpack.ssl.truststore.secure_password would have removed: keystore.seed, but purging is disabled

• If I leave the secrets hash out (i.e., according to my profile, it is set to the default value undef), then the keystore is re-created with only one entry, keystore.seed. It is not removed since the purge option is set to false. If I set it to true, then the module would remove it and leave the keystore empty.

You're right, my preferred solution would be to manage the keystore separately, in order to avoid plain text passwords in the configuration, but it is not working, since the module always overwrites the keystore, if I leave secrets unset. That's why I configured it. At least I have the right values in it and the cluster can start.

[tylerjl]

Hmm alright, we're getting a bit closer. You mention that the keystore is recreated if you don't define secrets anywhere - just to be clear, is the Puppet module/puppet apply emitting a message indicating that Puppet itself is recreating the keystore, or is the keystore coming into existing after the service starts up?

[ml14tlc]

Puppet is recreating it:

# puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for es01.my.domain
Info: Applying configuration version '6f922b05c65b0b247508445ba903230b45ce121e'
Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/File[/etc/elasticsearch/test-me/elasticsearch.keystore]/content:
Binary files /etc/elasticsearch/test-me/elasticsearch.keystore and /tmp/puppet-file20180917-68487-wb4cfi differ

Info: Computing checksum on file /etc/elasticsearch/test-me/elasticsearch.keystore
Info: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/File[/etc/elasticsearch/test-me/elasticsearch.keystore]: Filebucketed /etc/elasticsearch/test-me/elasticsearch.keystore to puppet with sum 509c9b9ddb32cc4ba5fdb38095dd4edd
Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/File[/etc/elasticsearch/test-me/elasticsearch.keystore]/content:

Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[test-me]/File[/etc/elasticsearch/test-me/elasticsearch.keystore]/content: content changed '{md5}509c9b9ddb32cc4ba5fdb38095dd4edd' to '{md5}135f17c8684ff52501ce2ccb6ad88135'
Notice: /Stage[main]/Profile::Db::Elasticsearch/Elasticsearch::Instance[corebanking]/Elasticsearch_keystore[corebanking]/settings: added: xpack.ssl.keystore.secure_password, xpack.ssl.truststore.secure_password would have removed

The node is up the whole time, I set restart_config_change to false

[tylerjl]

Alright, so I think I may have an idea what could be happening. I may need some additional help debugging this, so thank you in advance (and for your patience so far!) in helping to debug this to help the module get better.

When the module builds up your Elasticsearch instance's directory, it recursively copies in contents from /etc/elasticsearch/* in order to pull in default files dropped there by plugins, X-Pack, and so on (source code here). Note that the file resource selectively ignores a couple of files that will be provided on a per-instance basis, like elasticsearch.yml.

In your Puppet run output, the logs are explicitly mentioning File[/etc/elasticsearch/test-me/elasticsearch.keystore], but there is no explicit file resource for keystores in instance.pp, so my hunch is that this recursive copy may be to blame. When you run puppet, it'll copy the file at /etc/elasticsearch/elasticsearch.keystore into your instance directory (/etc/elasticsearch/test-me/elasticsearch.keystore). Then the elasticsearch_keystore resource changes any secrets, then at next run it'll re-copy in the keystore and do it all over again.

There's a workaround you could try for now, and if it works, that'll confirm my suspicion and I can commit a patch. Could you try adding the following resource to your Puppet manifest? It will amend the recursive directory copy to exclude the top-level keystore file (and will probably need to be near the end of your manifests):

File <| tag == 'elasticsearch_instance_configdir' |> {
  ignore +> "/etc/elasticsearch/elasticsearch.keystore"
}

[ml14tlc]

Hi @tylerjl ,

you got it, thanks, and thank you for your patience. I tried to apply your snippet of code, but probably I did in the wrong place. Anyhow, in /etc/elasticsearch/ there's a keystore. I didn't think about the folder, knowing that the module creates its own sub-directory for the instance configuration. I removed /etc/elasticsearch/elasticsearch.keystore and everything works as expected.

[tylerjl]

Great, thanks - I'm glad it's working. I'll need to do a little more experimenting, but this certainly seems like a bug and I'll need to determine what the right approach is between referencing the stock keystore and deferring to the instance-specific keystore. Thanks for working through the root cause analysis with me!

[coolacid]

This is still an issue?

I just had to delete /etc/elasticsearch/elasticsearch.keystore from 2 nodes before puppet would stop zeroing out the real keystore location. Once I deleted it, everything started per normal.

Alas, the elasticsearch dpkg packages try and re-create the /etc/elasticsearch/elasticsearch.keystore file on upgrade.

[ml14tlc]

Hi, no, it's not anymore. Please close the ticket. Thanks for the support. Matteo

…

On Fri, 29 Mar 2019, 18:27 Jason Kendall, ***@***.***> wrote: This is still an issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#983 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AaEWgZ0_G-taP47Aocg0kkOsIXWzlIltks5vbk0GgaJpZM4WoPLA> .

[dashcheulov]

Hi,
It's still a problem in the recent version
mod 'elasticsearch', :git => 'https://github.com/elastic/puppet-elasticsearch', :commit => 'dbd06d6'
Suggested workaround above didn't help.

File <| tag == 'elasticsearch_instance_configdir' |> {
  ignore +> "/etc/elasticsearch/elasticsearch.keystore"
}

I have to remove /etc/elasticsearch/elasticsearch.keystore to prevent refreshing on every puppet run. So, this helps:

file {'/etc/elasticsearch/elasticsearch.keystore':
  ensure => absent;
}

Moreover, if elasticsearch::restart_on_change: true is set, dependency cycle appears.

(Augeas[defaults_data-node] => Elasticsearch_keystore[data-node] => Elasticsearch::Service[data-node] => Elasticsearch::Service::Systemd[data-node] => Augeas[defaults_data-node])
(Augeas[defaults_master-node] => Elasticsearch_keystore[master-node] => Elasticsearch::Service[master-node] => Elasticsearch::Service::Systemd[master-node] => Augeas[defaults_master-node])
Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz