YDBOPS-12014 Logging, storage config generation, fixes

README.md

@@ -392,10 +392,11 @@ There are two possible ways to add new nodes into the cluster:
 
 ## Simple
 
-1. Update config.yaml - add new nodes into `hosts` section.
+1. Update config.yaml - add new nodes into `hosts` section and increase value of `storage_config_generation`
 
 Example. Before changes
 ```
+storage_config_generation: 0
 hosts:
 - host: ydb-node01.ru-central1.internal
   host_config_id: 1
@@ -414,6 +415,7 @@ hosts:
 
 Example. After changes
 ```
+storage_config_generation: 1
 hosts:
 - host: ydb-node01.ru-central1.internal
   host_config_id: 1
@@ -453,16 +455,17 @@ all:
 ```
 
 5. Install YDB on new nodes and start them
-`ansible-playbook ydb_platform.ydb.initial_setup -l ydb-node-NEW.ru-central1.internal --skip-tags password,create_database`
+`ansible-playbook ydb_platform.ydb.initial_setup -l ydb-node-NEW.ru-central1.internal --skip-tags password,create_database --extra-vars "ydb_storage_update_config=True"`
 6. Check the cluster
 
 ## Long
 
-1. Update config.yaml - add new nodes into `hosts` section.
+1. Update config.yaml - add new nodes into `hosts` section and increase value of `storage_config_generation`.
 
 Example. Before changes
 
 ```
+storage_config_generation: 0
 hosts:
 - host: ydb-node01.ru-central1.internal
   host_config_id: 1
@@ -482,6 +485,7 @@ hosts:
 Example. After changes
 
 ```
+storage_config_generation: 1
 hosts:
 - host: ydb-node01.ru-central1.internal
   host_config_id: 1
@@ -523,7 +527,7 @@ all:
 5. Prepare nodes for YDB:
 `ydb_platform.ydb.prepare_host -l ydb-node-NEW.ru-central1.internal`
 6. Install YDB on new static nodes and start them
-`ydb_platform.ydb.install_static -l ydb-node-NEW.ru-central1.internal --skip-tags password,create_database`
+`ydb_platform.ydb.install_static -l ydb-node-NEW.ru-central1.internal --skip-tags password,create_database --extra-vars "ydb_storage_update_config=True"`
 7. Install YDB on new dynamic nodes and start them
 `ydb_platform.ydb.install_dynamic -l ydb-node-NEW.ru-central1.internal --skip-tags password,create_database`
 8. Check the cluster
@@ -546,3 +550,15 @@ all:
     $ sudo systemctl stop ydbd-database-a
     $ sudo systemctl stop ydbd-database-b
     ```
+3) Q: How to solve problem with `init YDB storage if not initialized` step with `ItemConfigGenerationExpected` message?
+   A1: Add --extra-vars "ydb_storage_update_config=True" to command line
+   A2: Add `ydb_storage_update_config: True` in inventory
+
+4) Q: Restart takes a lot of time, no output, no activities. What to do?
+   A: You can switch on logging in this case - add --extra-vars "ydbops_log=/tmp/ydbops.log" to your command and logs will be recorded into `/tmp/ydbops.log`
+   A: Also you can change availability mode from `weak` to `force`. Be aware - this option could lead to cluster unavailability for some time.
+
+   ```bash
+   ansible-playbook ydb_platform.ydb.restart --extra-vars "ydbops_log=/tmp/ydbops.log"
+   ansible-playbook ydb_platform.ydb.restart --extra-vars "availability_mode=force"
+   ```

galaxy.yml

@@ -8,7 +8,7 @@ namespace: ydb_platform
 name: ydb
 
 # The version of the collection. Must be compatible with semantic versioning
-version: 1.1.0
+version: 1.2.0
 
 # The path to the Markdown (.md) readme file. This path is relative to the root of the collection
 readme: README.md

playbooks/factory_setup.yaml

@@ -41,3 +41,5 @@
     - name: setup monitoring service
       ansible.builtin.import_role:
         name: ydb_platform.ydb.monitoring_services
+      tags:
+        - monitoring

playbooks/logs.yaml

@@ -26,3 +26,14 @@
       tags:
         - database
         - dynamic
+    - name: Get syslog logs
+      ansible.builtin.shell: tail -n 1000 /var/log/syslog | grep ydbd
+      register: tail
+      become: true
+      tags:
+        - syslog
+    - name: Show syslog logs
+      debug: var=tail.stdout_lines
+      tags:
+        - syslog
+

playbooks/monitoring.yaml

@@ -0,0 +1,11 @@
+---
+
+- name: setup YDB monitoring on appliance factory
+  hosts: monitoring
+  become: true
+  tasks:
+    - name: setup monitoring service
+      ansible.builtin.import_role:
+        name: ydb_platform.ydb.monitoring_services
+      tags:
+        - monitoring

playbooks/update_config.yaml

@@ -1,9 +1,23 @@
 ---
+# PLAYBOOK: ydb_platform.ydb.update_config
+# PROPOSE : This playbook is designed for updating configs and restarting cluster
+#           
+# EXAMPLE:
+#
+# Update and restart all nodes
+#   ansible-playbook ydb_platform.ydb.update_config
+#
+# Only update configs on all nodes 
+#   ansible-playbook ydb_platform.ydb.update_config -t no_restart
+#
+# Update and restart all nodes with FORCE mode
+#   ansible-playbook ydb_platform.ydb.update_config --extra-vars "availability_mode=force"
 
 - name: Check if required variables are defined
   hosts: "{{ ansible_play_hosts | default('ydb') }}"
 
   tasks:
+
   - ansible.builtin.assert:
       that:
         - "{{ item }} is defined"
@@ -17,7 +31,9 @@
     - database
     - dynamic
     - no_restart
-- hosts: "{{ ansible_play_hosts | default('ydb') }}"
+
+- name: Load presettings
+  hosts: "{{ ansible_play_hosts | default('ydb') }}"
   roles:
   - role: preflight
   tags:
@@ -27,8 +43,15 @@
     - dynamic
     - no_restart
 
-- hosts: "{{ ansible_play_hosts | default('ydb') }}"
+- name: Update configs
+  hosts: "{{ ansible_play_hosts | default('ydb') }}"
+  become: true
+  become_method: sudo
+  become_user: root
+  vars:
+    ydb_storage_update_config: True
   tasks:
+
   - name: copy original configuration file
     copy:
       src: "{{ ydb_config }}"
@@ -52,9 +75,6 @@
       - storage
       - static
       - no_restart
-    become: true
-    become_method: sudo
-    become_user: root
   - name: create dynamic node configuration file
     command: "{{ ydb_dir }}/home/update_config_file.sh ydbd-config.yaml ydbd-config-dynamic.yaml COMPUTE {{ ydb_cores_dynamic }}"
     changed_when: false
@@ -64,6 +84,53 @@
       - no_restart
     become: true
     become_method: sudo
-    become_user: root
+
+  - name: Get ydb token
+    ydb_platform.ydb.get_token:
+      ydb_bin: "{{ ydb_dir }}/bin/ydb"
+      ca_file: "{{ ydb_dir }}/certs/ca.crt"
+      endpoint: "grpcs://{{ ydb_front | default(inventory_hostname) }}:2135"
+      database: "/{{ ydb_domain }}"
+      user: "{{ ydb_user }}"
+      password: "{{ ydb_password }}"
+    register: ydb_credentials
+    until: "'token' in ydb_credentials"
+    retries: 10
+    delay: 10
+    when: ydb_storage_update_config|bool
+    run_once: true
+    tags:
+      - storage
+      - no_restart
+
+  - name: init YDB storage if not initialized
+    ydb_platform.ydb.init_storage:
+      config_file: "{{ ydb_dir }}/cfg/ydbd-config-static.yaml"
+      ydbd_bin: "{{ ydb_dir }}/bin/ydbd"
+      ca_file: "{{ ydb_dir }}/certs/ca.crt"
+      endpoint: "grpcs://{{ ydb_front | default(inventory_hostname) }}:2135"
+      token: "{{ ydb_credentials.token }}"
+      update_config: "{{ ydb_storage_update_config }}"
+    register: init_storage
+    run_once: true
+    when: ydb_storage_update_config|bool
+    tags:
+      - storage
+      - no_restart
+
+  - name: wait for ydb healthcheck switch to "GOOD" status
+    ydb_platform.ydb.wait_healthcheck:
+      ydb_bin: "{{ ydb_dir }}/bin/ydb"
+      ca_file: "{{ ydb_dir }}/certs/ca.crt"
+      endpoint: "grpcs://{{ ydb_front | default(inventory_hostname) }}:2135"
+      database: "/{{ ydb_domain }}"
+      token: "{{ ydb_credentials.token }}"
+      enforce_user_token_requirement: "{{ ydb_enforce_user_token_requirement }}"
+    run_once: true
+    when: ydb_storage_update_config|bool
+    tags:
+      - storage
+      - no_restart
+
 - name: Restart YDB cluster
   ansible.builtin.import_playbook: restart.yaml

plugins/module_utils/cli.py

@@ -1,9 +1,9 @@
 import copy
-import socket
-
+import shlex
 
 class CLI:
     argument_spec = None
+    use_unsafe_shell = False
 
     def __init_subclass__(cls):
         if not isinstance(cls.argument_spec, dict):
@@ -34,12 +34,14 @@ def __call__(self, cmd, env=None):
         if not isinstance(cmd, list):
             raise ValueError('cmd must be list')
         cmd = self.common_options + cmd
+        if self.use_unsafe_shell:
+            cmd_format = getattr(self, "cmd_format", "{cmd}")
+            cmd = cmd_format.format(cmd=shlex.join(cmd))
         environ_update = copy.deepcopy(self.common_environ)
         if isinstance(env, dict):
             environ_update.update(env)
         self.module.log(f'calling command: {cmd}')
-        return self.module.run_command(cmd, environ_update=environ_update)
-
+        return self.module.run_command(cmd, environ_update=environ_update, use_unsafe_shell=self.use_unsafe_shell)
 
 class YDBD(CLI):
     argument_spec = dict(
@@ -148,10 +150,11 @@ class YdbOps(CLI):
         token=dict(type='str', default=None, no_log=True),
         token_file=dict(type='str', default=None),
         hosts=dict(type='str',default=None),
+        log_file=dict(type='str',default=None),
     )
 
     def __init__(self, module, ydbops_bin, ydbops_endpoint, ydbops_systemd_unit=None,
-                 ca_file=None, ssh_args=None, availability_mode=None, token=None, token_file=None, hosts=None):
+                 ca_file=None, ssh_args=None, availability_mode=None, token=None, token_file=None, hosts=None, log_file=None):
         self.module = module
 
         self.common_options = [ydbops_bin, 'restart', '--storage']
@@ -173,3 +176,6 @@ def __init__(self, module, ydbops_bin, ydbops_endpoint, ydbops_systemd_unit=None
             self.common_environ['YDB_TOKEN'] = token
         elif token_file is not None:
             self.common_options.extend(['--token-file', token_file])
+        if log_file is not None and log_file != "":
+            self.cmd_format = "{cmd} >> " + shlex.quote(log_file)
+            self.use_unsafe_shell = True

plugins/modules/init_storage.py

@@ -1,6 +1,6 @@
 import json
-import os
-
+import re
+import yaml
 
 from ansible.module_utils.basic import AnsibleModule
 from ansible_collections.ydb_platform.ydb.plugins.module_utils  import cli
@@ -9,6 +9,7 @@
 def main():
     argument_spec=dict(
         config_file=dict(type='str', required=True),
+        update_config=dict(type='bool', default=False)
     )
     cli.YDBD.add_arguments(argument_spec)
     cli.DsTool.add_arguments(argument_spec)
@@ -18,6 +19,7 @@ def main():
         ydbd_cli = cli.YDBD.from_module(module)
         ydb_dstool = cli.DsTool.from_module(module)
         rc, stdout, stderr = ydb_dstool(['box', 'list', '--format=json'])
+        update_config = module.params.get('update_config')
         if rc != 0:
             result['msg'] = 'dstool command box list failed'
             result['stdout'] = stdout
@@ -34,7 +36,7 @@ def main():
             result['stderr'] = stderr
             module.fail_json(**result)
 
-        if initialized:
+        if initialized and not update_config:
             result['msg'] = 'storage already initialized'
             result['stdout'] = stdout
             result['stderr'] = stderr
@@ -45,9 +47,34 @@ def main():
         ])
         if rc != 0:
             result['msg'] = 'blobstorage config init failed'
-            result['stdout'] = stdout
-            result['stderr'] = stderr
-            module.fail_json(**result)
+            if update_config:
+                # Detect required config generation
+                pattern = r'ItemConfigGenerationExpected: (\d+)'
+                match = re.search(pattern, stdout)
+                if match:
+                    expected_value = int(match.group(1))
+                    # Replace current generation in config
+                    with open(module.params.get('config_file'), 'r') as file:
+                        data = yaml.safe_load(file)
+                    if data is not None:
+                        if 'storage_config_generation' in data:
+                            data['storage_config_generation'] = expected_value
+                            new_config_file = module.params.get('config_file') + '-ansible'
+                            with open(new_config_file, 'w') as file:
+                                yaml.dump(data, file, default_flow_style=False)
+                                rc, stdout, stderr = ydbd_cli([
+                                    'admin', 'blobstorage', 'config', 'init', '--yaml-file', new_config_file
+                                ])
+                                if rc != 0:
+                                    result['msg'] = f"blobstorage config init failed with expected config"
+                    else:
+                        result['msg'] = f"blobstorage config init failed, Expected value of ItemConfigGenerationExpected: {expected_value}"
+                else:
+                    result['msg'] = f"blobstorage config init failed, Expected value was not found"
+            if rc != 0:
+                result['stdout'] = stdout
+                result['stderr'] = stderr
+                module.fail_json(**result)
 
         # Check PDisk status
         rc, stdout, stderr = ydb_dstool(['pdisk', 'list', '--format=json'])