Merge pull request #64 from ZLI-afk/v1.2

V1.2 update

Author: kevinwenminion

README.md

@@ -3,9 +3,11 @@
 </div>
 
 # APEX: Alloy Property EXplorer
+[![](https://img.shields.io/badge/release-1.2.0-blue.svg)](https://github.com/deepmodeling/APEX)
 
 [APEX](https://github.com/deepmodeling/APEX): Alloy Property EXplorer is a component of the [AI Square](https://aissquare.com/) project that involves the restructuring of the [DP-GEN](https://github.com/deepmodeling/dpgen) `auto_test` module to develop a versatile and extensible Python package for general alloy property calculations. This package enables users to conveniently establish a wide range of cloud-native property-test workflows by utilizing various computational approaches, including LAMMPS, VASP, ABACUS, and others.
 
+## v1.2 Main Features Update
 * Add a `retrieve` sub-command to allow results to be retrieved independently and manually for multiple properties (Remove `Distributor` and `Collector` OP)
 * Support common **dflow operations** with terminal commands
 * Incorporate results `archive` function to both local paths and NoSQL database ([MongoDB](https://www.mongodb.com/) and [DynamoDB](https://aws.amazon.com/cn/dynamodb/))
@@ -15,9 +17,24 @@
 * Add four additional **ML** pair styles (`snap`, `gap`, `rann` and `mace`) and an extra `meam-spline` in LAMMPS interation type support
 * Modify the single-step run command from `test` to `do` for improved clarity and consistencey
 
+## APEX Bohrium App
+[![](https://img.shields.io/badge/APP-BohriumApp-orange.svg)](https://app.bohrium.dp.tech/apex/)
+
+APEX also provides a web-based [Bohrium App](https://app.bohrium.dp.tech/apex/) for rapid and easy alloy property calculations without intensive JSON configuration (Note: one will need a Bohrium account to access this service).
+
+## How to cite APEX
+[![](https://img.shields.io/badge/DOI-10.48550/arXiv.2404.17330-red.svg)](https://doi.org/10.48550/arXiv.2404.17330)
+
+If you use APEX in your research, please cite the following paper for general purpose: 
+
+> Z. Li, T. Wen, Y. Zhang, X. Liu, C. Zhang, A. S. L. S. Pattamatta, X. Gong, B. Ye, H.Wang, L. Zhang, D. J. Srolovitz, An extendable cloud-native alloy property explorer (2024). arXiv:2404.17330.
+
 ## Table of Contents
 
 - [APEX: Alloy Property EXplorer](#apex-alloy-property-explorer)
+  - [v1.2 Main Features Update](#v12-main-features-update)
+  - [APEX Bohrium App](#apex-bohrium-app)
+  - [How to cite APEX](#how-to-cite-apex)
   - [Table of Contents](#table-of-contents)
   - [1. Overview](#1-overview)
   - [2. Easy Install](#2-easy-install)
@@ -109,7 +126,8 @@ The instructions regarding global configuration, [dflow](https://github.com/deep
   | group_size | Int | 1 | Number of tasks per parallel run group |
   | pool_size | Int | 1 | For multi tasks per parallel group, the pool size of multiprocessing pool to handle each task (1 for serial, -1 for infinity) |
   | upload_python_package | Optional[List] | None | Additional python packages required in the container |
-  | debug_pool_workers | Int | 1 | Pool size of parallel tasks running in the debug mode |
+  | debug_pool_workers | Int | 1 | Pool size of parallel tasks running in the debug mode | 
+  | flow_name | String | None | Specify name of workflow to be submitted (default: work path name) |
   | submit_only | Bool | False | Submit workflow only without automatic result retrieving |
 
 * **Dflow config**
@@ -252,11 +270,12 @@ Below are three examples (for detailed explanations of each parameter, please re
   | vol_abs | Bool | False | Whether to treat vol_start and vol_end as absolute volume, default = False |
 
 ##### 3.1.2.2. Elastic
-  | Key words    | Data structure | Example | Description                                        |
-  |:-------------|----------------|---------|----------------------------------------------------|
-  | norm_deform  | Float          | 0.01    | The deformation in xx, yy, zz, defaul = 1e-2       |
-  | shear_deform | Float          | 0.01    | The deformation in other directions, default = 1e-2 |
-  | conventional | Bool           | False   | Whether adopt conventional cell for deformation    |
+  | Key words    | Data structure | Example | Description                                                                                                                       |
+  |:-------------|----------------|---------|-----------------------------------------------------------------------------------------------------------------------------------|
+  | norm_deform  | Float          | 0.01    | The deformation in xx, yy, zz, defaul = 1e-2                                                                                      |
+  | shear_deform | Float          | 0.01    | The deformation in other directions, default = 1e-2                                                                               |
+  | conventional | Bool           | False   | Whether adopt conventional cell for deformation                                                                                   |
+  | ieee         | Bool           | True    | Whether rotate relaxed structure into IEEE-standard format before deformation ([ref](https://ieeexplore.ieee.org/document/26560)) |
 
 ##### 3.1.2.3. Surface
   | Key words | Data structure | Example | Description |

apex/__init__.py

@@ -27,5 +27,9 @@ def header():
     header_str += "  AAA           AAA   PPP           EEEEEEEEEE  XXX       XXX\n"
     header_str += "---------------------------------------------------------------\n"
     header_str += f"==>> Alloy Property EXplorer using simulations (v{__version__})\n"
+    header_str += "Please cite DOI: 10.48550/arXiv.2404.17330\n"
+    header_str += "Li et al, An extendable cloud-native alloy property explorer (2024).\n"
+    header_str += "See https://github.com/deepmodeling/APEX for more information.\n"
+    header_str += "---------------------------------------------------------------\n"
     header_str += "Checking input files..."
     print(header_str)

apex/archive.py

@@ -199,7 +199,7 @@ def archive2db_from_json(config, json_file):
     if config.archive_key:
         data_id = config.archive_key
     else:
-        data_id = str(data_dict["work_path"])
+        data_id = data_dict['archive_key']
     data_dict['_id'] = data_id
 
     archive2db(config, data_dict, data_id)

apex/config.py

@@ -50,7 +50,7 @@ class Config:
     remote_password: str = None
     port: int = 22
 
-    # calculator config
+    # basic run config
     run_image_name: str = None
     run_command: str = None
     apex_image_name: str = "zhuoyli/apex_amd64"
@@ -63,8 +63,11 @@ class Config:
     vasp_run_command: str = None
     abacus_image_name: str = None
     abacus_run_command: str = None
+
+    # common APEX config
     is_bohrium_dflow: bool = False
     submit_only: bool = False
+    flow_name: str = None
 
     database_type: str = 'local'
     archive_method: str = 'sync'

apex/core/property/Elastic.py

@@ -4,12 +4,13 @@
 import re
 from shutil import copyfile
 
-import numpy as np
 from monty.serialization import dumpfn, loadfn
 from pymatgen.analysis.elasticity.elastic import ElasticTensor
 from pymatgen.analysis.elasticity.strain import DeformedStructureSet, Strain
 from pymatgen.analysis.elasticity.stress import Stress
 from pymatgen.core.structure import Structure
+from pymatgen.core.tensors import Tensor
+from pymatgen.core.operations import SymmOp
 from pymatgen.io.vasp import Incar, Kpoints
 
 from apex.core.calculator.lib import abacus_utils
@@ -32,6 +33,8 @@ def __init__(self, parameter, inter_param=None):
             self.shear_deform = parameter["shear_deform"]
             parameter.setdefault("conventional", False)
             self.conventional = parameter["conventional"]
+            parameter.setdefault("ieee", True)
+            self.ieee = parameter["ieee"]
         parameter.setdefault("cal_type", "relaxation")
         self.cal_type = parameter["cal_type"]
         default_cal_setting = {
@@ -141,6 +144,13 @@ def make_confs(self, path_to_work, path_to_equi, refine=False):
                 ss = st.conventional_structure
                 ss.to(os.path.join(path_to_work, "POSCAR.conv"), "POSCAR")
 
+            # convert to IEEE-standard
+            if self.ieee:
+                rot = Tensor.get_ieee_rotation(ss)
+                op = SymmOp.from_rotation_and_translation(rot)
+                ss.apply_operation(op)
+                ss.to(os.path.join(path_to_work, "POSCAR.ieee"), "POSCAR")
+
             dfm_ss = DeformedStructureSet(
                 ss,
                 symmetry=False,
@@ -169,7 +179,7 @@ def make_confs(self, path_to_work, path_to_equi, refine=False):
                 dfm_ss.deformed_structures[ii].to("POSCAR", "POSCAR")
                 if self.inter_param["type"] == "abacus":
                     abacus_utils.poscar2stru("POSCAR", self.inter_param, "STRU")
-                    os.remove("POSCAR")
+                    #os.remove("POSCAR")
                 # record strain
                 df = Strain.from_deformation(dfm_ss.deformations[ii])
                 dumpfn(df.as_dict(), "strain.json", indent=4)

apex/flow.py

@@ -69,6 +69,23 @@ def __init__(
         self.executor = executor
         self.upload_python_packages = upload_python_packages
 
+    @staticmethod
+    def regulate_name(name):
+        """
+        Adjusts the given workflow name to conform to RFC 1123 subdomain requirements.
+        It ensures the name is lowercase, contains only alphanumeric characters and hyphens,
+        and starts and ends with an alphanumeric character.
+        """
+        # lowercase the name
+        name = name.lower()
+        # substitute invalid characters with hyphens
+        name = re.sub(r'[^a-z0-9\-]', '-', name)
+        # make sure the name starts and ends with an alphanumeric character
+        name = re.sub(r'^[^a-z0-9]+', '', name)
+        name = re.sub(r'[^a-z0-9]+$', '', name)
+
+        return name
+
     def _monitor_relax(self):
         print('Waiting for relaxation result...')
         while True:
@@ -77,7 +94,10 @@ def _monitor_relax(self):
             wf_status = self.workflow.query_status()
             if wf_status == 'Failed':
                 raise RuntimeError(f'Workflow failed (ID: {self.workflow.id}, UID: {self.workflow.uid})')
-            relax_post = step_info.get_step(name='relaxation-cal')[0]
+            try:
+                relax_post = step_info.get_step(name='relaxation-cal')[0]
+            except IndexError:
+                continue
             if relax_post['phase'] == 'Succeeded':
                 print(f'Relaxation finished (ID: {self.workflow.id}, UID: {self.workflow.uid})')
                 print('Retrieving completed tasks to local...')
@@ -246,12 +266,15 @@ def submit_relax(
             download_path: Union[os.PathLike, str],
             relax_parameter: dict,
             submit_only: bool = False,
+            name: Optional[str] = None,
             labels: Optional[dict] = None
     ) -> str:
         self.upload_path = upload_path
         self.download_path = download_path
         self.relax_param = relax_parameter
-        self.workflow = Workflow(name='relaxation', labels=labels)
+        flow_name = name if name else self.regulate_name(os.path.basename(download_path))
+        flow_name += '-relax'
+        self.workflow = Workflow(name=flow_name, labels=labels)
         relaxation = self._set_relax_flow(
             input_work_dir=upload_artifact(upload_path),
             relax_parameter=relax_parameter
@@ -272,12 +295,15 @@ def submit_props(
             download_path: Union[os.PathLike, str],
             props_parameter: dict,
             submit_only: bool = False,
+            name: Optional[str] = None,
             labels: Optional[dict] = None
     ) -> str:
         self.upload_path = upload_path
         self.download_path = download_path
         self.props_param = props_parameter
-        self.workflow = Workflow(name='property', labels=labels)
+        flow_name = name if name else self.regulate_name(os.path.basename(download_path))
+        flow_name += '-props'
+        self.workflow = Workflow(name=flow_name, labels=labels)
         subprops_list, subprops_key_list = self._set_props_flow(
             input_work_dir=upload_artifact(upload_path),
             props_parameter=props_parameter
@@ -299,13 +325,16 @@ def submit_joint(
             relax_parameter: dict,
             props_parameter: dict,
             submit_only: bool = False,
+            name: Optional[str] = None,
             labels: Optional[dict] = None
     ) -> str:
         self.upload_path = upload_path
         self.download_path = download_path
         self.relax_param = relax_parameter
         self.props_param = props_parameter
-        self.workflow = Workflow(name='joint', labels=labels)
+        flow_name = name if name else self.regulate_name(os.path.basename(download_path))
+        flow_name += '-joint'
+        self.workflow = Workflow(name=flow_name, labels=labels)
         relaxation = self._set_relax_flow(
             input_work_dir=upload_artifact(upload_path),
             relax_parameter=self.relax_param

apex/main.py

@@ -76,6 +76,11 @@ def parse_args():
         choices=['relax', 'props', 'joint'],
         help="(Optional) Specify type of workflow to submit: (relax | props | joint)"
     )
+    parser_submit.add_argument(
+        "-n", "--name",
+        type=str, default=None,
+        help="(Optional) Specify name of the workflow",
+    )
 
     ##########################################
     # Do single step locally
@@ -526,6 +531,7 @@ def main():
             config_file=args.config,
             work_dirs=args.work,
             indicated_flow_type=args.flow,
+            flow_name=args.name,
             submit_only=args.submit_only,
             is_debug=args.debug
         )

apex/report.py

@@ -50,6 +50,7 @@ def report_local(input_path_list):
         data_dict = loadfn(kk)
         try:
             workdir_id = data_dict.pop('work_path')
+            _ = data_dict.pop('archive_key')
         except KeyError:
             logging.warning(msg=f'Invalid json for result archive, will skip: {kk}')
             continue

apex/submit.py

@@ -41,6 +41,15 @@ def pack_upload_dir(
     os.chdir(work_dir)
     relax_confs = relax_param.get("structures", []) if relax_param else []
     prop_confs = prop_param.get("structures", []) if prop_param else []
+    relax_prefix = relax_param["interaction"].get("potcar_prefix", None) if relax_param else None
+    prop_prefix = prop_param["interaction"].get("potcar_prefix", None) if prop_param else None
+    include_dirs = set()
+    if relax_prefix:
+        relax_prefix_base = relax_prefix.split('/')[0]
+        include_dirs.add(relax_prefix_base)
+    if prop_prefix:
+        prop_prefix_base = prop_prefix.split('/')[0]
+        include_dirs.add(prop_prefix_base)
     confs = relax_confs + prop_confs
     assert len(confs) > 0, "No configuration path indicated!"
     conf_dirs = []
@@ -66,13 +75,11 @@ def pack_upload_dir(
                 backup_path(path_to_prop)
 
     """copy necessary files and directories into temp upload directory"""
-    # exclude 'all_result.json' from copy
-    conf_root_list = [conf.split('/')[0] for conf in conf_dirs]
-    conf_root_list = list(set(conf_root_list))
-    conf_root_list.sort()
-    ignore_copy_list = conf_root_list
-    ignore_copy_list.append("all_result.json")
-    copy_all_other_files(work_dir, upload_dir, ignore_list=ignore_copy_list)
+    copy_all_other_files(
+        work_dir, upload_dir,
+        exclude_files=["all_result.json"],
+        include_dirs=list(include_dirs)
+    )
     for ii in conf_dirs:
         build_conf_path = os.path.join(upload_dir, ii)
         os.makedirs(build_conf_path, exist_ok=True)
@@ -131,13 +138,15 @@ def submit(
         )
 
         flow_id = None
+        flow_name = wf_config.flow_name
         submit_only = wf_config.submit_only
         if flow_type == 'relax':
             flow_id = flow.submit_relax(
                 upload_path=tmp_dir,
                 download_path=work_dir,
                 relax_parameter=relax_param,
                 submit_only=submit_only,
+                name=flow_name,
                 labels=labels
             )
         elif flow_type == 'props':
@@ -146,6 +155,7 @@ def submit(
                 download_path=work_dir,
                 props_parameter=props_param,
                 submit_only=submit_only,
+                name=flow_name,
                 labels=labels
             )
         elif flow_type == 'joint':
@@ -155,6 +165,7 @@ def submit(
                 props_parameter=props_param,
                 relax_parameter=relax_param,
                 submit_only=submit_only,
+                name=flow_name,
                 labels=labels
             )
 
@@ -169,6 +180,7 @@ def submit_workflow(
     config_dict: dict,
     work_dirs: List[os.PathLike],
     indicated_flow_type: str,
+    flow_name: str = None,
     submit_only=False,
     is_debug=False,
     labels=None
@@ -188,6 +200,9 @@ def submit_workflow(
         config["debug_workdir"] = config_dict.get("debug_workdir", tmp_work_dir.name)
         s3_config["storage_client"] = None
 
+    if flow_name:
+        wf_config.flow_name = flow_name
+
     # judge basic flow info from user indicated parameter files
     (run_op, calculator, flow_type,
      relax_param, props_param) = judge_flow(parameter_dicts, indicated_flow_type)
@@ -269,6 +284,7 @@ def submit_from_args(
         config_file: os.PathLike,
         work_dirs,
         indicated_flow_type: str,
+        flow_name: str = None,
         submit_only=False,
         is_debug=False,
 ):
@@ -278,6 +294,7 @@ def submit_from_args(
         config_dict=load_config_file(config_file),
         work_dirs=work_dirs,
         indicated_flow_type=indicated_flow_type,
+        flow_name=flow_name,
         submit_only=submit_only,
         is_debug=is_debug,
     )