Merge pull request #503 from IBM/develop

Release zAppBuild 3.8.0

Author: dennis-behm

build.groovy

@@ -251,7 +251,7 @@ If buildFile is a text file (*.txt) then it is assumed to be a build list file.
 options:
 	'''
 
-	def cli = new CliBuilder(usage:usage,header:header)
+	def cli = new CliBuilder(usage:usage,header:header,stopAtNonOption: false)
 	// required sandbox options
 	cli.w(longOpt:'workspace', args:1, 'Absolute path to workspace (root) directory containing all required source directories')
 	cli.a(longOpt:'application', args:1, required:true, 'Application directory name (relative to workspace)')
@@ -675,8 +675,7 @@ def createBuildList() {
 	    || props.filePropertyValueKeySet().getAt("loadLanguageConfigurationProperties") 
 	    || (props.loadFileLevelProperties && props.loadFileLevelProperties.toBoolean())
 		|| (props.loadLanguageConfigurationProperties && props.loadLanguageConfigurationProperties.toBoolean())) {
-		println "** Populating file level properties from individual artifact properties files."
-		filePropUtils.loadFileLevelPropertiesFromFile(buildList)
+		filePropUtils.loadFileLevelPropertiesFromConfigFiles(buildList)
 	}
 
 	// Perform analysis and build report of external impacts

docs/FilePropertyManagement.md

@@ -2,13 +2,15 @@
 
 ## Table of contents
 
-- [Introduction](#introduction)
-- [Overriding build properties with DBB and zAppBuild](#overriding-build-properties-with-dbb-and-zappbuild)
-- [Default properties](#default-properties)
-- [Overriding properties](#overriding-properties)
-  - [DBB file properties](#dbb-file-properties)
-  - [Individual artifact properties file](#individual-artifact-properties-file)
-  - [Language configuration mapping](#language-configuration-mapping)
+- [File Property Management](#file-property-management)
+  - [Table of contents](#table-of-contents)
+  - [Introduction](#introduction)
+  - [Overriding build properties with DBB and zAppBuild](#overriding-build-properties-with-dbb-and-zappbuild)
+  - [Default properties](#default-properties)
+  - [Overriding properties](#overriding-properties)
+    - [DBB file properties](#dbb-file-properties)
+    - [Individual artifact properties file](#individual-artifact-properties-file)
+    - [Language configuration mapping](#language-configuration-mapping)
 
 ## Introduction
 
@@ -27,9 +29,10 @@ zAppBuild supports overriding the majority of build properties defined within it
 zAppBuild leverages DBB's API and allows you to define build parameters on three different levels for each language script:
 
 1. General defaults in the corresponding language properties files: For example, you can define the compile options for building COBOL programs in [application-conf/Cobol.properties](../samples/application-conf/Cobol.properties). Property keys make use of a language prefix; for instance, COBOL property keys are prefixed with `cobol_`.
-2. A group definition that overrides the general default in one of two ways:
+2. A group definition that overrides the general default in one of three ways:
    - By using DBB's file property syntax in [application-conf/file.properties](../samples/application-conf/file.properties), and specifying the application artifact group via a pattern filter on the path name(s)
-   - By mapping to a language configuration file to override the defaults, such as in [application-conf/languageConfigurationMapping.properties](../samples/MortgageApplication/application-conf/languageConfigurationMapping.properties)
+   - By mapping to a *language configuration* file to override the defaults, such as in [application-conf/languageConfigurationMapping.properties](../samples/MortgageApplication/application-conf/languageConfigurationMapping.properties) 
+   - By mapping the build file to a *language configuration* file using the DBB file property syntax file to specify the property `languageConfiguration` in [application-conf/file.properties](../samples/MortgageApplication/application-conf/file.properties#L46-L53)
 3. An individual file-level definition that overwrites the general default in one of two ways:
    - By using DBB's file properties syntax in [application-conf/file.properties](../samples/application-conf/file.properties), and specifying the application artifact's path as the file path pattern
    - By specifying multiple parameters for a specific file using the individual artifact properties file. For example: [epsmlist.cbl.properties](../samples/MortgageApplication/properties/epsmlist.cbl.properties).
@@ -132,12 +135,12 @@ You can view a sample individual artifact properties file, [epsmlist.cbl.propert
 
 ### Language configuration mapping
 
-An alternative way to define build properties for a **subgroup of files** is by leveraging a mapping approach. Rather than specifying individual parameters or properties for an individual application artifact, the application artifacts are mapped to a language configuration, which can then define multiple build parameters in a central language configuration properties file. All mapped application artifacts will inherit those defined build parameters.
+An alternative way to define build properties for a **subgroup of files** is by leveraging a mapping approach. Rather than specifying individual parameters or properties for an individual application artifact, the application artifacts are mapped to a language configuration, which can then define multiple build parameters in a central language configuration properties file for the language script. All mapped application artifacts will inherit those defined build parameters.
 
 This approach consists of:
 
-- Language configuration mapping file: A mapping of the application artifact(s) to a language configuration file
-- Language configuration properties file(s): For each language configuration, a properties file defining that language's build parameters
+- Language configuration mapping: Either via a mapping file of the application artifact(s) to the language configuration file or alternatively by using the DBB file property syntax to map the build file to the language configuration file.
+- Language configuration properties file(s): For each language configuration, a properties file defining that language's build parameters.
 
 The "language configuration mapping" approach can be enabled by setting the property `loadLanguageConfigurationProperties` in the `application-conf/application.properties` file to `true`. To enable this option for a specific file or a set of files, use the DBB file property syntax and set  `loadLanguageConfigurationProperties` to `true` in the `application-conf/file.properties` file. Below is a sample to enable language configuration mapping for all programs starting with `eps` and `lga` via the `application-conf/file.properties` file:
 
@@ -157,7 +160,9 @@ isCICS = true
 cobol_linkEditStream=    INCLUDE OBJECT(@{member})\n    INCLUDE SYSLIB(CUSTOBJ)
 ```
 
-To map files to a language configuration, create a `languageConfigurationMapping.properties` file in the `application-conf` folder of your application repository. Then, within this new language configuration mapping file, map each artifact to its corresponding language configuration using the syntax `<sourceFileName.extension>=<languageConfigurationPropertiesFileName>`.
+Two options exist to map build files to the language configuration:
+
+Either create a `languageConfigurationMapping.properties` file in the `application-conf` folder of your application repository. Then, within this new language configuration mapping file, map each artifact to its corresponding language configuration using the syntax `<sourceFileName.extension>=<languageConfigurationPropertiesFileName>`, or leverage the DBB file property syntax with the build property `languageConfiguration`.
 
 - For example, the following snippet in [application-conf/languageConfigurationMapping.properties](../samples/MortgageApplication/application-conf/languageConfigurationMapping.properties) maps both source files `epsnbrvl.cbl` and `epsmlist.cbl` to use the properties defined in `build-conf/language-conf/languageConfigProps01.properties`,  while the source file `epscmort.cbl` is mapped to use the properties defined in `build-conf/language-conf/languageConfigProps02.properties` for language configuration mapping overrides:
 
@@ -167,7 +172,17 @@ To map files to a language configuration, create a `languageConfigurationMapping
   epscmort.cbl=languageConfigProps02
   ```
 
-See [languageConfigurationMapping.properties](../samples/MortgageApplication/application-conf/languageConfigurationMapping.properties) for a sample language configuration mapping file.
+  See [languageConfigurationMapping.properties](../samples/MortgageApplication/application-conf/languageConfigurationMapping.properties) for a sample language configuration mapping file.
+
+- The alternative mapping approach using the DBB file property syntax allows you to leverage file patterns similarly to other file properties in the zAppBuild framework. The below configuration is providing the same information like the above approach using the `languageConfigurationMapping.properties` file.
+
+  ```properties
+  languageConfiguration = languageConfigProps01 :: **/cobol/epsnbrvl.cbl, **/cobol/epsmpmt.cbl
+  languageConfiguration = languageConfigProps02 ::  **/cobol/epscmort.cbl
+  ```
+  
+  See [application-conf/file.properties](../samples/MortgageApplication/application-conf/file.properties#L46-L53) for a sample language configuration mapping via the DBB file property syntax.
+
 
 [^1]: The term "artifact" and "file" in this document refer to program source code that will built (as opposed to JCL or other non-buildable items), for example by DBB.
 [^2]: DBB is managing the DBB file properties in its separate internal table compared to the default properties. This table leverages the combination of [property name + file pattern] as the key of the internal table. When the same key is declared a second time, it overrides the first one.

samples/MortgageApplication/application-conf/application.properties

@@ -4,8 +4,7 @@
 # Comma separated list of additional application property files to load
 # Supports both relative path (to ${application}/application-conf/) and absolute path
 applicationPropFiles=file.properties,BMS.properties,Cobol.properties,\
-LinkEdit.properties,languageConfigurationMapping.properties,\
-CRB.properties
+LinkEdit.properties,CRB.properties
 
 #
 # Comma separated list all source directories included in application build. Supports both absolute 
@@ -21,8 +20,13 @@ applicationSrcDirs=${application}
 buildOrder=BMS.groovy,Cobol.groovy,LinkEdit.groovy
 
 #
-# The main build branch.  Used for cloning collections for topic branch builds instead
-# of rescanning the entire application.
+# The main build branch. 
+# Used for cloning collections for first topic branch build instead
+#  of rescanning the entire application.
+# Used to retrieve the last successful build result for the first topic branch build to 
+#  obtain the baseline hash to calculate changed files for impact build scnearios.
+# Used for the build type --mergeBuild to perform the git diff to 
+#  identify changed files
 mainBuildBranch=main
 
 #

samples/MortgageApplication/application-conf/file.properties

@@ -41,4 +41,13 @@ loadFileLevelProperties = true :: **/cobol/epsmlist.cbl
 # Example 1: Enable language configuration mapping for all COBOL files: loadLanguageConfigurationProperties=true :: **/cobol/*.cbl
 # Example 2: The following file path pattern looks for a language configuration properties file mapping for epsnbrvl.cbl and epsmpmt.cbl 
 # in the application-conf/languageconfigurationMapping.properties file:
-loadLanguageConfigurationProperties = true :: **/cobol/epsnbrvl.cbl, **/cobol/epsmpmt.cbl
+loadLanguageConfigurationProperties = true :: **/cobol/epsnbrvl.cbl, **/cobol/epsmpmt.cbl, **/cobol/epscmort.cbl
+
+#
+# Properties mapping for assigning the language configuration to build files to specify the build configuration
+# Please see the detailed documentation on file property management at:
+# https://github.com/IBM/dbb-zappbuild/docs/FilePropertyManagement.md
+# This option replaces the mapping of build files to language configuration in languageConfigurationMapping.properties file
+# 
+languageConfiguration = languageConfigProps01 :: **/cobol/epsnbrvl.cbl, **/cobol/epsmpmt.cbl
+languageConfiguration = languageConfigProps02 ::  **/cobol/epscmort.cbl

samples/MortgageApplication/application-conf/languageConfigurationMapping.properties

@@ -1,11 +1,18 @@
 # This file maps the program file with the corresponding language configuration properties file in the build-conf/language-conf folder.
 
-# For example: As shown below, the programs `epsnbrvl.cbl` and `epsmpmt.cbl` will map to the language configuration file
+# The file needs to be added under the property applicationPropFiles in application-conf/application.properties.
+# The current implementation of MortgageApplication is using DBB File Properties.
+
+# Please see the detailed documentation on file property management at:
+# https://github.com/IBM/dbb-zappbuild/docs/FilePropertyManagement.md
+
+# In the below samples, the programs `epsnbrvl.cbl` and `epsmpmt.cbl` will map to the language configuration file
 # build-conf/language-conf/languageConfigProps01.properties.
 # The program `epscmort.cbl` will map to the language configuration file build-conf/language-conf/languageConfigProps02.properties.
 
 epsnbrvl.cbl=languageConfigProps01
 epsmpmt.cbl=languageConfigProps01
 epscmort.cbl=languageConfigProps02
 
-# Insert all the file ==> language configuration properties mapping here!
+# Insert all the file ==> language configuration properties mapping here
+# and make sure to load this properties file via applicationPropFiles in application-conf/application.properties!

samples/application-conf/README.md

@@ -34,14 +34,23 @@ impactResolutionRules | Comma separated list of resolution rule properties used
 impactSearch | Impact finder resolution search configuration leveraging the SearchPathImpactFinder API. Sample configurations are inlcuded below, next to the previous rule definitions. | true
 
 ### file.properties
-Location of file properties, script mappings and file-level property overrides. All file properties for the entire application, including source files in distributed repositories of the application need to be contained either in this file or in other property files in the `application-conf` directory. Look for the column 'Overridable' in the tables below for build properties that can have file-level property overrides. Additional file-level properties can be defined through individual artifact properties files in a separate directory of the repository. For more details, see [File Property Management](https://github.com/IBM/dbb-zappbuild/docs/FilePropertyManagement.md).
+
+Location of file properties, script mappings and file-level property overrides. All file properties for the entire application, including source files in distributed repositories of the application need to be contained either in this file or in other property files in the `application-conf` directory. Look for the column 'Overridable' in the tables below for build properties that can have file-level property overrides. Please also read the section [Build properties](https://www.ibm.com/docs/en/dbb/2.0?topic=apis-build-properties) in the official DBB documentation.
+
+Additional file-level properties can be defined through **individual artifact properties files** in a separate directory of the repository or through **language configuration** files to configure the language scripts. For more details, see [File Property Management](https://github.com/IBM/dbb-zappbuild/docs/FilePropertyManagement.md).
 
 Property | Description
 --- | ---
 dbb.scriptMapping | DBB configuration file properties association build files to language scripts
 dbb.scannerMapping | zAppBuild configuration override/expansion to map files extensions to DBB dependency scanner configurations
 cobol_testcase | File property to indicate a generated zUnit cobol test case to use a different set of source and output libraries
 
+Property | Description
+--- | ---
+loadFileLevelProperties | Flag to enable the zAppBuild capability to load individual artifact properties files for source files. See default configuration in [application.properties](#application.properties)
+loadLanguageConfigurationProperties | Flag to enable the zAppBuild capability to load language configuration properties files for source files. See default configuration in [application.properties](#application.properties)
+languageConfiguration | Properties mapping for assigning build files to language configuration to define the build configuration
+
 General file level overwrites to control the allocations of system datasets for compile and link steps or activation of preprocessing
 
 Property | Description

samples/application-conf/application.properties

@@ -37,9 +37,14 @@ testOrder=ZunitConfig.groovy
 # formatConsoleOutput=false
 
 #
-# The main build branch.  Used for cloning collections for topic branch builds instead
-# of rescanning the entire application.
-mainBuildBranch=master
+# The main build branch. 
+# Used for cloning collections for first topic branch build instead
+#  of rescanning the entire application.
+# Used to retrieve the last successful build result for the first topic branch build to 
+#  obtain the baseline hash to calculate changed files for impact build scnearios.
+# Used for the build type --mergeBuild to perform the git diff to 
+#  identify changed files
+mainBuildBranch=main
 
 #
 # The git repository URL of the application repository to establish links to the changed files

samples/application-conf/file.properties

@@ -83,8 +83,50 @@ isMQ = true :: **/cobol/member.cbl
 #  
 isIMS = true :: **/cobol/DLIBATCH.cbl
 
-# Please check for available file property overwrites within samples/application-conf/README.md
 
+#########
+# Property File Management
+#
+# > Please check for available file property overwrites within samples/application-conf/README.md <
+# The tables indicate which build parameters can be overwritten for each build file to overwrite the default settings.
+# 
+# The subsequent section provides the configuration parameters to outline
+# - file level property overwrites by loading individual artifact properties files
+# - loading language configurations that define multiple build parameters for the mapped files
+#
+# Please see the detailed documentation on file property management at:
+# https://github.com/IBM/dbb-zappbuild/docs/FilePropertyManagement.md
+#
+#########
+
+#
+# Flag to enable the zAppBuild capability to load individual artifact properties files for source files.
+# Example 1: The following file path pattern looks for an individual artifact properties file for epsmlist.cbl:
+# loadFileLevelProperties = true :: **/cobol/epsmlist.cbl 
+# Example 2: Enable individual artifact properties file for all COBOL files: loadFileLevelProperties=true :: **/cobol/*.cbl
+# loadFileLevelProperties = true :: **/cobol/*.cbl 
+
+#
+# Language Configuration properties management
+#
+# Flag to enable the zAppBuild capability to load language configuration properties files for source files.
+# Example 1: Enable language configuration mapping for all COBOL files: loadLanguageConfigurationProperties=true :: **/cobol/*.cbl
+# loadLanguageConfigurationProperties = true :: **/cobol/*.cbl
+# Example 2: The following file path pattern looks for a language configuration properties file mapping for epsnbrvl.cbl and epsmpmt.cbl 
+# in the application-conf/languageconfigurationMapping.properties file:
+# loadLanguageConfigurationProperties = true :: **/cobol/epsnbrvl.cbl, **/cobol/epsmpmt.cbl
+
+#
+# Properties mapping for assigning the language configuration to build files to specify the build configuration
+# This option is an alternative approach for mapping build files to the language configuration file
+# 
+# languageConfiguration = languageConfigProps01 :: **/cobol/epsnbrvl.cbl, **/cobol/epsmpmt.cbl
+# languageConfiguration = languageConfigProps02 ::  **/cobol/epscmort.cbl
+
+
+#########
+# zUnit Integration
+#########
 #
 # file mapping for generated zUnit Test case programs (Cobol) to use a seperate set of libraries
 # cobol_testcase = true :: **/testcase/*.cbl