diff --git a/README.md b/README.md index b90334464..8aea28448 100644 --- a/README.md +++ b/README.md @@ -6,26 +6,26 @@ ## Overview -Power Apps Test Engine is an open source project with the aim of providing makers with a single automated testing platform for all Power Apps apps. Test Engine has the following benefits: +Power Apps Test Engine is an open source project with the aim of providing makers with a single automated testing platform for all Power Apps apps. -- Power Fx test authoring - Makers can author tests in YAML format using the familiar Power Fx language. +Test Engine currently supports Power Apps canvas apps. Model driven apps are not currently supported. + +Test Engine has the following benefits: + +- Power Fx test authoring - Makers author tests in YAML format using the familiar Power Fx language. - DOM abstraction - Tests are authored using references to control names that are defined at design-time. Test authors do not need to write JavaScript, and do not need to be familiar with the browser DOM of the app's rendered output. - Connector mocking - Test authors can optionally create mocks of network calls, typically used when Power Apps make calls to connectors. This allows the app to be tested without modification to the app itself while avoiding any unwanted side-effects of the external APIs. -- Screenshot and video recording support - Test Engine can take screenshots at any point during your test execution, and records videos of the test run. This can be very helpful to diagnose failed tests and to understand what the actual experience of the failed test case was. +- Screenshot and video recording support - The Test Engine can take screenshots at any point during your test execution, and may records videos of the test run. This can be very helpful to diagnose failed tests. Build this project using the instructions below. This will create a local executable that can be used to run tests from your machine. -Test Engine uses [Playwright](https://playwright.dev) to orchestrate the tests. - -Test Engine currently supports Power Apps canvas apps. - ## Getting Started To get started, you will need to clone the Test Engine code from GitHub, locally build the project, and install the browser(s) you wish to use to execute the tests. Once you have the Test Engine executable built, the repo contains a library of sample test plans and apps you can use to exercise the tool. ### Prerequisites for building Test Engine -1. Install [.NET Core 6.0.x SDK](https://dotnet.microsoft.com/en-us/download/dotnet/6.0) +1. Install [.NET Core 6.0.x SDK](https://dotnet.microsoft.com/en-us/download/dotnet/6.0) or later 1. Ensure that your `MSBuildSDKsPath` environment variable is pointing to [.NET Core 6.0.x SDK](https://dotnet.microsoft.com/en-us/download/dotnet/6.0). 1. Make sure [PowerShell](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.2) is installed. @@ -106,7 +106,7 @@ This refers to the account that Test Engine will use to execute the test. Test Engine does not support multi-factor authentication. Use an account that requires only a username and password to sign in for your tests. -Test credentials cannot be stored in test plan files. Rather, they are stored in PowerShell environment variables. The test plan file contains references to which environment variables are used for credentials. For example, the following snippet indicates that the `user1Email` and `user1Password` environment variables will be used: +For safety reasons test credentials cannot be stored in test plan files. Rather, they are stored in environment variables. The test plan file contains references to which environment variables are used for credentials. For example, the following snippet indicates that the `user1Email` and `user1Password` environment variables will be used: ```yaml environmentVariables: diff --git a/docs/Yaml/README.md b/docs/Yaml/README.md index 4a725e0bc..5283d592d 100644 --- a/docs/Yaml/README.md +++ b/docs/Yaml/README.md @@ -8,6 +8,6 @@ View the [samples](../../samples/) folder for detailed examples. | Property | Description | | -- | -- | -| [test](./test.md) | Defines one test suite, the test cases in the test suite and configuration specific to the test suite | +| [testSuite](./testSuite.md) | Defines one test suite, the test cases in the test suite and configuration specific to the test suite | | [testSettings](./testSettings.md) | Defines settings for the test suite that are reused across multiple test cases | | [environmentVariables](./environmentVariables.md) | Defines variables that could potentially change as the app is ported across different environments | \ No newline at end of file diff --git a/docs/Yaml/TestEngineSchema.json b/docs/Yaml/TestEngineSchema.json new file mode 100644 index 000000000..6e38742e0 --- /dev/null +++ b/docs/Yaml/TestEngineSchema.json @@ -0,0 +1,363 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema#", + "$comment": "1.0.0", + "$id": "https://github.com/microsoft/PowerApps-Language-Tooling/blob/master/docs/TestEngine-schema.json", + "title": "Microsoft Test Engine test schema", + "description": "A schema for describing the structure of tests executed by the PowerApps test engine.", + "additionalProperties": false, + "type": "object", + "properties": { + "testSuite": { + "type": "object", + "description": "Information about the test suite being run.", + "properties": { + "testSuiteName": { + "type": "string", + "description": "The name of the test suite.", + "examples": [ + "MyTestSuite", + "TheTestSuite" + ] + }, + "testSuiteDescription": { + "type": "string", + "description": "Additional information about the test suite.", + "examples": [ + "This test suite contains tests for the MyTestApp app." + ] + }, + "persona": { + "type": "string", + "description": "The user that is logged in to perform the test. The persona is defined by the personaName in the environmentVariables." + }, + "appLogicalName": { + "type": "string", + "description": "The application logical name." + }, + "appId": { + "type": "string", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", + "examples": [ + "5f30b052-2edc-4da7-b950-b894978a1eba" + ], + "description": "The ID of the app that is to be launched. Required and used only when app logical name isn't present. App IDs should be used only for canvas apps that aren't in the solution." + }, + "networkRequestMocks": { + "type": "array", + "items": [ + { + "$ref": "#/$definitions/networkRequestMockInfo" + } + ] + }, + "onTestCaseStart": { + "type": "string", + "description": "The name of the function to be called when a test case starts." + }, + "onTestCaseComplete": { + "type": "string", + "description": "The name of the function to be called when a test case completes." + }, + "onTestSuiteComplete": { + "type": "string", + "description": "The name of the function to be called when a test suite completes." + }, + "testCases": { + "type": "array", + "items": [ + { + "$ref": "#/$definitions/testCase" + } + ] + } + }, + "additionalProperties": false, + "oneOf": [ + { + "required": [ + "appLogicalName" + ], + "not": { + "required": [ + "appId" + ] + } + }, + { + "required": [ + "appId" + ], + "not": { + "required": [ + "appLogicalName" + ] + } + } + ], + "required": [ + "testSuiteName", + "persona" + ] + }, + "testSettings": { + "description": "Settings defining the settings under which the tests are run.", + "oneOf": [ + { + "type": "object", + "description": "Options defining the test settings inline.", + "properties": { + "locale": { + "type": "string", + "description": "The locale to use for the test. See Global Support in Microsoft Power Fx for more information. If unspecified, the current culture is used for the locale by default for parsing the test steps.", + "examples": [ + "en-us", + "de" + ] + }, + "browserConfigurations": { + "type": "array", + "items": [ + { + "$ref": "#/$definitions/browserConfigurationInfo" + } + ] + }, + "recordVideo": { + "type": "boolean", + "description": "Indicates whether to record a video of the test run. The video is saved in the test results folder." + }, + "headless": { + "type": "boolean", + "description": "Indicates whether to run the test in headless mode. The default is True. Headless mode is a browser mode that runs without a UI." + }, + "timeout": { + "type": "integer", + "description": "The timeout in milliseconds for the test run. The default is 300 seconds. If any operation takes longer than the timeout limit, it ends the test in a failure.", + "examples": [ + "30000" + ] + } + }, + "required": [ + "locale", + "browserConfigurations" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "filePath": { + "type": "string", + "description": "The file path to a separate yaml file with all the test settings. If provided, it will override all the test settings in the test plan." + } + }, + "required": [ + "filePath" + ], + "additionalProperties": false + } + ] + }, + "environmentVariables": { + "description": "The environment variables to use for the test. These can be provided inline (under users) or in a separate yaml file (under filePath).", + "oneOf": [ + { + "type": "object", + "properties": { + "users": { + "$ref": "#/$definitions/userInformation" + } + }, + "required": [ + "users" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "filePath": { + "type": "string", + "description": "The file path to a separate yaml file with all the environment variables." + } + }, + "additionalProperties": false, + "required": [ + "filePath" + ] + } + ], + "additionalItems": false + } + }, + "required": [ + "testSuite" + ], + "$definitions": { + "browserConfigurationInfo": { + "type": "object", + "description": "The browser configurations to use for the test.", + "properties": { + "browser": { + "enum": [ + "Chromium", + "Edge", + "Firefox", + "Safari", + "IE" + ], + "description": "The browser type to use for the test.", + "examples": [ + "Edge", + "IE" + ] + }, + "device": { + "enum": [ + "Desktop", + "Mobile" + ], + "description": "The device type to use for the test." + }, + "screenHeight": { + "type": "integer", + "description": "The screen height to use for the test. If specified, screenWidth must also be specified." + }, + "screenWidth": { + "type": "integer", + "description": "The screen width to use for the test. If specified, screenHeight must also be specified." + } + }, + "required": [ + "browser" + ], + "additionalProperties": false + }, + "networkRequestMockInfo": { + "type": "object", + "description": "Defines network request mocks needed for the test.", + "properties": { + "requestURL": { + "type": "string", + "description": "Defines network request mocks needed for the test." + }, + "responseDataFile": { + "type": "string", + "description": "A text file with the mock response content. All text in this file is read as the response." + }, + "method": { + "enum": [ + "GET", + "POST", + "PUT", + "PATCH", + "DELETE" + ], + "description": "The HTTP method of the request." + }, + "headers": { + "type": "object", + "description": "The request headers consisting of pairs [fieldName: fieldValue].", + "examples": [ + "x-ms-request-method: GET" + ] + }, + "requestBodyFile": { + "type": "string", + "description": "A text file with the request body content. All text in this file is read as the request body." + } + }, + "required": [ + "requestURL", + "responseDataFile" + ] + }, + "networkRequestMockHeaderInfo": { + "type": "object", + "description": "Defines a network request mock header.", + "examples": [ + "x-ms-request-method: GET" + ], + "properties": { + "fieldName": { + "type": "string", + "description": "The name of the header field." + }, + "fieldValue": { + "type": "string", + "description": "The value of the header field." + } + }, + "required": [ + "fieldName", + "fieldValue" + ] + }, + "testCase": { + "type": "object", + "description": "Information about a test case. ", + "properties": { + "testCaseName": { + "type": "string", + "description": "The name of the test case.", + "examples": [ + "My Test case" + ] + }, + "testCaseDescription": { + "type": "string", + "description": "Additional information about the test case.", + "examples": [ + "This test case contains tests for the MyTestApp app." + ] + }, + "testSteps": { + "type": "string", + "description": "This is a sequence of Power Fx calls separated by ;. Use = as the first character.", + "examples": [ + "Select(Calculator_1.Add);" + ] + } + }, + "additionalProperties": false, + "required": [ + "testCaseName", + "testSteps" + ] + }, + "userInformation": { + "type:": "object", + "description": "Information about a user.", + "properties": { + "personaName": { + "type": "string", + "description": "The name of environment variable containing the user name.", + "examples": [ + "UserNameVar" + ] + }, + "emailKey": { + "type": "string", + "description": "The environment variable used to get the user's email.", + "examples": [ + "EmailKeyVar" + ] + }, + "passwordKey": { + "type": "string", + "description": "The environment variable containing the user's password.", + "examples": [ + "PasswordKeyVar" + ] + } + }, + "additionalProperties": false, + "required": [ + "personaName", + "emailKey", + "passwordKey" + ] + } + } +} \ No newline at end of file diff --git a/docs/Yaml/testSettings.md b/docs/Yaml/testSettings.md index 28a687433..cce5eeb0f 100644 --- a/docs/Yaml/testSettings.md +++ b/docs/Yaml/testSettings.md @@ -1,12 +1,12 @@ # testSettings -This is used to define settings for the tests in the test plan +Defines settings for the tests in the test plan ## YAML schema definition | Property | Required | Description | | -- | -- | -- | -| locale | Yes | The locale/culture syntax in which the test cases or test steps are written in. See [Global Support in Microsoft Power Fx](https://learn.microsoft.com/en-us/power-platform/power-fx/global) for more info. If unspecified, `CultureInfo.CurrentCulture` will be used for the locale by default for parsing the test steps. | +| locale | Yes | The locale/culture syntax that the test cases or test steps are written in. See [Global Support in Microsoft Power Fx](https://learn.microsoft.com/en-us/power-platform/power-fx/global) for more info. If unspecified, `CultureInfo.CurrentCulture` will be used for the locale by default for parsing the test steps. | | browserConfigurations | Yes | A list of browser configurations to be tested. At least one browser must be specified. | | recordVideo | No | Default is false. If set to true, a video recording of the test is captured. | | headless | No | Default is true. If set to false, the browser will show up during test execution. | diff --git a/docs/Yaml/test.md b/docs/Yaml/testSuite.md similarity index 70% rename from docs/Yaml/test.md rename to docs/Yaml/testSuite.md index f676b00ec..1fadbb231 100644 --- a/docs/Yaml/test.md +++ b/docs/Yaml/testSuite.md @@ -1,6 +1,6 @@ -# test +# testSuite -This is used to define one test. +This is used to define one test, consisting of a number of test cases. ## YAML schema definition @@ -13,9 +13,9 @@ This is used to define one test. | appId | No | This is the id of the app that is to be launched. This is required and used only when app logical name is not present. App id should be used only for canvas apps that are not in the solution | networkRequestMocks | No | Defines network request mocks needed for the test | | testCases | Yes | Defines test cases in the test suite. Test cases contained in test suites are run sequentially. The app state is persisted across all test cases in a suite | -| onTestCaseStart | No | Defines the steps that need to be triggered for every test case in a suite before the case begins executing | -| onTestCaseComplete | No | Defines the steps that need to be triggered for every test case in a suite after the case finishes executing | -| onTestSuiteComplete | No | Defines the steps that need to be triggered after the suite finishes executing | +| onTestCaseStart | No | Defines the steps triggered prior to each test case in a suite | +| onTestCaseComplete | No | Defines the steps triggered post each test case | +| onTestSuiteComplete | No | Defines the steps triggered after the suite finishes executing | ### NetworkRequestMocks @@ -27,7 +27,7 @@ This is used to define one test. | Headers | No | This is a list of header fields in the request in the format of [fieldName : fieldValue] | | requestBodyFile | No | This is a text file with the request body. All text in this file will be read as the request body | -For optional properties, if no value is specified, the routing applies to all. For example, if Method is null, we send back the mock response whatever the method is as long as the other properties all match. +If no values are specified for optional properties, the routing applies to all. For example, if Method is null, we send back the mock response whatever the method is as long as the other properties all match. For Sharepoint/Dataverse/Connector apps, requestURL and Method can be the same for all requests. `x-ms-request-method` and `x-ms-request-url` in headers may need to be configured in that case to identify different requests. @@ -35,13 +35,13 @@ For Sharepoint/Dataverse/Connector apps, requestURL and Method can be the same f | Property | Required | Description | | -- | -- | -- | -| testCaseName | Yes | This is the name of the test case, it will be used in reporting success and failure | -| testCaseDescription | No | Additional information describe what the test case does | -| testSteps | Yes | A set of Power FX functions describing the steps needed to perform the test case | +| testCaseName | Yes | This is the name of the test case, that will be used in reporting success and failure | +| testCaseDescription | No | Additional information describing what the test case does | +| testSteps | Yes | A set of Power Fx functions describing the steps needed to perform the test case | ### TestSteps - This can use any existing [Power Fx](https://docs.microsoft.com/en-us/power-platform/power-fx/overview) functions or [specific test functions](../PowerFX/README.md) defined by this framework. - It should start with a | to allow for multiline YAML expressions followed by an = sign to indicate that it is a Power Fx expression -- Functions should be separated by a ; -- Comments can be used and should start with // +- Functions should be separated by a semicolon (';') character +- Comments are can be used and should start with // (single line comment) or be surrounded by /* ... */.