Skip to content

Commit e7b7307

Browse files
telemetry(webview): Emit toolkit_X module telemetry on auth webview (aws#6791)
## Problem: With our telemetry, we do not know when the frontend webview UI has actually loaded. The current process looks like the following: - We create a webview and set the HTML to load, but after that we do not have a formal way to detect if the webview actually loaded the HTML/JS successfully. We only know that the process started (`toolkit_willOpenModule`) ## Solution: Emit certain metrics during the webview loading process to get a better idea of if the webview UI successfully completed its initial load. - `toolkit_willOpenModule`, indicates intent to render a webview. It does not mean the user is seeing anything. - `toolkit_didLoadModule`, indicates the final result of loading the webview - We know a `result: Succeeded` when the frontend send a successful message to the backend. It knows this by ensuring there were no errors and that a certain HTML element can be found, then once the page finishes its initial load it will send a success message to the backend. - On `result: Failed`, what happens is a timer has timed out after 10 seconds. We assume that since there was no response from the frontend, it failed to fully execute the HTML/JS. - State is shared between `toolkit_willOpenModule` and `toolkit_didLoadModule` so that we can connect them through telemetry. This includes `traceId` and the `duration` which is the time between the 2 metrics. This PR only applies to the Login and Reauth page for now, and future Vue webviews will need to implement some things on their end to get this functionality. ## TODO - Generalize this solution in a more robust way for other webviews to easily implement this functionality --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: nkomonen-amazon <nkomonen@amazon.com>
1 parent b7ce8b4 commit e7b7307

File tree

10 files changed

+246
-38
lines changed

10 files changed

+246
-38
lines changed

packages/core/src/amazonq/webview/messages/messageDispatcher.ts

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ import globals from '../../../shared/extensionGlobals'
1616
import { openUrl } from '../../../shared/utilities/vsCodeUtils'
1717
import { DefaultAmazonQAppInitContext } from '../../apps/initContext'
1818

19+
const qChatModuleName = 'amazonqChat'
20+
1921
export function dispatchWebViewMessagesToApps(
2022
webview: Webview,
2123
webViewToAppsMessagePublishers: Map<TabType, MessagePublisher<any>>
@@ -29,8 +31,8 @@ export function dispatchWebViewMessagesToApps(
2931
* This would be equivalent of the duration between "user clicked open q" and "ui has become available"
3032
* NOTE: Amazon Q UI is only loaded ONCE. The state is saved between each hide/show of the webview.
3133
*/
32-
telemetry.webview_load.emit({
33-
webviewName: 'amazonq',
34+
telemetry.toolkit_didLoadModule.emit({
35+
module: qChatModuleName,
3436
duration: performance.measure(amazonqMark.uiReady, amazonqMark.open).duration,
3537
result: 'Succeeded',
3638
})
@@ -86,12 +88,19 @@ export function dispatchWebViewMessagesToApps(
8688
}
8789

8890
if (msg.type === 'error') {
89-
const event = msg.event === 'webview_load' ? telemetry.webview_load : telemetry.webview_error
90-
event.emit({
91-
webviewName: 'amazonqChat',
92-
result: 'Failed',
93-
reasonDesc: msg.errorMessage,
94-
})
91+
if (msg.event === 'toolkit_didLoadModule') {
92+
telemetry.toolkit_didLoadModule.emit({
93+
module: qChatModuleName,
94+
result: 'Failed',
95+
reasonDesc: msg.errorMessage,
96+
})
97+
} else {
98+
telemetry.webview_error.emit({
99+
webviewName: qChatModuleName,
100+
result: 'Failed',
101+
reasonDesc: msg.errorMessage,
102+
})
103+
}
95104
return
96105
}
97106

packages/core/src/amazonq/webview/ui/main.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ export const createMynahUI = (
6363
const { error, message } = e
6464
ideApi.postMessage({
6565
type: 'error',
66-
event: connector.isUIReady ? 'webview_error' : 'webview_load',
66+
event: connector.isUIReady ? 'webview_error' : 'toolkit_didLoadModule',
6767
errorMessage: error ? error.toString() : message,
6868
})
6969
})

packages/core/src/login/webview/commonAuthViewProvider.ts

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,9 +135,10 @@ export class CommonAuthViewProvider implements WebviewViewProvider {
135135
enableCommandUris: true,
136136
localResourceRoots: [dist, resources],
137137
}
138-
webviewView.webview.html = this._getHtmlForWebview(this.extensionContext.extensionUri, webviewView.webview)
139138
// register the webview server
140139
await this.webView?.setup(webviewView.webview)
140+
141+
webviewView.webview.html = this._getHtmlForWebview(this.extensionContext.extensionUri, webviewView.webview)
141142
}
142143

143144
private _getHtmlForWebview(extensionURI: Uri, webview: vscode.Webview) {

packages/core/src/login/webview/vue/amazonq/backend_amazonq.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ const className = 'AmazonQLoginWebview'
2727
export class AmazonQLoginWebview extends CommonAuthWebview {
2828
public override id: string = 'aws.amazonq.AmazonCommonAuth'
2929
public static sourcePath: string = 'vue/src/login/webview/vue/amazonq/index.js'
30+
public override supportsLoadTelemetry: boolean = true
3031

3132
override onActiveConnectionModified = new vscode.EventEmitter<void>()
3233

packages/core/src/login/webview/vue/backend.ts

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -62,18 +62,25 @@ export abstract class CommonAuthWebview extends VueWebview {
6262
}
6363

6464
private didCall: { login: boolean; reauth: boolean } = { login: false, reauth: false }
65-
public setUiReady(state: 'login' | 'reauth') {
66-
// Prevent telemetry spam, since showing/hiding chat triggers this each time.
67-
// So only emit once.
65+
/**
66+
* Called when the UI load process is completed, regardless of success or failure
67+
*
68+
* @param errorMessage IF an error is caught on the frontend, this is the message. It will result in a failure metric.
69+
* Otherwise we assume success.
70+
*/
71+
public setUiReady(state: 'login' | 'reauth', errorMessage?: string) {
72+
// Only emit once to prevent telemetry spam, since showing/hiding chat triggers this each time.
73+
// TODO: Research how to not trigger this on every show/hide
6874
if (this.didCall[state]) {
6975
return
7076
}
7177

72-
telemetry.webview_load.emit({
73-
passive: true,
74-
webviewName: state,
75-
result: 'Succeeded',
76-
})
78+
if (errorMessage) {
79+
this.setLoadFailure(state, errorMessage)
80+
} else {
81+
this.setDidLoad(state)
82+
}
83+
7784
this.didCall[state] = true
7885
}
7986

packages/core/src/login/webview/vue/login.vue

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,7 @@
145145
></SelectableItem>
146146
<button
147147
class="continue-button"
148+
id="connection-selection-continue-button"
148149
:disabled="selectedLoginOption === 0"
149150
v-on:click="handleContinueClick()"
150151
>
@@ -368,8 +369,6 @@ export default defineComponent({
368369
// Pre-select the first available login option
369370
await this.preselectLoginOption()
370371
await this.handleUrlInput() // validate the default startUrl
371-
372-
await client.setUiReady('login')
373372
},
374373
methods: {
375374
toggleItemSelection(itemId: number) {
@@ -590,6 +589,18 @@ export default defineComponent({
590589
},
591590
},
592591
})
592+
593+
/**
594+
* The ID of the element we will use to determine that the UI has completed its initial load.
595+
*
596+
* This makes assumptions that we will be in a certain state of the UI (eg showing a form vs. a loading bar).
597+
* So if the UI flow changes, this may need to be updated.
598+
*/
599+
export function getReadyElementId() {
600+
// On every initial load, we ASSUME that the user will always be in the connection selection state,
601+
// which is why we specifically look for this button.
602+
return 'connection-selection-continue-button'
603+
}
593604
</script>
594605

595606
<style>

packages/core/src/login/webview/vue/reauthenticate.vue

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ and the final results are retrieved by the frontend. For this Component to updat
6161
</div>
6262

6363
<div>
64-
<button id="reauthenticate" v-on:click="reauthenticate">Re-authenticate</button>
64+
<button id="reauthenticate-button" v-on:click="reauthenticate">Re-authenticate</button>
6565
<div v-if="errorMessage" id="error-message" style="color: red">{{ errorMessage }}</div>
6666
</div>
6767

@@ -127,9 +127,6 @@ export default defineComponent({
127127
128128
this.doShow = true
129129
},
130-
async mounted() {
131-
await client.setUiReady('reauth')
132-
},
133130
methods: {
134131
async reauthenticate() {
135132
client.emitUiClick('auth_reauthenticate')
@@ -148,6 +145,16 @@ export default defineComponent({
148145
},
149146
},
150147
})
148+
149+
/**
150+
* The ID of the element we will use to determine that the UI has completed its initial load.
151+
*
152+
* This makes assumptions that we will be in a certain state of the UI (eg showing a form vs. a loading bar).
153+
* So if the UI flow changes, this may need to be updated.
154+
*/
155+
export function getReadyElementId() {
156+
return 'reauthenticate-button'
157+
}
151158
</script>
152159
<style>
153160
@import './base.css';
@@ -199,7 +206,7 @@ export default defineComponent({
199206
flex-direction: column;
200207
}
201208
202-
button#reauthenticate {
209+
button#reauthenticate-button {
203210
cursor: pointer;
204211
background-color: var(--vscode-button-background);
205212
color: white;
@@ -231,7 +238,7 @@ button#cancel {
231238
cursor: pointer;
232239
}
233240
234-
body.vscode-high-contrast:not(body.vscode-high-contrast-light) button#reauthenticate {
241+
body.vscode-high-contrast:not(body.vscode-high-contrast-light) button#reauthenticate-button {
235242
background-color: white;
236243
color: black;
237244
}

packages/core/src/login/webview/vue/root.vue

Lines changed: 71 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ configure app to AMAZONQ if for Amazon Q login
1616
</template>
1717
<script lang="ts">
1818
import { PropType, defineComponent } from 'vue'
19-
import Login from './login.vue'
20-
import Reauthenticate from './reauthenticate.vue'
19+
import Login, { getReadyElementId as getLoginReadyElementId } from './login.vue'
20+
import Reauthenticate, { getReadyElementId as getReauthReadyElementId } from './reauthenticate.vue'
2121
import { AuthFlowState, FeatureId } from './types'
2222
import { WebviewClientFactory } from '../../../webviews/client'
2323
import { CommonAuthWebview } from './backend'
@@ -51,15 +51,84 @@ export default defineComponent({
5151
})
5252
5353
await this.refreshAuthState()
54+
55+
// We were recieving the 'load' event before refreshAuthState() resolved (I'm assuming behavior w/ Vue + browser loading not blocking),
56+
// so post refreshAuhState() if we detect we already loaded, then execute immediately since the event already happened.
57+
if (didLoad) {
58+
handleLoaded()
59+
} else {
60+
window.addEventListener('load', () => {
61+
handleLoaded()
62+
})
63+
}
5464
},
5565
methods: {
5666
async refreshAuthState() {
5767
await client.refreshAuthState()
5868
this.authFlowState = await client.getAuthState()
69+
70+
// Used for telemetry purposes
71+
if (this.authFlowState === 'LOGIN') {
72+
;(window as any).uiState = 'login'
73+
;(window as any).uiReadyElementId = getLoginReadyElementId()
74+
} else if (this.authFlowState && this.authFlowState !== undefined) {
75+
;(window as any).uiState = 'reauth'
76+
;(window as any).uiReadyElementId = getReauthReadyElementId()
77+
}
78+
5979
this.refreshKey += 1
6080
},
6181
},
6282
})
83+
84+
// ---- START ---- The following handles the process of indicating the UI has loaded successfully.
85+
// TODO: Move this in to a reusable class for other webviews, it feels a bit messy here
86+
let didSetReady = false
87+
88+
// Setup error handlers to report. This may not actually be able to catch certain errors that we'd expect,
89+
// so this may have to be revisited.
90+
window.onerror = function (message) {
91+
if (didSetReady) {
92+
return
93+
}
94+
95+
setUiReady((window as any).uiState, message.toString())
96+
}
97+
document.addEventListener(
98+
'error',
99+
(e) => {
100+
if (didSetReady) {
101+
return
102+
}
103+
104+
setUiReady((window as any).uiState, e.message)
105+
},
106+
true
107+
)
108+
109+
let didLoad = false
110+
window.addEventListener('load', () => {
111+
didLoad = true
112+
})
113+
const handleLoaded = () => {
114+
// in case some unexpected behavior triggers this flow again, skip since we already emitted for this instance
115+
if (didSetReady) {
116+
return
117+
}
118+
119+
const foundElement = !!document.getElementById((window as any).uiReadyElementId)
120+
if (!foundElement) {
121+
setUiReady((window as any).uiState, `Could not find element: ${(window as any).uiReadyElementId}`)
122+
} else {
123+
// Successful load!
124+
setUiReady((window as any).uiState)
125+
}
126+
}
127+
const setUiReady = (state: 'login' | 'reauth', errorMessage?: string) => {
128+
client.setUiReady(state, errorMessage)
129+
didSetReady = true
130+
}
131+
// ---- END ----
63132
</script>
64133
<style>
65134
body {

packages/core/src/shared/telemetry/vscodeTelemetry.json

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -962,16 +962,6 @@
962962
],
963963
"passive": true
964964
},
965-
{
966-
"name": "webview_load",
967-
"description": "Represents a webview load event",
968-
"metadata": [
969-
{
970-
"type": "webviewName",
971-
"required": true
972-
}
973-
]
974-
},
975965
{
976966
"name": "webview_error",
977967
"description": "Represents an error that occurs in a webview",

0 commit comments

Comments
 (0)