telemetry(amazonq): calculate % of non-generated (user-written) code #5991

leigaol · 2024-11-13T01:16:41Z

Problem

With the release of many Q features(Inline Suggestion, chat, inline chat, /dev, /test, /doc, /review, /transform), we need to know the % code written by all Q features. This requires calculating and reporting the user written code. The reporting of the code contribution of each Q features was already implemented.

Solution

Calculate and report the user written code for each language by listening to document change events while Q is not making changes to the editor.

We add flags to know whether Q is making temporary changes for suggestion rendering or Q suggestion is accepted, by doing so, the document change events are coming from the user.

We ignore certain document changes when their length of new characters exceeds 50. Previous data driven research has shown that user tend to copy a huge file from one place to another, making the user written code count skyrocketing but that is actually some existing code not written by the user.

We plan to first collect data from IDEs and let it run in the background in shadow mode before we finish the service side aggregation, fix possible bugs and eventually present the AI code written % to the customers.

Note: The JB PR aws/aws-toolkit-jetbrains#5215. The JB implementation depends on a reliable JB internal message bus to pass information. Using VSC event listener might mess up the boolean state of Q editing or not.

License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

github-actions · 2024-11-13T01:17:00Z

This pull request modifies code in src/* but no tests were added/updated.
- Confirm whether tests should be added or ensure the PR description explains why tests are not required.
This pull request implements a feat or fix, so it must include a changelog entry (unless the fix is for an unreleased feature). Review the changelog guidelines.
- Note: beta or "experiment" features that have active users should announce fixes in the changelog.
- If this is not a feature or fix, use an appropriate type from the title guidelines. For example, telemetry-only changes should use the telemetry type.

justinmk3 · 2024-12-16T17:28:50Z

⛔️ Not planned for tomorrow's release.

packages/amazonq/test/unit/codewhisperer/tracker/userWrittenCodeTracker.test.ts

packages/core/src/codewhisperer/commands/basicCommands.ts

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts

…deTracker.test.ts Co-authored-by: Justin M. Keyes <jmkeyes@amazon.com>

justinmk3

Approving with the expectation that this followup work with land soon:

Our expectation is to completely get rid of the old CodeCoverageTracker.ts in a few weeks.

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts

…i_gen

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts

…ws#5991 ## Problem With the release of many Q features(Inline Suggestion, chat, inline chat, /dev, /test, /doc, /review, /transform), we need to know the % code written by all Q features. This requires calculating and reporting the user written code. The reporting of the code contribution of each Q features was already implemented. ## Solution Calculate and report the user written code for each language by listening to document change events while Q is not making changes to the editor. We add flags to know whether Q is making temporary changes for suggestion rendering or Q suggestion is accepted, by doing so, the document change events are coming from the user. We ignore certain document changes when their length of new characters exceeds 50. Previous data driven research has shown that user tend to copy a huge file from one place to another, making the user written code count skyrocketing but that is actually some existing code not written by the user. We plan to first collect data from IDEs and let it run in the background in shadow mode before we finish the service side aggregation, fix possible bugs and eventually present the AI code written % to the customers. Note: The JB PR aws/aws-toolkit-jetbrains#5215. The JB implementation depends on a reliable JB internal message bus to pass information. Using VSC event listener might mess up the boolean state of Q editing or not.

With the release of many Q features(Inline Suggestion, chat, inline chat, /dev, /test, /doc, /review, /transform), we need to know the % code written by all Q features. This requires calculating and reporting the user written code. The reporting of the code contribution of each Q features was already implemented. % Code Written by Q = Code Written by Q / ( Code Written by Q + Code Written by User) Ref: aws/aws-toolkit-vscode#5991 Calculate and report the user written code for each language by listening to document change events while Q is not making changes to the editor. We add flags to know whether Q is making temporary changes for suggestion rendering or Q suggestion is accepted, by doing so, the document change events are coming from the user. We ignore certain document changes when their length of new characters exceeds 50. Previous data driven research has shown that user tend to copy a huge file from one place to another, making the user written code count skyrocketing but that is actually some existing code not written by the user. We plan to first collect data from IDEs and let it run in the background in shadow mode before we finish the service side aggregation, fix possible bugs and eventually present the AI code written % to the customers.

…ws#5991 ## Problem With the release of many Q features(Inline Suggestion, chat, inline chat, /dev, /test, /doc, /review, /transform), we need to know the % code written by all Q features. This requires calculating and reporting the user written code. The reporting of the code contribution of each Q features was already implemented. ## Solution Calculate and report the user written code for each language by listening to document change events while Q is not making changes to the editor. We add flags to know whether Q is making temporary changes for suggestion rendering or Q suggestion is accepted, by doing so, the document change events are coming from the user. We ignore certain document changes when their length of new characters exceeds 50. Previous data driven research has shown that user tend to copy a huge file from one place to another, making the user written code count skyrocketing but that is actually some existing code not written by the user. We plan to first collect data from IDEs and let it run in the background in shadow mode before we finish the service side aggregation, fix possible bugs and eventually present the AI code written % to the customers. Note: The JB PR aws/aws-toolkit-jetbrains#5215. The JB implementation depends on a reliable JB internal message bus to pass information. Using VSC event listener might mess up the boolean state of Q editing or not.

leigaol requested review from a team as code owners November 13, 2024 01:16

leigaol marked this pull request as draft November 13, 2024 01:16

reduce system status poll freq

6363f66

leigaol force-pushed the ai_gen branch from 393a9a2 to d6aabd9 Compare December 10, 2024 18:48

leigaol added 5 commits December 11, 2024 10:48

Merge branch 'master' of github.com:leigaol/aws-toolkit-vscode

6d36cbe

ai gen

ce1733a

ai code gen

0036757

more code

dd72f01

update

ad0e018

leigaol force-pushed the ai_gen branch from d6aabd9 to ad0e018 Compare December 11, 2024 18:49

leigaol and others added 9 commits December 11, 2024 13:04

update

288617f

minimize changes

26150f4

update

d99e4d1

fix cyclic dependency

f75c58a

add unit test

d2c9ff1

fix test case

7b6f975

use runtime language

8eaf48d

simplify changes

69b374c

Merge branch 'aws:master' into ai_gen

c6e2161

leigaol requested review from zixlin7 and Will-ShaoHua December 12, 2024 23:09

lint

cf1c069

leigaol marked this pull request as ready for review December 13, 2024 18:05

leigaol requested review from a team as code owners December 13, 2024 18:05

leigaol mentioned this pull request Dec 19, 2024

telemetry(amazonq): AI code gen % for Q features aws/aws-toolkit-jetbrains#5215

Merged

6 tasks

merge

32a4af5

justinmk3 reviewed Jan 7, 2025

View reviewed changes

packages/amazonq/test/unit/codewhisperer/tracker/userWrittenCodeTracker.test.ts Outdated Show resolved Hide resolved

justinmk3 reviewed Jan 7, 2025

View reviewed changes

packages/core/src/codewhisperer/commands/basicCommands.ts Show resolved Hide resolved

justinmk3 reviewed Jan 7, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Show resolved Hide resolved

leigaol and others added 2 commits January 7, 2025 15:56

Update packages/amazonq/test/unit/codewhisperer/tracker/userWrittenCo…

3781ef9

…deTracker.test.ts Co-authored-by: Justin M. Keyes <jmkeyes@amazon.com>

update Q editing reset timer

97e8d56

justinmk3 approved these changes Jan 14, 2025

View reviewed changes

Will-ShaoHua reviewed Jan 14, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Outdated Show resolved Hide resolved

Will-ShaoHua reviewed Jan 14, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Show resolved Hide resolved

Will-ShaoHua reviewed Jan 14, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Outdated Show resolved Hide resolved

Merge branch 'master' of github.com:leigaol/aws-toolkit-vscode into a…

57260e8

…i_gen

Will-ShaoHua reviewed Jan 14, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Show resolved Hide resolved

add comments

f9a9b00

Will-ShaoHua reviewed Jan 14, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Show resolved Hide resolved

Will-ShaoHua reviewed Jan 15, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Outdated Show resolved Hide resolved

Will-ShaoHua reviewed Jan 15, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Outdated Show resolved Hide resolved

Will-ShaoHua reviewed Jan 15, 2025

View reviewed changes

packages/core/src/codewhisperer/tracker/userWrittenCodeTracker.ts Show resolved Hide resolved

Merge branch 'master' into ai_gen

955b9f6

Will-ShaoHua self-requested a review January 15, 2025 23:37

feedback fixes

f65acde

Will-ShaoHua approved these changes Jan 15, 2025

View reviewed changes

fix ci

7a724ab

leigaol removed the request for review from zixlin7 January 16, 2025 00:40

leigaol changed the title ~~feat(amazonq):AI code gen % for Q features~~ telemetry(amazonq):AI code gen % for Q features Jan 16, 2025

justinmk3 merged commit f805f9b into aws:master Jan 16, 2025
25 of 26 checks passed

justinmk3 changed the title ~~telemetry(amazonq):AI code gen % for Q features~~ telemetry(amazonq): calculate % of non-generated (user-written) code Jan 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

telemetry(amazonq): calculate % of non-generated (user-written) code #5991

telemetry(amazonq): calculate % of non-generated (user-written) code #5991

leigaol commented Nov 13, 2024 •

edited

Loading

github-actions bot commented Nov 13, 2024

justinmk3 commented Dec 16, 2024 •

edited

Loading

justinmk3 left a comment

telemetry(amazonq): calculate % of non-generated (user-written) code #5991

telemetry(amazonq): calculate % of non-generated (user-written) code #5991

Conversation

leigaol commented Nov 13, 2024 • edited Loading

Problem

Solution

github-actions bot commented Nov 13, 2024

justinmk3 commented Dec 16, 2024 • edited Loading

justinmk3 left a comment

Choose a reason for hiding this comment

leigaol commented Nov 13, 2024 •

edited

Loading

justinmk3 commented Dec 16, 2024 •

edited

Loading