Table Viewer optimization in ITC #1466

danila-grobov · 2025-05-03T16:54:03Z

Summary
Currently the table viewer is not usable for ITC connection, especially if you have a lot of columns, it takes like 20 seconds if not more to load. Related to the issue #1439 and #1233

I've also fixed #1132 in this. The table looks correct now:

I've mostly fixed that by making use of ADO DB connector that SAS includes with interop library. I borrowed a bunch of setup for this from #1388, so changes in CodeRunner for example are just a copy paste from that PR.

I was made aware of the crashing issue that others have faced with using this connector, however I didn't envounter it and I've tried oppening up a bunch of different tables.

I've also added a column type detection function, which I ripped from SAS Studio. So it interprets the formats in the same way and now we can see all of the beautiful icons.

Now it takes around 7 seconds including column loading, compared to 5 seconds that it takes in Enterprise Guide for the same table.

I did decide to stop at this point just to see what you guys think and release this into the wild. If this goes well, we can look into improving this further. There are at least three areas of pottential improvement:

Adding loading indicators, which I think would make it seem a bit more responsive.
Fetching metadata of the table and the data at the same time. Which I think would require a bunch of changes in ITC adapter architecture, because currently it is setup to allow only a single thing happening at the same time.
Reducing the amount of initial columns. This trick is used in all other SAS clients, so might as well.

Testing

I don't have access to locally running SAS, so all of this has been tested on a remote IOM connection to a server running on RHEL.

Opening a table with contents that are more than 100 row limit
Opening a table with contents that are less than 100 row limit
Opening an empty table
Downloading a table
Scrolling through a large table
Test encoding
Test table with a single column
Test table with multiple columns
Test table with a single row

I, Danila Grobov (s4642g) <danila.grobov@seb.se>, hereby add my Signed-off-by to this commit: 176ab4b I, Danila Grobov (s4642g) <danila.grobov@seb.se>, hereby add my Signed-off-by to this commit: c12016f I, Danila Grobov (s4642g) <danila.grobov@seb.se>, hereby add my Signed-off-by to this commit: 9d1aff6 I, Danila Grobov (s4642g) <danila.grobov@seb.se>, hereby add my Signed-off-by to this commit: 328b22d Signed-off-by: Danila Grobov (s4642g) <danila.grobov@seb.se>

Signed-off-by: Danila Grobov (s4642g) <danila.grobov@seb.se>

scnwwu

Thank you @danila-grobov, overall looks great!

So it appears that 512 # adCmdTableDirect doesn't crash. I used general command select * from in my previous trial #1132 (comment) and it will crash.

client/src/connection/itc/script.ts

DCO Remediation Commit for Danila Grobov (s4642g) <danila.grobov@seb.se> I, Danila Grobov (s4642g) <danila.grobov@seb.se>, hereby add my Signed-off-by to this commit: 2ab7999 Signed-off-by: Danila Grobov (s4642g) <danila.grobov@seb.se>

Signed-off-by: Danila Grobov (s4642g) <danila.grobov@seb.se>

danila-grobov · 2025-05-11T20:34:12Z

@scnwwu Fixed the tests: cd0ef99

It got a bit messy with mocking of the uuid 😅

Signed-off-by: Danila Grobov <danila.grob@gmail.com>

…zation Signed-off-by: Danila Grobov <danila.grob@gmail.com>

danila-grobov

@scnwwu Third time's the charm, I promise the tests will pass now.

Signed-off-by: Danila Grobov <danila.grob@gmail.com>

danila-grobov · 2025-05-13T18:56:18Z

@scnwwu Fixed linting issues.

client/src/connection/itc/script.ts

scnwwu · 2025-05-19T02:18:01Z

Hi @danila-grobov, now that #1388 is merged. Please rebase your change with latest main branch. Thanks.
I'm not clear why you need to mock uuid while #1388 seems fine.

…zation Signed-off-by: Danila Grobov <danila.grob@gmail.com>

DmitryMK · 2025-05-30T20:28:42Z

I have tested the ITC connection with the latest version, works good and it is incredibly faster comparing to the previous approach. Great work @danila-grobov.

DmitryMK · 2025-06-02T20:17:55Z

@scnwwu @scottdover I have rewritten getLibraries and getTables using the same approach @danila-grobov used and it changed loading time of libraries from seconds to milliseconds in my case (which in my opinion makes user experience much better). I think it will be nice to be released together with this change. Should I submit a separate PR or add a commit in this one?

scnwwu · 2025-06-03T01:44:05Z

@scnwwu @scottdover I have rewritten getLibraries and getTables using the same approach @danila-grobov used and it changed loading time of libraries from seconds to milliseconds in my case (which in my opinion makes user experience much better). I think it will be nice to be released together with this change. Should I submit a separate PR or add a commit in this one?

Thank you. That would be a good improvement. This PR is from @danila-grobov's forked repo I'm not sure if you're able to add a commit to the branch. I guess you can create a PR to @danila-grobov's branch and if @danila-grobov approved and merged it, it will be in this PR. Or you can create a separate PR after this one merged.

DmitryMK · 2025-06-03T02:27:16Z

@scnwwu @scottdover I have rewritten getLibraries and getTables using the same approach @danila-grobov used and it changed loading time of libraries from seconds to milliseconds in my case (which in my opinion makes user experience much better). I think it will be nice to be released together with this change. Should I submit a separate PR or add a commit in this one?

Thank you. That would be a good improvement. This PR is from @danila-grobov's forked repo I'm not sure if you're able to add a commit to the branch. I guess you can create a PR to @danila-grobov's branch and if @danila-grobov approved and merged it, it will be in this PR. Or you can create a separate PR after this one merged.

In this case I think it will be better if I create a separate PR, I'll try to do it tomorrow.

DmitryMK

Everything looks good, the date/time format is nice to have.

DmitryMK · 2025-06-03T14:58:36Z

client/src/connection/util.ts

+      "MONTH",
+      "MON",
+      "DOWNAME",
+    ].some((f) => format.includes(f)) &&


Can we add 8601 formats? They are common in clinical programming.

Date: E8601DA, E8601DN, D8601DA, D8601DN: ([B|E]8601D[N|A])
Datetime: ([B|E]8601(D[T|Z|X]|LX))
Time: ([B|E]8601(T[M|Z|X]|LZ))

@scnwwu: Perhaps enhancing SAS format support to include the lateset available SAS-provided formats should be moved into a dedicated pull request?

@DmitryMK: Also wanted to create awarenes for the fact the the doc link mentioned in the previous comment for National Language Support formats is really old. Here are doc links that will always point to the latest production:

doc link for Dictionary of Formats

doc link for the Dictionary of Formats for NLS.

Tip: Use .../default/... instead of .../v_xxx/... in SAS Viya 4 doc links, so they resolve to the latest documentation.

…ssoftware#1502) Bumps the docusaurus group in /website with 4 updates: [@docusaurus/preset-classic](https://github.com/facebook/docusaurus/tree/HEAD/packages/docusaurus-preset-classic), [@docusaurus/theme-mermaid](https://github.com/facebook/docusaurus/tree/HEAD/packages/docusaurus-theme-mermaid), [@docusaurus/module-type-aliases](https://github.com/facebook/docusaurus/tree/HEAD/packages/docusaurus-module-type-aliases) and [@docusaurus/tsconfig](https://github.com/facebook/docusaurus/tree/HEAD/packages/docusaurus-tsconfig). Updates `@docusaurus/preset-classic` from 3.7.0 to 3.8.0 - [Release notes](https://github.com/facebook/docusaurus/releases) - [Changelog](https://github.com/facebook/docusaurus/blob/main/CHANGELOG.md) - [Commits](https://github.com/facebook/docusaurus/commits/v3.8.0/packages/docusaurus-preset-classic) Updates `@docusaurus/theme-mermaid` from 3.7.0 to 3.8.0 - [Release notes](https://github.com/facebook/docusaurus/releases) - [Changelog](https://github.com/facebook/docusaurus/blob/main/CHANGELOG.md) - [Commits](https://github.com/facebook/docusaurus/commits/v3.8.0/packages/docusaurus-theme-mermaid) Updates `@docusaurus/module-type-aliases` from 3.7.0 to 3.8.0 - [Release notes](https://github.com/facebook/docusaurus/releases) - [Changelog](https://github.com/facebook/docusaurus/blob/main/CHANGELOG.md) - [Commits](https://github.com/facebook/docusaurus/commits/v3.8.0/packages/docusaurus-module-type-aliases) Updates `@docusaurus/tsconfig` from 3.7.0 to 3.8.0 - [Release notes](https://github.com/facebook/docusaurus/releases) - [Changelog](https://github.com/facebook/docusaurus/blob/main/CHANGELOG.md) - [Commits](https://github.com/facebook/docusaurus/commits/v3.8.0/packages/docusaurus-tsconfig) --- updated-dependencies: - dependency-name: "@docusaurus/preset-classic" dependency-version: 3.8.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: docusaurus - dependency-name: "@docusaurus/theme-mermaid" dependency-version: 3.8.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: docusaurus - dependency-name: "@docusaurus/module-type-aliases" dependency-version: 3.8.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: docusaurus - dependency-name: "@docusaurus/tsconfig" dependency-version: 3.8.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: docusaurus ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…sassoftware#1504) Bumps the dev group with 3 updates in the / directory: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node), [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) and [eslint](https://github.com/eslint/eslint). Updates `@types/node` from 22.15.19 to 22.15.29 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) Updates `@typescript-eslint/eslint-plugin` from 8.32.1 to 8.33.0 - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.33.0/packages/eslint-plugin) Updates `@typescript-eslint/parser` from 8.32.1 to 8.33.0 - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.33.0/packages/parser) Updates `eslint` from 9.27.0 to 9.28.0 - [Release notes](https://github.com/eslint/eslint/releases) - [Changelog](https://github.com/eslint/eslint/blob/main/CHANGELOG.md) - [Commits](eslint/eslint@v9.27.0...v9.28.0) --- updated-dependencies: - dependency-name: "@types/node" dependency-version: 22.15.29 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: dev - dependency-name: "@typescript-eslint/eslint-plugin" dependency-version: 8.33.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: dev - dependency-name: "@typescript-eslint/parser" dependency-version: 8.33.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: dev - dependency-name: eslint dependency-version: 9.28.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: dev ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Zhirong2022 · 2025-06-09T06:23:10Z

Tested the PR with connection type COM and IOM

It can show table in correct encoding (SAS.viewTable is not showing data with correct encoding #1132)
It has a big improvement of loading large size table

scnwwu · 2025-06-09T06:26:57Z

@danila-grobov would you resolve the conflict and I'll merge it. Thank you.

danila-grobov · 2025-06-09T06:48:28Z

@scnwwu will do later today

…zation

Signed-off-by: Danila Grobov <danila.grob@gmail.com>

danila-grobov · 2025-06-09T21:04:13Z

@scnwwu Done, you can go ahead and merge.

scnwwu · 2025-06-10T01:37:36Z

Thank you again. Great contribution!

DanilaGrobovSEB and others added 7 commits April 13, 2025 20:53

Add changes from file explorer feature.

176ab4b

Use data connection instead of code execution.

c12016f

Improve columns loading time.

9d1aff6

Add column type mapping to the format.

328b22d

Merge branch 'main' into table_viewer_optimization

70069a6

Support empty arrays in json

16eedec

Signed-off-by: Danila Grobov (s4642g) <danila.grobov@seb.se>

scnwwu reviewed May 6, 2025

View reviewed changes

client/src/connection/itc/script.ts Outdated Show resolved Hide resolved

client/src/connection/itc/script.ts Show resolved Hide resolved

client/src/connection/itc/script.ts Outdated Show resolved Hide resolved

scnwwu requested a review from scottdover May 6, 2025 06:09

DanilaGrobovSEB and others added 3 commits May 6, 2025 22:43

Fix incorrect label.

2ab7999

Remove unnecessary parameters in data connection

e78ff54

DCO Remediation Commit for Danila Grobov (s4642g) <danila.grobov@seb.se> I, Danila Grobov (s4642g) <danila.grobov@seb.se>, hereby add my Signed-off-by to this commit: 2ab7999 Signed-off-by: Danila Grobov (s4642g) <danila.grobov@seb.se>

Merge branch 'main' into table_viewer_optimization

5383d38

danila-grobov requested a review from scnwwu May 6, 2025 20:19

Merge branch 'main' into table_viewer_optimization

ed6a9ed

This was linked to issues May 9, 2025

view large dataset is much slower than SAS 9.4 & SAS Enterprise guide when connecting to local SAS #1233

Closed

Table Viewer - Lazy Loading/Pagination #1439

Closed

Fix tests

cd0ef99

Signed-off-by: Danila Grobov (s4642g) <danila.grobov@seb.se>

danila-grobov added 2 commits May 13, 2025 00:04

Use proxyquire for mocking uuid.

019bd3d

Signed-off-by: Danila Grobov <danila.grob@gmail.com>

Merge remote-tracking branch 'upstream/main' into table_viewer_optimi…

d39ce5d

…zation Signed-off-by: Danila Grobov <danila.grob@gmail.com>

danila-grobov commented May 12, 2025

View reviewed changes

danila-grobov added 2 commits May 13, 2025 21:45

Merge branch 'main' into table_viewer_optimization

bbcc56d

Fix linting problems.

192c22f

Signed-off-by: Danila Grobov <danila.grob@gmail.com>

DmitryMK reviewed May 14, 2025

View reviewed changes

client/src/connection/itc/script.ts Show resolved Hide resolved

DmitryMK reviewed May 14, 2025

View reviewed changes

client/src/connection/itc/script.ts Show resolved Hide resolved

DanilaGrobovSEB and others added 3 commits May 25, 2025 20:34

Merge remote-tracking branch 'upstream/main' into table_viewer_optimi…

7ec2b55

…zation Signed-off-by: Danila Grobov <danila.grob@gmail.com>

Fix package-lock

7e4d8a2

Bring the file explorer changes back.

b5785aa

Merge branch 'main' into table_viewer_optimization

3e92c0d

scnwwu approved these changes Jun 3, 2025

View reviewed changes

scnwwu added the verification-needed label Jun 3, 2025

scnwwu assigned enzhihpp Jun 3, 2025

scnwwu added this to the 1.15.0 milestone Jun 3, 2025

DmitryMK approved these changes Jun 3, 2025

View reviewed changes

dependabot bot added 2 commits June 3, 2025 21:31

enzhihpp assigned Zhirong2022 and unassigned enzhihpp Jun 4, 2025

DmitryMK mentioned this pull request Jun 4, 2025

Library Viewer optimization in ITC #1508

Open

Zhirong2022 added testing Test the pull requests and removed verification-needed labels Jun 5, 2025

Zhirong2022 approved these changes Jun 9, 2025

View reviewed changes

Zhirong2022 added testing-complete Complete the pull requests testing and removed testing Test the pull requests labels Jun 9, 2025

DanilaGrobovSEB and others added 2 commits June 9, 2025 23:48

Merge remote-tracking branch 'upstream/main' into table_viewer_optimi…

ea8e546

…zation

Fix formatting.

6260bdc

Signed-off-by: Danila Grobov <danila.grob@gmail.com>

scnwwu merged commit 9d55dce into sassoftware:main Jun 10, 2025
2 checks passed

Zhirong2022 added ready for release Code pushed, but not released in VS code marketplace yet and removed testing-complete Complete the pull requests testing labels Jun 10, 2025

Table Viewer optimization in ITC #1466

Table Viewer optimization in ITC #1466

Uh oh!

Conversation

danila-grobov commented May 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scnwwu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danila-grobov commented May 11, 2025

Uh oh!

danila-grobov left a comment

Choose a reason for hiding this comment

Uh oh!

danila-grobov commented May 13, 2025

Uh oh!

Uh oh!

Uh oh!

scnwwu commented May 19, 2025

Uh oh!

DmitryMK commented May 30, 2025

Uh oh!

DmitryMK commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scnwwu commented Jun 3, 2025

Uh oh!

DmitryMK commented Jun 3, 2025

Uh oh!

DmitryMK left a comment

Choose a reason for hiding this comment

Uh oh!

DmitryMK Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

snlwih Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zhirong2022 commented Jun 9, 2025

Uh oh!

scnwwu commented Jun 9, 2025

Uh oh!

danila-grobov commented Jun 9, 2025

Uh oh!

danila-grobov commented Jun 9, 2025

Uh oh!

Uh oh!

scnwwu commented Jun 10, 2025

Uh oh!

Uh oh!

danila-grobov commented May 3, 2025 •

edited

Loading

DmitryMK commented Jun 2, 2025 •

edited

Loading

snlwih Jun 5, 2025 •

edited

Loading