Reduce contracts and deployments loading time #4233

samaradel · 2025-06-12T12:56:55Z

Description

Deployments:

Load all deployments in parallel
Handle migrations in parallel if needed
Add grid client cache
Add batch processing utility
Create grid clients in parallel with caching
Load contracts and node IDs in parallel for each machine
Process consumptions in batches
Process WireGuard configs in batches

Contracts:

Node info caching
Batch processing for contract normalization, just like Consumptions and WireGuard
Parallel data fetching

Changes

Before: deployments loading time takes around 20 to 25 seconds
After: deployments, loading time takes around 15 to 20 seconds

Related Issues

Profile Contract / Deployment Loading [3 days] #4223

Tested Scenarios

Check deployments loading Time taken in the console for deployments and contracts

Ramez test cases: #4233 (review)

Documentation PR

For UI changes, Please provide the Documentation PR on info_grid

To consider

Preliminary Checks:

Preliminary Checks
- Does it completely address the issue linked?
- What about edge cases?
- Does it meet the specified acceptance criteria?
- Are there any unintended side effects?
- Does the PR adhere to the team's coding conventions, style guides, and best practices?
- Does it integrate well with existing features?
- Does it impact the overall performance of the application?
- Are there any bottlenecks or slowdowns?
- Has it been optimized for efficiency?
- Has it been adequately tested with unit, integration, and end-to-end tests?
- Are there any known defects or issues?
- Is the code well-documented?
- Are changes to documentation reflected in the code?

UI Checks:

UI Checks
- If a UI design is provided/ does it follow it?
- Does every button work?
- Is the data displayed logical? Is it what you expected?
- Does every validation work?
- Does this interface feel intuitive?
- What about slow network? Offline?
- What if the gridproxy/graphql/chain is failing?
- What would a first time user have a hard time navigating here?

Code Quality Checks:

Code Quality Checks
- Code formatted/linted? Are there unused imports? .. etc
- Is there redundant/repeated code?
- Are there conditionals that are always true or always false?
- Can we write this more concisely?
- Can we reuse this code? If yes, where?
- Will the changes be easy to maintain and update in the future?
- Can this code become too complex to understand for other devs?
- Can this code cause future integration problems?

Testing Checklist

Does it Meet the specified acceptance criteria.
Test if changes can affect any other functionality.
Tested with unit, integration, and end-to-end tests.
Tested the un-happy path and rollback scenarios.
Documentation updated to meet the latest changes.
Check automated tests working successfully with the changes.
Can be covered by automated tests.

General Checklist

- Handle migrations in parallel if needed - Add grid client cache - Add batch processing utility - Create grid clients in parallel with caching - Load contracts and node IDs in parallel for each machine - Process consumptions in batches - Process wireguard configs in batches

packages/playground/src/components/k8s_deployment_table.vue

0oM4R

please use allsettled instead of promise.all for better error handling

packages/playground/src/components/k8s_deployment_table.vue

packages/playground/src/components/vm_deployment_table.vue

packages/playground/src/utils/load_deployment.ts

0oM4R

there are many places that are still using promise.all

please make sure to use promise.allsettled everywhere

packages/playground/src/weblets/tf_contracts_list.vue

0oM4R · 2025-06-23T08:19:06Z

packages/playground/src/weblets/tf_contracts_list.vue

+
+  const normalizedContracts: NormalizedContract[] = [];
+  for (const batch of batches) {
+    const results = await Promise.all(


also here still using promise.all

packages/playground/src/components/k8s_deployment_table.vue

packages/playground/src/components/vm_deployment_table.vue

0oM4R · 2025-06-23T08:22:53Z

there are other places please make sure to check them as well and replace the promise.all with promise.allsettled

0oM4R

partial review will review packages/playground/src/utils/load_deployment.ts
later

packages/playground/src/weblets/tf_contracts_list.vue

0oM4R · 2025-06-25T10:50:26Z

packages/playground/src/weblets/tf_contracts_list.vue

+async function getNodeInfoWithCache(nodeIds: number[]) {
+  const uncachedIds = nodeIds.filter(id => !nodeInfoCache.has(id));
+  if (uncachedIds.length > 0) {
+    const newInfo = await getNodeInfo(uncachedIds, []);
+    for (const [id, info] of Object.entries(newInfo)) {
+      nodeInfoCache.set(Number(id), info);
+    }
+  }
+  return nodeIds.map(id => nodeInfoCache.get(id));
+}


This function should be moved to the utils, imo

Need to add error handling for getNodeInfo

I have a concern about invalidating the cache as well

packages/playground/src/components/k8s_deployment_table.vue

0oM4R · 2025-06-25T11:14:53Z

packages/playground/src/components/k8s_deployment_table.vue

+      })
+      .then(res => res[0])
+      .catch(() => undefined);
+  } finally {


Please add a catch block; to at least log the errors

packages/playground/src/components/vm_deployment_table.vue

0oM4R · 2025-06-25T11:19:24Z

packages/playground/src/components/vm_deployment_table.vue

+    const results = await Promise.allSettled([
+      loadVms(grid!),
+      loadVms(updateGrid(grid!, { projectName: props.projectName.toLowerCase() })),
+      showAllDeployments.value && props.projectName.toLowerCase() === ProjectName.VM.toLowerCase()


we may group the chunk 3 condition in both files into a separate variable then use it here
example

loadChunk3 ? loadVms(updateGrid(grid!, { projectName: "" })) : Promise.resolve({ count: 0, items: [], failedDeployments: [] }),

0oM4R · 2025-06-25T11:29:56Z

packages/playground/src/components/vm_deployment_table.vue

+      await Promise.allSettled([
+        chunk1.count > 0 && migrateModule(grid!.gateway),
+        chunk2.count > 0 && migrateModule(grid!.gateway),
+        chunk3.count > 0 && migrateModule(grid!.gateway),


i do not understand why we are calling the migrateModule threetimes with the same args,
also the && means if a chunk's count is 0, false is passed to Promise.allSettled() instead of a Promise, which works but isn't ideal.
please use

chunk1.count > 0 ? migrateModule(x) : Promise.resolve(),

0oM4R · 2025-06-25T11:33:14Z

packages/playground/src/utils/batch_process.ts

+  const results: R[] = [];
+  for (let i = 0; i < items.length; i += batchSize) {
+    const batch = items.slice(i, i + batchSize);
+    const batchResults = await processFn(batch);
+    results.push(...batchResults);
+  }
+  return results;
+}


we can run the batches in parallel. Also please add error handling and unit test for it

- Call getTitalCost function only if there is no type defined

0oM4R

partal review

packages/playground/src/utils/batch_process.ts

packages/playground/src/utils/get_nodeInfo_with_cache.ts

packages/playground/src/components/vm_deployment_table.vue

- Remove get_nodeInfo_with_cache file and use it as a seperate function - Create test unit for getNodeInfoWithCache function - Extract helper functions to reduce the nesting

0oM4R

in vm the network got loaded twice, any idea

…dules

samaradel · 2025-07-13T17:03:09Z

@0oM4R fixed now

ramezsaeed

Test Scenarios:

Test loading contracts time in contract page when have 20 contracts in the account and compare the old time Vs the new time.
Test deploy a patch VMs and compare time for list deployment and for contract page.
Deploy a Full VM with all networks enabled and compare listing time Vs new time after this PR.
Test if we can have a failed deployment and how the list deployments time will react.

Additional tests:

Test list deployment time when have VMs in grace period.
Test list deployment time when have VMs on offline nodes.

0oM4R

i have 38 deployment but i can list 41 only

on https://dashboard.dev.grid.tf/

0oM4R · 2025-07-15T12:32:06Z

I tried to make the deployment listing fail

The error is not accurate, imo
original

new branch

0oM4R · 2025-07-15T12:35:57Z

listing contract have almost no enhancment

listing 74 (node only) contract
the original branch 19.877s
the new branch 19.094s

list deployment has a significant enhancement

listing 10 micro vm deployments

original 35.7s
old 71.8155s
jitsi deployments
new branch 43.15s
old 75.9s

add 31 name contract so the total now 100 node contract and 31 name contract

new branch 38.211 seconds
old one 50.6765s

ramezsaeed · 2025-07-17T15:06:57Z

Tested:

Loading time enhanced with 50 VM contracts

samaradel · 2025-07-20T06:07:47Z

@0oM4R I see the pagination error and loading deployments timeout exist in development, for the contracts I tested with more than 50 contracts, and it takes 13 seconds

samaradel added 4 commits June 12, 2025 15:51

Optimize contracts fetching

3e900a9

Merge branch 'development' into dev_load_deployments_performance

23301da

Remove unused vars

236bbbb

samaradel marked this pull request as ready for review June 19, 2025 14:17

samaradel requested review from AhmedHanafy725, zaelgohary, MohamedElmdary, Mahmoud-Emad, amiraabouhadid, AlaaElattar and 0oM4R as code owners June 19, 2025 14:17

0oM4R reviewed Jun 22, 2025

View reviewed changes

packages/playground/src/components/k8s_deployment_table.vue Show resolved Hide resolved

samaradel requested a review from 0oM4R June 22, 2025 12:12

0oM4R requested changes Jun 22, 2025

View reviewed changes

samaradel marked this pull request as draft June 23, 2025 06:25

Use promise.allSettled instead

2cb4f99

samaradel requested a review from 0oM4R June 23, 2025 08:04

samaradel marked this pull request as ready for review June 23, 2025 08:04

0oM4R reviewed Jun 23, 2025

View reviewed changes

samaradel marked this pull request as draft June 23, 2025 09:36

Replace promise.all accross all deployments files

0482f2c

samaradel marked this pull request as ready for review June 23, 2025 10:51

samaradel requested a review from 0oM4R June 23, 2025 10:51

0oM4R reviewed Jun 25, 2025

View reviewed changes

samaradel marked this pull request as draft June 25, 2025 12:10

samaradel added 2 commits June 25, 2025 16:12

- Run the three independent calls using Promise.allSettled

3484c4c

- Call getTitalCost function only if there is no type defined

Fully parallel cluster detail loading

500da01

samaradel requested a review from 0oM4R June 25, 2025 14:42

0oM4R reviewed Jul 7, 2025

View reviewed changes

packages/playground/src/utils/batch_process.ts Outdated Show resolved Hide resolved

packages/playground/src/utils/get_nodeInfo_with_cache.ts Outdated Show resolved Hide resolved

packages/playground/src/components/vm_deployment_table.vue Show resolved Hide resolved

samaradel marked this pull request as draft July 8, 2025 09:08

- Remove dead code

2d1bb02

- Remove get_nodeInfo_with_cache file and use it as a seperate function - Create test unit for getNodeInfoWithCache function - Extract helper functions to reduce the nesting

samaradel requested a review from 0oM4R July 8, 2025 19:30

samaradel marked this pull request as ready for review July 8, 2025 19:31

Fix batch process calling

91d0faf

samaradel marked this pull request as draft July 10, 2025 07:42

samaradel marked this pull request as ready for review July 10, 2025 07:48

samaradel mentioned this pull request Jul 10, 2025

Profile Contract / Deployment Loading [3 days] #4223

Open

0oM4R reviewed Jul 10, 2025

View reviewed changes

samaradel marked this pull request as draft July 13, 2025 10:25

samaradel added 4 commits July 13, 2025 17:52

Use cached getGridClient instead of updateGrid to avoid recreating mo…

13747a8

…dules

Merge branch 'development' into dev_load_deployments_performance

c62179c

Fix conflicts

3d9814c

Fix load network duplicates

469b79b

samaradel marked this pull request as ready for review July 13, 2025 17:03

samaradel requested a review from 0oM4R July 13, 2025 17:03

0oM4R requested a review from ramezsaeed July 14, 2025 08:19

ramezsaeed reviewed Jul 14, 2025

View reviewed changes

ramezsaeed mentioned this pull request Jul 14, 2025

QA progress July 2025 threefoldtech/home#1586

Closed

samaradel requested a review from ramezsaeed July 15, 2025 07:19

0oM4R reviewed Jul 15, 2025

View reviewed changes

samaradel marked this pull request as draft July 16, 2025 07:20

samaradel added 2 commits July 17, 2025 16:03

Merge branch 'development' into dev_load_deployments_performance

784df78

Fix conflicts

3796e69

Reduce contracts and deployments loading time #4233

Are you sure you want to change the base?

Reduce contracts and deployments loading time #4233

Conversation

samaradel commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Deployments:

Contracts:

Changes

Related Issues

Tested Scenarios

Documentation PR

To consider

Preliminary Checks:

UI Checks:

Code Quality Checks:

Testing Checklist

General Checklist

Uh oh!

Uh oh!

0oM4R left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0oM4R left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

0oM4R commented Jun 23, 2025

Uh oh!

0oM4R left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0oM4R left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0oM4R left a comment

Choose a reason for hiding this comment

Uh oh!

samaradel commented Jul 13, 2025

Uh oh!

ramezsaeed left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0oM4R left a comment

Choose a reason for hiding this comment

Uh oh!

0oM4R commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

samaradel commented Jun 12, 2025 •

edited

Loading

ramezsaeed left a comment •

edited

Loading

0oM4R commented Jul 15, 2025 •

edited

Loading

0oM4R commented Jul 15, 2025 •

edited

Loading

samaradel commented Jul 20, 2025 •

edited

Loading