Skip to content

Reduce contracts and deployments loading time #4233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 18 commits into
base: development
Choose a base branch
from
Draft

Reduce contracts and deployments loading time #4233

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
1f525e8
- Load all deployments in parallel
samaradel Jun 12, 2025
3e900a9
Optimize contracts fetching
samaradel Jun 19, 2025
23301da
Merge branch 'development' into dev_load_deployments_performance
samaradel Jun 19, 2025
236bbbb
Remove unused vars
samaradel Jun 19, 2025
2cb4f99
Use promise.allSettled instead
samaradel Jun 23, 2025
0482f2c
Replace promise.all accross all deployments files
samaradel Jun 23, 2025
3484c4c
- Run the three independent calls using Promise.allSettled
samaradel Jun 25, 2025
500da01
Fully parallel cluster detail loading
samaradel Jun 25, 2025
f90894e
- Handle the failure of VM deployments
samaradel Jun 29, 2025
26d9a93
- Run batches in parallel
samaradel Jun 29, 2025
2d1bb02
- Remove dead code
samaradel Jul 8, 2025
91d0faf
Fix batch process calling
samaradel Jul 8, 2025
13747a8
Use cached getGridClient instead of updateGrid to avoid recreating mo…
samaradel Jul 13, 2025
c62179c
Merge branch 'development' into dev_load_deployments_performance
samaradel Jul 13, 2025
3d9814c
Fix conflicts
samaradel Jul 13, 2025
469b79b
Fix load network duplicates
samaradel Jul 13, 2025
784df78
Merge branch 'development' into dev_load_deployments_performance
samaradel Jul 17, 2025
3796e69
Fix conflicts
samaradel Jul 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 73 additions & 27 deletions packages/playground/src/components/k8s_deployment_table.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@
<span>
This might happen because the node is down or it's not reachable
<span v-if="showEncryption">
or the deployment{{ count - items.length > 1 ? "s are" : " is" }} encrypted by another key
</span>.
or the deployment{{ count - items.length > 1 ? "s are" : " is" }} encrypted by another key </span>.
</span>
<v-tooltip location="top" text="Show failed deployments">
<template #activator="{ props: tooltipProps }">
Expand Down Expand Up @@ -168,7 +167,7 @@ import { getNodeHealthColor, NodeHealth } from "@/utils/get_nodes";
import { useProfileManager } from "../stores";
import { getGrid, updateGrid } from "../utils/grid";
import { markAsFromAnotherClient } from "../utils/helpers";
import { type K8S, type LoadedDeployments, loadK8s, mergeLoadedDeployments } from "../utils/load_deployment";
import { loadK8s, mergeLoadedDeployments } from "../utils/load_deployment";
const profileManager = useProfileManager();
const showDialog = ref(false);
const showEncryption = ref(false);
Expand All @@ -188,35 +187,82 @@ const loading = ref(false);

onMounted(loadDeployments);
async function loadDeployments() {
const start = performance.now();
items.value = [];
loading.value = true;
const grid = await getGrid(profileManager.profile!, props.projectName);
const chunk1 = await loadK8s(grid!);
const chunk2 = await loadK8s(updateGrid(grid!, { projectName: props.projectName.toLowerCase() }));
let chunk3: LoadedDeployments<K8S> = { count: 0, items: [], failedDeployments: [] };

if (showAllDeployments.value) {
chunk3 = await loadK8s(updateGrid(grid!, { projectName: "" }));
chunk3.items = chunk3.items.map(i => {
return !i.projectName || i.projectName === "Kubernetes" ? markAsFromAnotherClient(i) : i;
try {
const grid = await getGrid(profileManager.profile!, props.projectName);
if (!grid) {
loading.value = false;
return;
}
const [chunk1, chunk2, chunk3] = await Promise.all([
loadK8s(grid),
loadK8s(updateGrid(grid, { projectName: props.projectName.toLowerCase() })),
showAllDeployments.value
? loadK8s(updateGrid(grid, { projectName: "" }))
: Promise.resolve({ count: 0, items: [], failedDeployments: [] }),
]);
if (chunk3.items) {
chunk3.items = chunk3.items.map(i => {
return !i.projectName || i.projectName === "Kubernetes" ? markAsFromAnotherClient(i) : i;
});
}
const clusters = mergeLoadedDeployments(chunk1, chunk2, chunk3);
failedDeployments.value = clusters.failedDeployments;
count.value = clusters.count;
items.value = clusters.items.map(item => {
const master = item.masters[0];
const publicIP = master.publicIP?.ip;
return {
...item,
name: item.deploymentName,
ipv4: publicIP ? publicIP.split("/")?.[0] || publicIP : "-",
ipv6: master.publicIP?.ip6?.replace(/\/64$/, "") || "-",
planetary: master.planetary || "-",
workersLength: item.workers.length,
billing: undefined,
wireguard: undefined,
detailsLoading: false,
};
});

setTimeout(() => {
items.value.forEach(item => fetchClusterDetails(item));
}, 0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the need for settimeout here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I think those can be loaded in parallel

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it will be good, I'll implement it

} catch (error) {
console.error("Error loading deployments:", error);
items.value = [];
count.value = 0;
failedDeployments.value = [];
} finally {
loading.value = false;
const end = performance.now();
console.log(`Time taken: ${(end - start) / 1000} seconds`);
}
}

const clusters = mergeLoadedDeployments(chunk1, chunk2, chunk3);
failedDeployments.value = clusters.failedDeployments;

count.value = clusters.count;
items.value = clusters.items.map((item: any) => {
item.name = item.deploymentName;
item.ipv4 = item.masters[0].publicIP?.ip?.split("/")?.[0] || item.masters[0].publicIP?.ip || "-";
item.ipv6 = item.masters[0].publicIP?.ip6.replace(/\/64$/, "") || "-";
item.planetary = item.masters[0].planetary || "-";
item.workersLength = item.workers.length;
item.billing = item.masters[0].billing;
item.created = item.masters[0].created;
return item;
});
loading.value = false;
async function fetchClusterDetails(item: any) {
if (item.detailsLoading || (item.billing !== undefined && item.wireguard !== undefined)) return;
item.detailsLoading = true;
try {
const grid = await getGrid(profileManager.profile!, item.projectName || props.projectName);
if (!grid) {
item.detailsLoading = false;
return;
}
const consumption = await grid.contracts.getConsumption({ id: item.masters[0].contractId }).catch(() => undefined);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about passing the grid as arg then using the update grid,
also here may need to indicate failure

item.billing = consumption ? consumption.amountBilled : "No Data Available";
item.wireguard = await grid.networks
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can get the consumption and the wireguard in parallel; they are independent calls.

.getWireGuardConfigs({
name: item.masters[0].interfaces[0].network,
ipRange: item.masters[0].interfaces[0].ip,
})
.then(res => res[0])
.catch(() => undefined);
} finally {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a catch block; to at least log the errors

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

item.detailsLoading = false;
}
}

defineExpose({ loadDeployments });
Expand Down
151 changes: 39 additions & 112 deletions packages/playground/src/components/vm_deployment_table.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,18 @@
<template>
<div>
<v-alert
v-if="errorMessage"
type="error"
variant="tonal"
>
<v-alert v-if="errorMessage" type="error" variant="tonal">
{{ errorMessage }}
</v-alert>
<v-alert
v-if="!loading && count && items.length < count"
type="warning"
variant="tonal"
>
<v-alert v-if="!loading && count && items.length < count" type="warning" variant="tonal">
Failed to load <strong>{{ count - items.length }}</strong> deployment{{ count - items.length > 1 ? "s" : "" }}.

<span>
This might happen because the node is down or it's not reachable
<span v-if="showEncryption">or the deployment{{ count - items.length > 1 ? "s are" : " is" }} encrypted by another key</span>.
</span>
<v-tooltip
location="top"
text="Show failed deployments"
>
<v-tooltip location="top" text="Show failed deployments">
<template #activator="{ props: slotProps }">
<v-icon
v-bind="slotProps"
class="custom-icon"
@click="showDialog = true"
>
<v-icon v-bind="slotProps" class="custom-icon" @click="showDialog = true">
mdi-file-document-refresh-outline
</v-icon>
</template>
Expand All @@ -41,15 +26,10 @@
attach="#modals"
>
<v-card>
<v-card-title style="font-weight: bold">
Failed Deployments
</v-card-title>
<v-card-title style="font-weight: bold"> Failed Deployments </v-card-title>
<v-divider color="#FFCC00" />
<v-card-text>
<v-alert
type="error"
variant="tonal"
>
<v-alert type="error" variant="tonal">
Failed to load
<strong>{{ count - items.length }}</strong> deployment{{ count - items.length > 1 ? "s" : "" }}.

Expand All @@ -58,23 +38,14 @@
<span v-if="showEncryption">or the deployment{{ count - items.length > 1 ? "s are" : " is" }} encrypted by another key</span>.
</span>
</v-alert>
<v-list
:items="failedDeploymentList"
item-props
lines="three"
>
<v-list :items="failedDeploymentList" item-props lines="three">
<template #subtitle="{ subtitle }">
<div v-html="subtitle" />
</template>
</v-list>
</v-card-text>
<v-card-actions class="justify-end my-1 mr-2">
<v-btn
color="anchor"
@click="showDialog = false"
>
Close
</v-btn>
<v-btn color="anchor" @click="showDialog = false"> Close </v-btn>
</v-card-actions>
</v-card>
</v-dialog>
Expand Down Expand Up @@ -139,10 +110,7 @@
</template>

<template #[`item.flist`]="{ item }">
<v-tooltip
:text="item.flist"
location="bottom right"
>
<v-tooltip :text="item.flist" location="bottom right">
<template #activator="{ props: slotProps }">
<p v-bind="slotProps">
{{ renameFlist(item.flist) }}
Expand All @@ -158,40 +126,18 @@
{{ toHumanDate(item.created) }}
</template>
<template #[`item.actions`]="{ item }">
<v-chip
v-if="deleting && ($props.modelValue || []).includes(item)"
color="error"
>
Deleting...
</v-chip>
<v-btn-group
v-else
variant="tonal"
>
<slot
:name="projectName + '-actions'"
:item="item"
:update="updateItem"
/>
<v-chip v-if="deleting && ($props.modelValue || []).includes(item)" color="error"> Deleting... </v-chip>
<v-btn-group v-else variant="tonal">
<slot :name="projectName + '-actions'" :item="item" :update="updateItem" />
</v-btn-group>
</template>

<template #[`item.status`]="{ item }">
<v-chip :color="getNodeHealthColor(item.status as string).color">
<v-tooltip
v-if="item.status == NodeHealth.Error"
activator="parent"
location="top"
>
{{
item.message
}}
<v-tooltip v-if="item.status == NodeHealth.Error" activator="parent" location="top">
{{ item.message }}
</v-tooltip>
<v-tooltip
v-if="item.status == NodeHealth.Paused"
activator="parent"
location="top"
>
<v-tooltip v-if="item.status == NodeHealth.Paused" activator="parent" location="top">
The deployment contract is in grace period
</v-tooltip>
<span class="text-uppercase">
Expand All @@ -201,20 +147,10 @@
</template>
<template #[`item.health`]="{ item }">
<v-chip :color="getNodeHealthColor(item[0].workloads[0].result.state as string).color">
<v-tooltip
v-if="item[0].workloads[0].result.state == NodeHealth.Error"
activator="parent"
location="top"
>
{{
item.message
}}
<v-tooltip v-if="item[0].workloads[0].result.state == NodeHealth.Error" activator="parent" location="top">
{{ item.message }}
</v-tooltip>
<v-tooltip
v-if="item[0].workloads[0].result.state == NodeHealth.Paused"
activator="parent"
location="top"
>
<v-tooltip v-if="item[0].workloads[0].result.state == NodeHealth.Paused" activator="parent" location="top">
The deployment contract is in grace period
</v-tooltip>
<span class="text-uppercase">
Expand All @@ -224,10 +160,7 @@
</template>

<template #no-data-text>
<div
v-if="failedDeploymentList.length > 0"
class="text-center"
>
<div v-if="failedDeploymentList.length > 0" class="text-center">
<p v-text="'Couldn\'t load any of your ' + projectTitle + ' deployments.'" />
<VBtn
class="mt-4"
Expand All @@ -238,10 +171,7 @@
@click="loadDeployments"
/>
</div>
<p
v-else
v-text="'No ' + projectTitle + ' deployments found on this account.'"
/>
<p v-else v-text="'No ' + projectTitle + ' deployments found on this account.'" />
</template>
</ListTable>
</div>
Expand All @@ -255,7 +185,7 @@ import { getNodeHealthColor, NodeHealth } from "@/utils/get_nodes";
import { useProfileManager } from "../stores";
import { getGrid, updateGrid } from "../utils/grid";
import { markAsFromAnotherClient } from "../utils/helpers";
import { type LoadedDeployments, loadVms, mergeLoadedDeployments } from "../utils/load_deployment";
import { loadVms, mergeLoadedDeployments } from "../utils/load_deployment";

const profileManager = useProfileManager();

Expand Down Expand Up @@ -303,6 +233,7 @@ async function loadDomains() {
}

async function loadDeployments() {
const start = performance.now();
if (props.projectName.toLowerCase() === ProjectName.Domains.toLowerCase()) {
return loadDomains();
}
Expand All @@ -313,27 +244,23 @@ async function loadDeployments() {
loading.value = true;
const grid = await getGrid(profileManager.profile!, props.projectName);
try {
const chunk1 = await loadVms(grid!);
if (chunk1.count > 0 && migrateGateways) {
await migrateModule(grid!.gateway);
const [chunk1, chunk2, chunk3] = await Promise.all([
loadVms(grid!),
loadVms(updateGrid(grid!, { projectName: props.projectName.toLowerCase() })),
showAllDeployments.value && props.projectName.toLowerCase() === ProjectName.VM.toLowerCase()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may group the chunk 3 condition in both files into a separate variable then use it here
example

loadChunk3 ? loadVms(updateGrid(grid!, { projectName: "" })) :  Promise.resolve({ count: 0, items: [], failedDeployments: [] }),

? loadVms(updateGrid(grid!, { projectName: "" }))
: Promise.resolve({ count: 0, items: [], failedDeployments: [] }),
]);

if (migrateGateways) {
await Promise.all([
chunk1.count > 0 && migrateModule(grid!.gateway),
chunk2.count > 0 && migrateModule(grid!.gateway),
chunk3.count > 0 && migrateModule(grid!.gateway),
]);
}

const chunk2 = await loadVms(updateGrid(grid!, { projectName: props.projectName.toLowerCase() }));
if (chunk2.count > 0 && migrateGateways) {
await migrateModule(grid!.gateway);
}

let chunk3: LoadedDeployments<any[]> = { count: 0, items: [], failedDeployments: [] };
if (showAllDeployments.value) {
chunk3 =
props.projectName.toLowerCase() === ProjectName.VM.toLowerCase()
? await loadVms(updateGrid(grid!, { projectName: "" }))
: { count: 0, items: [], failedDeployments: [] };

if (chunk3.count > 0 && migrateGateways) {
await migrateModule(grid!.gateway);
}

if (chunk3.items) {
chunk3.items = chunk3.items.map(markAsFromAnotherClient);
}

Expand All @@ -346,8 +273,8 @@ async function loadDeployments() {
} finally {
loading.value = false;
}

loading.value = false;
const end = performance.now();
console.log(`Time taken: ${(end - start) / 1000} seconds`);
}

const filteredHeaders = computed(() => {
Expand Down
13 changes: 13 additions & 0 deletions packages/playground/src/utils/batch_process.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
export async function batchProcess<T, R>(
items: T[],
batchSize: number,
processFn: (batch: T[]) => Promise<R[]>,
): Promise<R[]> {
const results: R[] = [];
for (let i = 0; i < items.length; i += batchSize) {
const batch = items.slice(i, i + batchSize);
const batchResults = await processFn(batch);
results.push(...batchResults);
}
return results;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can run the batches in parallel. Also please add error handling and unit test for it

Loading