You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/features/runs/2_Analyzing Runs/3_artifacts.md
+18Lines changed: 18 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,6 +10,24 @@ Artifacts are the output produced by the training process.
10
10
11
11
Anytime your script saves artifacts, Grid captures those for you. It does not matter which folder you save artifacts to... Grid will automatically detect them.
12
12
13
+
Grid uploads your artifacts to long-term storage while your experiment is running *and* at the end of your experiment. This is to ensure that even if your experiment crashes, your artifacts will be saved.
14
+
15
+
:::note
16
+
This may have two side effects, depending on how large and how many artifacts your code generates:
17
+
18
+
1. Extra Memory Usage
19
+
20
+
Grid may need to use extra memory while your experiment is running to sync your artifacts to long term storage.
21
+
- Grid will only use extra memory to sync artifacts larger than 1 Gib
22
+
- Grid will use 50 Mi of extra memory to sync artifacts larger than 1 Gib
23
+
- Grid will not use more than 200 Mi of extra memory
24
+
- If you are running memory intensive experiments *and* are generating artifacts over 1 Gib, it is recommended to use a larger instance type
25
+
26
+
2. Increased run time
27
+
28
+
Your experiment will not be marked `done` until Grid has finished syncing your artifacts to long-term storage. If you are generating lots of artifacts per experiment, this may increase your experiment run times.
29
+
:::
30
+
13
31
Try [hello.py](https://github.com/williamFalcon/hello/blob/main/hello.py) as an example, to see how artifacts are generated and saved.
0 commit comments