docs: explain file upload strategy

Author: BCerki

docs/file_uploads.md

@@ -1,18 +1,32 @@
 # File uploads in BCIERS app
 
-Could use some optimization: https://github.com/bcgov/cas-registration/issues/2123
-
 Two methods are available:
 
 - with RJSF
 - without RJSF using `FormData`
 
-## With RJSF (using data-urls)
+#### Model
+
+The `FileField` is where django does the magic, along with the `STORAGES` configuration in settings.py.
+More documentation [here](https://docs.djangoproject.com/en/5.1/ref/models/fields/#filefield)
+
+```python
+  class Document(TimeStampedModel):
+    file = models.FileField(upload_to="documents", db_comment="The file format, metadata, etc.")
+```
+
+#### Connection with GCS
+
+- Some setup is done in cas-registration/bc_obps/bc_obps/settings.py, will need env variables
+- GCS is not set up in CI so we skip endpoint tests related to files, and we don't have any file stuff in our mock data
+
+## With RJSF
+
+Because files can be large and slow to process, we only pass the file from the front end to the back end when the user first uploads it. We never pass the file itself from back to front; instead we pass a url where the user can download it.
 
 ### Frontend
 
-- RJSF supports file with data-urls: https://rjsf-team.github.io/react-jsonschema-form/docs/usage/widgets/#file-widgets
-- `FileWidget`: this started as a copy/paste from RJSF's [FileWidget](https://github.com/rjsf-team/react-jsonschema-form/blob/main/packages/core/src/components/widgets/FileWidget.tsx) and @marcelmueller did some styling to match the designs. It now additionally includes a check for max file size, and possibly other stuff including state mgt.
+We send files to the backend using FormData. Because RJSF stores data as json, in the handleSubmit, we have to convert to FormData (see `convertRjsfFormData` from "@/registration/app/components/operations/registration/OperationInformationForm"). The conversion happens in the form component (e.g. `OperationInformationForm`). The `FileWidget` handles previewing and downloading the file, and additionally includes a check for max file size.
 
 In the rjsf schema:
 
@@ -21,76 +35,99 @@ In the rjsf schema:
 statutory_declaration: {
   type: "string",
   title: "Statutory Declaration",
-  format: "data-url",
 }
 
 # uiSchema
 statutory_declaration: {
   "ui:widget": "FileWidget",
   "ui:options": {
-    filePreview: true,
     accept: ".pdf",
   }
 }
 ```
 
 ### Backend
 
-#### Ninja field validator
+#### Endpoints
+
+To make django ninja happy, we have to separate out form data and file data. There are a few ways to do this, see the docs, and we've chosen to go with this for a POST endpoint:
+
+```python
+def update_operation(
+    request: HttpRequest, operation_id: UUID,
+    details: Form[OperationAdminstrationIn], # OperationAdminstrationIn is a ModelSchema or Schema
+    boundary_map: UploadedFile = File(None),
+    process_flow_diagram: UploadedFile = File(None),
+    new_entrant_application: UploadedFile = File(None)
+)
+```
 
-The field is declared as a string, and we validate that we can convert it to a file
+Notes:
 
-`In` schema:
+- File is always optional because if the user hasn't changed the file, we don't send anything.
+- Django ninja doesn't support files on PUT, so we have to use POST for anything file-related
+
+A GET endpoint requires a conversion from file to download link in the ninja schema:
+
+`Out` schema:
 
 ```python
 ...
-  boundary_map: str
+  class OutWithDocuments(ModelSchema):
+    boundary_map: Optional[str] = None
 
-  @field_validator("boundary_map")
-  @classmethod
-  def validate_boundary_map(cls, value: str) -> ContentFile:
-    return data_url_to_file(value)
+    @staticmethod
+    def resolve_boundary_map(obj: Operation) -> Optional[str]:
+        return str(obj.get_boundary_map().file.url)
 ```
 
-The reverse can be done for the `Out` schema if needed:
+The `FileWidget` and `ReadOnlyFileWidget` can handle both File and string values.
 
-```python
-  boundary_map: Optional[str] = None
+#### Service
 
-  @staticmethod
-  def resolve_boundary_map(obj: Operation) -> Optional[str]:
-    boundary_map = obj.get_boundary_map()
-    if boundary_map:
-      return file_to_data_url(boundary_map)
+We have two services for file uploads:
 
-    return None
-```
+- DocumentServiceV2.create_or_replace_operation_document
+- DocumentDataAccessServiceV2.create_document
 
-#### Service
+### Testing
 
-```python
-  DocumentService.create_or_replace_operation_document(
-    user_guid,
-    operation.id,
-    payload.boundary_map, # type: ignore # mypy is not aware of the schema validator
-    'boundary_map',
-  ),
-```
+We have mock files in both our FE and BE constants.
 
-#### Model
+In vitests, we have to mock `createObjectURL`, which is used in the `FileWidget` (e.g. `global.URL.createObjectURL = vi.fn(() => "this is the link to download the File",);`).
 
-The `FileField` is where django does the magic, along with the `STORAGES` configuration in settings.py.
-More documentation [here](https://docs.djangoproject.com/en/5.1/ref/models/fields/#filefield)
+We check mocked calls like this:
 
-```python
-  class Document(TimeStampedModel):
-    file = models.FileField(upload_to="documents", db_comment="The file format, metadata, etc.")
+```
+expect(actionHandler).toHaveBeenCalledWith(
+          "registration/operations/b974a7fc-ff63-41aa-9d57-509ebe2553a4/registration/operation",
+          "POST",
+          "",
+          { body: expect.any(FormData) },
+        );
+        const bodyFormData = actionHandler.mock.calls[1][3].body;
+        expect(bodyFormData.get("registration_purpose")).toBe(
+          "Reporting Operation",
+        );
+        expect(bodyFormData.getAll("activities")).toStrictEqual(["1", "2"]);
+        ...
 ```
 
-#### Connection with GCS
+When using pytests, we have to mock payloads that include files like this (note the array []):
 
-- Some setup is done in cas-registration/bc_obps/bc_obps/settings.py, will need env variables
-- GCS is not set up in CI so we skip endpoint tests related to files, and we don't have any file stuff in our mock data
+```python
+mock_payload = {
+        'registration_purpose': ['Reporting Operation'],
+        'operation': ['556ceeb0-7e24-4d89-b639-61f625f82084'],
+        'activities': ['31'],
+        'name': ['Barbie'],
+        'type': [Operation.Types.SFO],
+        'naics_code_id': ['20'],
+        'operation_has_multiple_operators': ['false'],
+        'process_flow_diagram': ContentFile(bytes("testtesttesttest", encoding='utf-8'), "testfile.pdf"),
+        'boundary_map': ContentFile(bytes("testtesttesttest", encoding='utf-8'), "testfile.pdf"),
+    }
+```
 
 ## Without RJSF