Skip to content

Conversation

julien-nc
Copy link
Member

@julien-nc julien-nc commented Aug 26, 2025

closes #340

Add declarative UI capabilities for clients.
Add one hook for each of our file actions (summarize, stt and tts).

Currently unknown:

  • Which HTTP method is used to reach the hooks
  • How is the file ID passed
  • How to define multiple filters for one hook

Copy link
Member

@marcelklehr marcelklehr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable

@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch from a5b2673 to 72e2f65 Compare August 29, 2025 08:20
@julien-nc
Copy link
Member Author

julien-nc commented Aug 29, 2025

When combining the capabilities declared by various apps, the server combines the arrays. For objects, all good. For lists, as the keys are preserved and a Php list is almost an object with integer keys, the combining goes wrong and each list overwrites the other ones.

Here is my suggestion for the declarativeui capability structure:

  • declarativeui is an object indexed by app IDs. So in each item of this object, an app can do whatever it wants, the content will not be combined with others
  • Inside an "appId" object, we can have 2 list of endpoints, one under context-menu and one under create-new
  • An endpoint is an object with those keys:
    • name
    • url
    • method (default: GET)
    • mimetype_filters (comma separated list of mimetype filters, for now these are not regexps, clients will accept files which mime type starts with any of the filters)
    • icon is a URL to the icon
    • bodyParams is an object with all the parameters that should be passed in the request body. The {fileId} placeholder is supported there too. Example: "bodyParams": "{\"fileId\": \"{fileid}\"}",

Object in the declarativeui key of the capabilites:

{
  "assistant": {
    "context-menu": [
      {
        "name": "Summarize",
        "url": "/ocs/v2.php/apps/assistant/api/v1/file-action/{fileId}/core:text2text:summary",
        "method": "POST",
        "mimetype_filters": "text/, application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/vnd.oasis.opendocument.text, application/pdf",
        "icon": "/apps-extra/assistant/img/declarativeui/summarize.svg"
      },
      {
        "name": "Transcribe audio",
        "url": "/ocs/v2.php/apps/assistant/api/v1/file-action/{fileId}/core:audio2text",
        "method": "POST",
        "mimetype_filters": "audio/",
        "icon": "/apps-extra/assistant/img/declarativeui/speech_to_text.svg"
      },
      {
        "name": "Text to speech",
        "url": "/ocs/v2.php/apps/assistant/api/v1/file-action/{fileId}/core:text2speech",
        "method": "POST",
        "mimetype_filters": "text/, application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/vnd.oasis.opendocument.text, application/pdf",
        "icon": "/apps-extra/assistant/img/declarativeui/text_to_speech.svg"
      }
    ]
  },
  "cospend": {
    "context-menu": [
      {
        "name": "cospend fake",
        "url": "hello"
      }
    ]
  }
}

@julien-nc
Copy link
Member Author

julien-nc commented Aug 29, 2025

@tobiasKaminsky You can test this branch by checking it out and building it:

npm ci
npm run dev
composer i --no-dev

For the capabilities to be declared by the assistant, you need some AI providers. There are some fake ones in the testing app (summarize and STT).

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
…add multi filters, add method

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch 3 times, most recently from 944ea1e to e4f9dc8 Compare September 4, 2025 13:00
…n file

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
@julien-nc julien-nc force-pushed the enh/340/declarative-ui branch from e4f9dc8 to 7d3d17a Compare September 4, 2025 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Declarative UI: audio file -> create a transcript (md)
2 participants