Skip to content

Conversation

julien-nc
Copy link
Member

@julien-nc julien-nc commented Jul 15, 2025

closes #304

3 File actions are registered:

  • summarize for text and markdown files
  • STT for audio files
  • TTS for text and markdown files

A notification is sent after a file action process finishes or fails.

Main changes:

  • Restructure the notifier
  • New endpoint in the API controller to run the actions
  • 2 new listeners for TaskFailed and TaskSuccess to handle tasks related with file actions

@julien-nc julien-nc added enhancement New feature or request 3. to review labels Jul 15, 2025
@julien-nc julien-nc force-pushed the enh/304/file-actions branch 2 times, most recently from 6fc2627 to 46b1fb4 Compare July 16, 2025 10:03
@julien-nc
Copy link
Member Author

julien-nc commented Jul 16, 2025

@marcoambrosini @jancborchardt I made a first implementation. Let's adjust it a bit.
Right now, for text files and wav/mp3 files, we have a new file action group:
image
For text files:
image
For audio files:
image

Once it is clicked, there's a top-right toast saying "A file will be created soon".
Then the task is scheduled and when it's done, a new file is created next to the source one with a suffix (_summarized, _tts or _stt).

Questions:

  • Are you fine with the action names and icons?
  • Should we adjust the toast msg?
  • Are you fine with the behaviour or creating a file next to the source one?
  • Are you fine with the suffix?
  • Should there be a notification when the new file is created?

Copy link
Member

@jancborchardt jancborchardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome awesome @julien-nc! :)

Questions:

* Are you fine with the action names and icons?

The names are good! Best to use the Material Symbols for it:

* Should we adjust the toast msg?

Can you check for the wording @marcoambrosini? It's about the showSuccess and showError texts I assume.

* Are you fine with the behaviour or creating a file next to the source one?
* Are you fine with the suffix?

Sure, seems great to put it next to it. Just the naming should be adjusted to be human readable:

  • " - summarized"
  • " - text to speech"
  • " - transcribed"
* Should there be a notification when the new file is created?

Good question, I guess you should be notified when it's done just like the Talk recording.

And one more thing: What about direct transcription + summarization? Would be a nice feature too and could just chain the 2 functions in the back?

@julien-nc julien-nc force-pushed the enh/304/file-actions branch from 829e56d to 6f5560d Compare July 22, 2025 09:47
@julien-nc julien-nc marked this pull request as ready for review July 22, 2025 09:47
@julien-nc julien-nc requested a review from jancborchardt July 22, 2025 09:47
@julien-nc
Copy link
Member Author

julien-nc commented Jul 22, 2025

@jancborchardt Thanks for the feedback.

  • Target file names have been adjusted
  • There is now a notification when a file action succeeds and fails (because the task failed or because something else failed). The source file is a rich object (/f/ link), the action button is also a /f/ link to the target file. The notifications looks like that:
image
  • Material symbols are now used (see updated screenshots in previous comment)

And one more thing: What about direct transcription + summarization? Would be a nice feature too and could just chain the 2 functions in the back?

I don't have time for that these days. We can keep it as a followup task. (Maybe it's worth creating another GH issue for that).

Copy link
Member

@lukasdotcom lukasdotcom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Tested it out and it worked.
Just a nitpick and some things that should be discussed.

return t('assistant', 'Assistant')
},
enabled(nodes, view) {
return !actionIgnoreLists.includes(view.id)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be disabled if none of the assistant tasks are available.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the case, the group action is not registered if none of the task types are available. Check the end of src/files/fileActions.js.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had meant for filetypes that currently don't have an assistant file action.

&& nodes.length === 1
&& !nodes.some(({ permissions }) => (permissions & Permission.READ) === 0)
&& nodes.every(({ type }) => type === FileType.File)
&& nodes.every(({ mime }) => ['text/plain', 'text/markdown'].includes(mime))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing here as for summarize

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, instead of toggling the action with its enabled attr, we conditionally register it at the end of the file.

&& nodes.length === 1
&& !nodes.some(({ permissions }) => (permissions & Permission.READ) === 0)
&& nodes.every(({ type }) => type === FileType.File)
&& nodes.every(({ mime }) => ['text/plain', 'text/markdown'].includes(mime))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth it to support all files that parseTextFromFile supports instead of just markdown and plain text.

public function parseTextFromFile(string $userId, ?string $filePath = null, ?int $fileId = null): string {
try {
$userFolder = $this->rootFolder->getUserFolder($userId);
} catch (\OC\User\NoUserException|NotPermittedException $e) {
throw new \Exception('Could not access user storage.');
}
try {
if ($filePath !== null) {
$file = $userFolder->get($filePath);
} else {
$file = $userFolder->getFirstNodeById($fileId);
}
} catch (NotFoundException $e) {
throw new \Exception('File not found.');
}
try {
if ($file instanceof File) {
$fileContent = $file->getContent();
} else {
throw new \Exception('Provided path does not point to a file.');
}
} catch (LockedException|GenericFileException|NotPermittedException $e) {
throw new \Exception('File could not be accessed.');
}
$mimeType = $file->getMimeType();
switch ($mimeType) {
default:
case 'text/plain':
{
$text = $fileContent;
break;
}
case 'text/markdown':
{
$parser = new Parsedown();
$text = $parser->text($fileContent);
// Remove HTML tags:
$text = strip_tags($text);
break;
}
case 'text/rtf':
{
$text = $this->parseRtfDocument($fileContent);
break;
}
case 'application/vnd.openxmlformats-officedocument.wordprocessingml.document':
case 'application/msword':
case 'application/vnd.oasis.opendocument.text':
{
$tempFilePath = $this->tempManager->getTemporaryFile();
file_put_contents($tempFilePath, $fileContent);
$text = $this->parseDocument($tempFilePath, $mimeType);
$this->tempManager->clean();
break;
}
case 'application/pdf':
{
$parser = new Parser();
$pdf = $parser->parseContent($fileContent);
$text = $pdf->getText();
break;
}
}
return $text;
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense!
I created an issue to keep track of that: #321
We can do it later, after having merged this one.

Copy link
Member

@jancborchardt jancborchardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, let’s do the rest in follow-ups! :)

Copy link
Member

@lukasdotcom lukasdotcom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the clarification on what I meant with disabling the group action when there is no task available for that file everything else LGTM

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
…e material symbols

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
…d action button in notifications to open the result task in the assistant

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
Signed-off-by: Julien Veyssier <julien-nc@posteo.net>
@julien-nc julien-nc force-pushed the enh/304/file-actions branch from 6277a0b to 9d99e95 Compare July 29, 2025 15:08
@julien-nc julien-nc requested a review from kyteinsky July 29, 2025 15:08
@julien-nc
Copy link
Member Author

@kyteinsky Thanks for the review. All done.

Copy link
Contributor

@kyteinsky kyteinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@julien-nc julien-nc merged commit 9d787da into main Jul 30, 2025
16 checks passed
@julien-nc julien-nc deleted the enh/304/file-actions branch July 30, 2025 08:20
@janepie janepie mentioned this pull request Aug 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3. to review enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Files actions using TaskProcessing
4 participants