Add text input controller infrastructure for mobile IME support#10557
Add text input controller infrastructure for mobile IME support#10557tilladam wants to merge 13 commits into
Conversation
This commit implements PREP-001 and PREP-002 from the text input architecture preparation plan, providing the foundation for Android/iOS IME integration. PREP-001: Make InputMethodRequest a public API - Move handle_input_method_request() from WindowAdapterInternal to WindowAdapter - Export InputMethodRequest and InputMethodProperties from slint::platform - Update all backends to implement the public trait method PREP-002: Expose InputMethodProperties mutability - Add preedit_cursor and composing_region fields to InputMethodProperties - Add TextInputError enum for error reporting - Add composing_region field to TextInput for persisting IME state - Add public IME methods to TextInput: ime_commit_text, ime_set_preedit, ime_clear_preedit, ime_set_composing_region, ime_delete_surrounding, ime_set_selection - Add corresponding Window methods for platform backends to call - All methods validate UTF-8 byte offsets and return appropriate errors
This commit implements PREP-003 from the text input architecture plan, providing a controller abstraction for mobile IME protocols. New file: internal/core/text_input_controller.rs - TextInputController trait with query and mutation methods for platform IME to interact with TextInput (Android InputConnection, iOS UITextInput) - CoreTextInputController default implementation using weak references - Byte offset utility functions for UTF-8 character boundary handling - Unit tests for byte offset utilities WindowAdapter trait additions: - text_input_focused(controller): Called when TextInput gains focus, provides controller for platform IME integration - text_input_unfocused(): Called when TextInput loses focus TextInput focus handling now: - Creates CoreTextInputController on FocusIn - Calls text_input_focused() to notify platform - Calls text_input_unfocused() on FocusOut Key design decisions: - Controller uses weak references (auto-invalidates on focus loss) - Main-thread only (no Send+Sync) matching platform IME requirements - Batch edit with nesting counter (Android InputConnection style)
Platform backends can now report soft keyboard visibility and geometry via Window::set_soft_keyboard_state(). This enables layouts to adjust when the keyboard appears on mobile devices. The implementation integrates with existing virtual_keyboard_* properties on the Window item and automatically scrolls focused elements into view.
Add styling properties for IME composition rendering: - preedit_color, preedit_underline_color, preedit_underline_width, preedit_background for preedit text styling - composing_underline_color, composing_underline_width for composing region styling Update TextInputVisualRepresentation to include composing_range and all styling properties so renderers can customize how preedit and composing regions are displayed.
Create text_input.rs module with AndroidTextInputHandler that: - Stores TextInputController when TextInput gains focus - Provides InputConnection method stubs for future JNI integration - Documents next steps for full InputConnection implementation Implement text_input_focused/text_input_unfocused WindowAdapter methods to integrate with the new TextInputController API from PREP-003.
Create ios/text_input.rs module with IOSTextInputHandler that: - Stores TextInputController when TextInput gains focus - Provides UITextInput protocol method stubs for future implementation - Documents next steps for full UITextInput protocol integration Implement text_input_focused/text_input_unfocused WindowAdapter methods (iOS-only) to integrate with the TextInputController API from PREP-003.
Expand test coverage for the text input controller infrastructure: - Add tests for byte offset utilities with multibyte chars, emoji, surrogate pairs, and combining characters - Add tests for CoreTextInputController behavior when invalid (all query methods return defaults, all mutations return false) - Add tests for batch edit nesting logic - Add MockWindowAdapter for creating invalid weak references in tests
Add comprehensive tests for platform-specific text input handlers: Android (AndroidTextInputHandler): - Handler lifecycle tests (focus/unfocus, validity) - Text query tests (get_text_before/after_cursor, selected_text) - Text mutation tests (commit_text, composing, delete, selection) - Cursor offset conversion tests (Android 1-based to 0-based) iOS (IOSTextInputHandler): - Handler lifecycle tests (focus/unfocus, validity) - UITextInput protocol method tests (text_in_range, selection, marked text) - Document boundary tests (beginning/end_of_document) - Text mutation tests (replace_range, insert_text, delete_backward) Both test suites use a MockTextInputController to verify method delegation and argument conversion without requiring actual TextInput elements. Tests are platform-gated and run on respective platforms.
Add test infrastructure for simulating IME input: - simulate_ime_preedit(): Set preedit/composition text on focused TextInput - simulate_ime_commit(): Commit text, replacing any active preedit - simulate_ime_set_composing_region(): Mark existing text as being edited Add comprehensive preedit integration tests (preedit.slint): - Preedit property accessibility and initial state - Normal keyboard input doesn't set preedit - IME preedit simulation with Japanese text - IME commit replaces preedit with final text - Clear preedit without committing (cancel composition) - Preedit with cursor position within composition - Multiple incremental preedit updates (simulates typing) - Commit with cursor offset positioning
Add testing infrastructure for soft keyboard state: - simulate_set_soft_keyboard_state() to set keyboard visibility/height - get_soft_keyboard_state() to query current state Add integration test verifying: - Default state (keyboard hidden, zero dimensions) - Keyboard showing updates virtual keyboard properties - Content layout adjusts to keyboard height - Keyboard height changes are properly tracked - Keyboard hiding clears virtual keyboard rect
Android (Java) and iOS (NSString) use UTF-16 code unit offsets for text positions, while Rust strings use UTF-8 byte offsets. Without proper conversion, any non-ASCII text (emoji, CJK, accented characters) would cause incorrect cursor/selection positioning. New functions in text_input_controller.rs: - utf16_offset_to_byte_offset(): Convert UTF-16 offset to byte offset Returns None for invalid offsets (inside surrogate pairs, beyond string) - byte_offset_to_utf16_offset(): Convert byte offset to UTF-16 offset Panics if byte offset is not on a valid UTF-8 boundary - utf16_offset_to_byte_offset_clamped(): Same as above but clamps invalid offsets to nearest valid position instead of returning None Updated platform handlers to use these conversions: - AndroidTextInputHandler: set_selection, set_composing_region, get_cursor_and_selection, delete_surrounding_text - IOSTextInputHandler: text_in_range, replace_range, selected_text_range, set_selected_text_range, marked_text_range, set_marked_text, end_of_document Includes 20 unit tests covering ASCII, BMP characters (CJK), surrogate pairs (emoji), combining characters, and roundtrip conversions.
|
I've started work on the iOS adapter for AccessKit and I think text input will be a tricky part. Modern iOS/UIAccessibility don't contain separate APIs to expose text inputs to the assistive technologies. We instead have to implement the UITextInput protocol. If my understanding of this PR is correct, Slint chooses to take full control over text edit and would stop relying on winit. For AccessKit that would be better but we'd need to make sure the UITextInput objects actually contain the full text and is not just here for IME. We also need to make sure there is one UITextInput per Slint text input. FYI AccessKit will subclass the winit view at runtime to expose the accessibility tree. AccessKit nodes will implement the UIAccessibility informal protocol. For text inputs this will mean returning a UITextInput object from accessibilityTextInputResponder so we'll need a way to get it from Slint or for you to provide it to us. CC @tronical, @ogoffart it would be best to figure this out early. |
|
I haven't reviewed the full patch yet (that's a lot of code!). The other thing is this adds is properties to the TextInput, but it doesn't really add these properties are it is not added in the builtins.slint. Also i'm wondering what's their purpose. I'd like to know exactly what is the goal.
is a bit vague. We already have some foundational infrastructure. What exactly was missing? |
I agree that we seem to need an implementation of the It's unclear to me how we can most efficiently share the implementation. Perhaps Slint can pass a The existing interface between the run-time library and the backends here is this: The run-time library communicates state changes to the backend (a possible UITextInput implementation, etc.) via the The other way around, when the input method decides to enter composition mode, etc. we've got the The idea was that once we've got enough implementations of input methods based on this two-way interface, we can make it public. I think we should continue to follow that. This PR renames the It also introduces a third channel of communication: The I'll comment a bit more on the implementation, but I think it would be good to have a chat about this first as humans :) My guts feeling is that
|
|
(converted this to a draft while the smaller PRs are in flight) |
4 participants
Summary
This PR adds foundational infrastructure for mobile text input (IME) support on Android and iOS. It establishes the architecture and interfaces that platform-specific implementations will build upon.
What's included:
Core Infrastructure:
TextInputControllertrait - Platform-agnostic abstraction for IME integrationCoreTextInputController- Default implementation that bridges to Slint's TextInputSoftKeyboardState- Reports keyboard visibility and geometry for layout adjustmentime_set_preedit(),ime_commit_text(),ime_set_selection(),soft_keyboard_state()TextInput Enhancements:
preedit-textpropertypreedit-foreground,preedit-underline-color,composing-underline-colorime_set_preedit(),ime_commit_text(),ime_clear_preedit(),ime_set_composing_region()Platform Scaffolding:
AndroidTextInputHandler- Scaffolding for Android InputConnection integrationIOSTextInputHandler- Scaffolding for iOS UITextInput protocol integrationTesting Infrastructure:
simulate_ime_preedit(),simulate_ime_commit(),simulate_ime_set_composing_region()simulate_set_soft_keyboard_state(),get_soft_keyboard_state()Stats:
Known Limitations (follow-up work needed)
This is scaffolding - the platform handlers are not yet wired to actual platform APIs.
UTF-16 ↔ UTF-8 offset conversion - Both Android (Java) and iOS (NSString) use UTF-16 code unit offsets, while Rust uses UTF-8 byte offsets. The scaffolding currently passes offsets directly without conversion, which will cause issues with non-ASCII text (emoji, CJK, etc.).
Composing text cursor positioning -
AndroidTextInputHandler::set_composing_textuses simplified cursor logic (> 0 → end,<= 0 → start). Android'snewCursorPositionhas more nuanced semantics for positioning relative to surrounding text.Platform handler tests - Tests are gated by
#[cfg(target_os = "...")]and will only run on actual Android/iOS devices or emulators.Test plan
cargo test -p test-driver-rust text_preedit- Preedit integration tests passcargo test -p test-driver-rust text_soft_keyboard- Soft keyboard tests pass