Modifier keys don't work well at the moment because they trigger functions in the UI instead of being restricted to manipulating the annotation border