Improvise is implemented as a Java desktop application. Its user interfaces are implemented in Java Swing. More recently, JavaFX has become increasingly popular as the user interface library of choice for developing new Java applications. Compared with its predecessor Swing, JavaFX incorporates newer technologies such as animation, styling with Cascading Style Sheets (CSS), graphical effects, and touch interaction. All four have strong potential to support advanced capabilities in visualization designs, whether it is through tweening of interaction-driven or query-driven transitions (animation), thematic enhancement of designs particularly in ambient visualization (styling), utilization of blurring and shadows in visual encodings (effects), or exploiting the fine-grained control of human hand movements for expression of subtle geometries (touch).

We are particularly interested in touch interaction and its potential for use in gesture design. JavaFX already supports gestures for general UI navigation, including rotating, scrolling, swiping, and zooming. We considered the viability of user JavaFX touch for geometric data editing in our gesture architecture as well. Touch may support more direct and flexible expression of geometries and geometric characteristics than mouse or keyboard. It may also facilitate gesture performance on devices more suited to touch than mouse or keyboard, such as phones and tablets. In anticipation of supporting touch input in data editing gestures, the implementation of the gesture architecture in Improvise supports not only Swing input event types but also all JavaFX input event types, including mouse, keyboard, touch, and its four gesture event types (rotate, scroll, swipe, and zoom).

Improvise includes a wide variety of view types for use in visualization layouts, but they are all implemented in Swing. It is necessary to reimplement those views to take advantage of touch and other JavaFX input types. While in principle such reimplementation is straightforward, there are practical concerns regarding the correspondence between implementations of any given view. An especially important concern is performance. JavaFX is architected to support sophisticated graphical effects and transformations under a frame-based rendering model. Its scalability for larger data sets must be ascertained for it to be viable in many visualization designs, particularly designs that contain many views and/or highly interactive visual querying.

Our work in this direction began in the Fall 2020 semester and is continuing. We set out to assess the performance of JavaFX for highly interactive visual querying that we anticipate for gesture-based data editing in coordinated multiple views, asking foremost the question: Is JavaFX fast enough? Many visualizations display significant amounts of data (tens of thousands items or more) in each view, and must run queries and render results fast enough to update those views in response to inputs happening at timescales of tens of milliseconds or less. Maintaining responsiveness, preferably below the commonly accepted guideline of 100ms for users to perceive immediacy, is key. Because JavaFX supports both low-level graphical primitives and high-level graphics effects, we also set out to compare performance spanning the range of basic to advanced visual encoding choices that might arise in visualization designs that include views implemented in JavaFX.

We developed two simple test applications that display multiple items animating simultaneously (see above). The first application contains one window. The second application contains three windows of the same size. We measured performance in frames per second (FPS), varying both window size and the number of items displayed (see below).

At 10,000 items, performance is around 10 FPS. In both applications performance above that number is inversely proportional to the number of items. Performance does not vary dramatically as a function of window size. We conclude that overall performance in JavaFX is good enough for anticipated applications of gesture-based data editing, in which the number of items in views—or at least the number of items likely to be manipulated at any given time—will usually be small, often substantially so. Consequently, we concluded that views implemented in JavaFX are suitable for general inclusion in Improvise and as promising targets for gesture-based data editing designs, including with touch inputs.

The next step was to add a new JavaFX-based view to the collection of existing Swing views in Improvise. As one of the most common and general-purpose view types, we chose a scatterplot/2-D Cartesian view, called a "Plane2D View" in Improvise, as the first new view to add. Each view module in Improvise includes a collection of "glyph" classes that encapsulate the individual visually encoded data items in a view. We have so far reimplemented many of the existing Swing-based glyphs for the JavaFX view, starting with glyph classes for basic and common geometric shapes: circle, rectangle, arc (on a circle), bow (like a bowtie), and line (segment). Event handling and glyph rendering code were adapted to work with JavaFX. The implementation of Swing and JavaFX versions are otherwise nearly identical, including the code to interface the view with the Live Properties and Coordinated Queries architectures, which are not specific to Swing or JavaFX. As a result, the JavaFX view can be incorporated into coordinated multiple views layouts with any of the views implemented in Swing, including the original Plane2D View (see examples below).

The diagram below summarizes input handling and rendering in JavaFX views. JavaFX input events are handed off (asynchronously from the JavaFX main thread to the Swing event handling thread) to a Handler for translation into changes to the parameters in the coordination graph of the visualization. Upon receiving any notifications from Coordinated Queries of changes to parameters, including as an indirect consequence of the view's own input event handling, the Renderer creates a collection of glyph objects that describe the visual encoding of data items to be displayed in the view.

The Handler and Renderer notify the Processor to update the scene graph displayed in the view. From the Handler, update notifications involve parameters that affect the appearance of the view overall, including the background/fill color and the horizontal and vertical ranges that determine the visible extent of the view (through application of translation and scaling affine transformations to its scene graph). From the Renderer, update notifications generate a subgraph of JavaFX nodes from the "rendered" collection of glyph objects, for inclusion in the view's scene graph to represent its visually encoded data contents.

The JFXPane manages the JavaFX scene graph inside a Swing JComponent and redisplays the latest scene graph at regular intervals (in concert with the low-level rendering and screen update pipeline in JavaFX). This approach allows JavaFX views inside Improvise's overall Swing UI to update seamlessly in response to both user inputs performed both locally in the JavaFX view itself and globally in any of the other Swing or JavaFX views throughout the visualization.

The JavaFX Plane2D View currently implements most of the essential features of its Swing counterpart. Features that remain to be translated over include multiple data layers, optional decorations (including grid lines and cursor coordinate labeling), and the full set of more exotic and nested view glyph types. We are also working on implementing new glyph types that utilize the graphical effects features of JavaFX such as blurring and shadowing. A longer term goal is to add functions to the Coordinated Queries expression language to support JavaFX-style animation of graphical elements rendered inside glyphs.

While it is straightforward to add touch handling to the JavaFX version, we discovered that such handling is limited to touch screens and does not yet work with more common touch devices like trackpads. Such devices are much more commonly available for anticipated users of gesture-based data editing, particularly the digital humanities users who are our primary target users for this project. In our ongoing JavaFX development efforts we intend to return to touch handling when it is more broadly supported on common touch devices.