On the other hand, on-the-fly gesture design calls for careful dynamic management of event streams and processing state for a potentially large number of bound gestures. A typical multiple view visualization with tens of views could potentially require instantiation of hundreds of DataGestures. (Note that writing code for more than a few, special-purpose gestures will rarely be necessary in practice. Improvise provides a feature to load a pre-defined library of expressions. This allows easy reuse and variation of the most common visualization gestures, such as zooming with the keyboard and lasso selection with the mouse, during design.) The architecture is structured to share both raw and filtered event streams across gestures for sake of time and space efficiency.

Each View delegates gesture processing to a GestureProxy that encapsulates all gesture processing for that View. The GestureProxy evaluates the four expressions of the DataGesture to map inputs into outputs along the gesture processing pipeline. Because Improvise is a live design system, each GestureProxy must reconstruct its pipeline in response to any changes to gesture expressions. We currently rebuild the entire pipeline, including raw and filtered stream dependencies. Although in theory this approach could wipe out an in-progress gesture that a user wishes to continue, it's unlikely to happen often in practice, and even if it did, it would be the user making the choice to transition to visualization design. pipeline, it xpression doesn't depend on individual events.

A pattern expression describes an EventPattern. An EventPattern is simply a sequence of EventSegments, each described in terms of a source of raw events (EventStream), a filter to apply to it (EventPredicate), and how many times that type of filtered event can or should recur in that portion of the pattern (EventQuantifier). EventStreams and EventQuantifiers are simple enumerations, allowing a designer to simply select them from a list. EventPredicates are simple boolean expressions defined in terms of an event's attributes (point on the screen, keyboard modifier attributes, etc.) For instance, the designer might define an EventSegment as (SwingMouseClick, if(ShiftDown), OnceOrMore) for a subsequence consisting of any number of shift-click events in the View.

The GestureProxy extracts a list of EventStream-EventPredicate pairs from the sequence of EventSegments. For each unique pair it creates an EventAccumulator that maintains a list of events from the stream that pass the predicate. From the EventAccumulators it creates an EventCombinator that interleaves the events from these EventAccumulators. As a result, the EventCombinator accumulates all and only the events that are pertinant to its gesture. Upon receiving each event, the EventCombinator applies the EventPattern to the accumulated event list, producing a sequence of GestureMatches. Any GestureMatches are passed down the pipeline in order. If a GestureMatch is complete, events up through its end are forgotten by the EventCombinator. If there is a most recent, incomplete GestureMatch, it will contain only match information up through the most recent event, and the EventCombinator will remember events from its start to match against as the gesture completes (or not).

We current implement pattern matching by encoding the pattern as a regular expression, using the regular expression capabilities built into Java. The EventCombinator's accumulated event list is encoded as a string to match against. The regular expression is encoded in a way that prohibits overlapping matches. Perhaps surprisingly, the architecture does not require that any pattern expression establish certain termination for a gesture. It is left up to the visualization designer to specify gestures that co-exist well as a useful and usable interaction suite in each View. It is also left to designer to incorporate terminating sentinals in final EventSegments where needed. There are a variety of ways to do this using combinations of predicates and quantifiers. Even so, very large accumulations are allowed, and the possibility of unbounded accumulation of event streams is problematic in principle. We currently rely on the user as human-in-the-loop to preclude accumulations of a size (likely at least thousands of events over tens of seconds) that might unduly overwhelm computational resources.

The remainder of the pipeline is substantially simpler. Each GestureMatch contains a list of the events involved in a match. Events are numbered by the matching EventSegment in which they happened (using regular expression match groups). The GestureProxy evaluates the geometry expression to map each GestureMatch into one or more GestureGeometries. Using the existing declarative language in Improvise, the expression can map individual and aggregated event attributes into common 2-D geometric objects such as points, line segments, circles, rectangles, polygons, simplicies, lines, rays, cones, etc. Operators like the conditional cascade can be used to generate appropriate geometries from the incoming events in an incomplete GestureMatch. For instance, a lasso matched by the pattern (Once(MousePressed), NoneOrMore(MouseDragged), Once(MouseReleased)) can be calculated as lasso = polyline(point@1 + points@3 + points@5).