Study leads to a system that lets people use simple English to create complex machine learning-driven visualizations

The ubiquity and sheer volume of data generated today give experts in virtually every domain ample information to track everything from financial trends, disaster evacuation routes, and street traffic, to animal migrations, weather patterns, and disease vectors. But using this data to build visualizations of complex predictive models using machine learning is a challenge to experts who lack the requisite computer science skills.

A team at the NYU Tandon School of Engineering's Visualization and Data Analytics (VIDA) lab, led by Claudio Silva, professor in the department of computer science and engineering, developed a framework called VisFlow, by which those who may not be experts in machine learning can create highly flexible data visualizations from almost any data. Furthermore, the team made it easier and more intuitive to edit these models by developing an extension of VisFlow called FlowSense, which allows users to synthesize data exploration pipelines through a natural language interface.

The research, "FlowSense: A Natural Language Interface for Visual Data Exploration with a Dataflow System" won the best-paper award at this year's IEEE Conference on Visual Analytics Science and Technology (VAST).

On Tuesday, October 22, Bowen Yu, who received his doctorate at NYU Tandon under Silva, will present the paper at the opening plenary session of the IEEE Visualization Conference (IEEE VIS) in Vancouver, British Columbia. The study is one of several papers spotlighting VIDA research that will be presented at IEEE VIS, the leading venue for visualization research and a premier conference for computer graphics.

At the conference, collaborators with VIDA, which has established itself as a leading research center for data visualization, will present visualization modeling projects with applications in astronomy, medicine, and climate research developed at or with the center:

• OpenSpace, a System for Astrographics is being used worldwide in planetariums, museums, and other contexts to explore the solar system and universe
• Motion Browser: Visualizing and Understanding Complex Upper Limb Movement under Obstetrical Brachial Plexus Injuries is a collaboration between computer scientists, orthopedic surgeons, and rehabilitation physicians that could lead to new treatments for brachial nerve injuries and hypotheses for future research
• The Effect of Color Scales on Climate Scientists' Objective and Subjective Performance in Spatial Data Analysis Tasks is a web-based user study that takes a close look at the efficacy of the widely used practice of superimposing color scales on geographic maps

VisFlow, introduced in 2017 and funded in part by the Defense Advanced Research Projects Agency's Data Driven Discovery of Models program is a web-based framework that allows the user to use simple drag-and-drop actions to interact with data easily, letting users create visual data models based on time series, networks, geographical locations, and more, all of which can be formed into a compact and interactive visualization dashboard.

Yu said FlowSense takes these capabilities a step further. "Imagine if one could simply speak or type a sentence to activate a dataflow diagram," he said. "This capability would make non-experts more comfortable users, while providing experienced users with shortcuts. We believe that with natural language support we can mitigate the learning curve for a system like this and make dataflow more accessible" he said.

Silva, an IEEE Fellow who is affiliated with NYU's Courant Institute for Mathematical Sciences, Center for Data Science, Center for Urban Science and Progress, and Center for Advanced Technology in Telecommunications, added, "We're offering VisFlow and FlowSense as open-source, free-to-all code-based frameworks on github, as a way to motivate further development for visualization purposes. There really is a lot more research that could be done in this area, and it is my hope that FlowSense will be a major stimulant for more collaborative work in making dataflow systems more flexible, easy to use, and popular among data analysts."