Update on WEAVE government data visualization software - O'Reilly Radar
Update on WEAVE government data visualization software
User-friendly visualization tools are coming early next year.
by Andy Oram | @praxagora | Comments: 1 | 18 November 2010
On Tuesday I heard the dynamic University of Massachusetts at Lowell professor Georges Grinstein talk about WEAVE (Web-based Analysis and Visualization Environment), a visualization tool for public data. One of the coolest things about WEAVE is the very idea of it. About 10 government agencies decided three years ago (before the Gov 2.0 movement was hot) to put their data out for easy public consumption, and to collaborate around it with the hope of eventually being able to combine all their data. These governments have combined into the Open Indicators Consortium to fund and guide development.
When WEAVE started it was pretty ground-breaking; now one can cite lots of related projects. Data.gov alone (the major US federal site for data) boasted 305,692 datasets when I checked it right after Grinstein's talk. But as Grinstein points out, most sites think they're being hip just by putting out computer-consumable data sets. These are a big step up from PDFs, but the missing piece is ways to interpret the data, which is being left up to outside programmers.
Note, for instance, that one of the top prize winners in the 2010 CivicApps contest run by Portland Oregon, was just a tool for turning Portland's data into a format easier for programs to consume. WEAVE goes a lot further, providing a bunch of cool visualization tools that sites can drop in for the public to enjoy.
Between vision and utopia lies a lot of stumbling and fumbling, and Grinstein let us in on a little of it on Tuesday. One stumble was dropped Internet connectivity in our high-style hotel right in Cambridge's Harvard Square -- a pocket of Third-World deprivation in the midst of one of the world's broadband utopias. So we didn't get to see much of the visualizations--and all the people located remote from us who wanted to get a live feed were shut out -- but from the stills I could tell that WEAVE offered many of the same animated, interactive visualizations that one can build with the Processing language.
These apps typically use color, texture, size, and position in creative ways and then let you drag, click, zoom in and out, and manipulate the data yourself. For instance, if you see a state map and click on a county, it might zoom out to occupy the screen while in the background your browser requests detailed county data from the server.
Java-based Processing is supposed to be able to handle data sets many orders of magnitude greater than the ones amenable to Flash/Flex, which is the basis of WEAVE. (Apple's decision not to support Flash on iOS devices is clearly another stumble that lies outside of Grinstein's control.) The WEAVE team is excited about the potential of rewriting the display engine in HTML5, but they have to see whether their backers will fund development. I suspect that the government agencies don't understand what HTML5 will enable (although telling them it will run on iOS devices may persuade them) but I also trust that the WEAVE team will get the port done by hook or by crook.
The WEAVE team designed their server to use generic, open source components so that installation would be easy -- but even so, there were difficulties with the different host server security aspects and the team had to spend a lot of time they didn't budget for on sysadmin support, another stumble outside their control.
Tuesday's presentation was typically Cantabrigian, from the Boston accents (we heard a lot about "visualizing dayter") to the interminable questions about formats, architecture, and other technical details from an audience of non-profit reps who had spent as much time tuning a computer system as distributing meals to the needy. There wasn't time for a lot of technical discussion, but I caught that WEAVE doesn't have to run on the server of the agency that provides the data; the agency can feed data to a WEAVE server running somewhere else. The end-user needs nothing except a browser with a Flash plugin.
We also had a discussion about the code and license status of WEAVE, which will be released in or shortly before March of next year. The University of Massachusetts at Lowell decided there's too much intellectual property tied up in WEAVE to release it as open source, and Grinstein feels fine about that because in his experience, good open source projects mature in a closed environment. I thought of two counter-examples that I'm using right now, GNU/Linux and GNOME, and I exchanged some email with Grinstein where he gave his interpretation of their history and how they too reflect the importance of a long gestation.
As it is, the source code will be published and anyone can use WEAVE free for non-commercial use, while commercial users will pay very modest fees. Donations of code will definitely be appreciated, but Grinstein expects that one-quarter of the team's time will be spent checking over, testing, and vetting donations. Not a free lunch for them. Grinstein wants to make sure that consortium members have good code, because their public users will have a low tolerance for bugs.
Still, in a few years, WEAVE may well go out under an open source license.
What's in the future for WEAVE? One of the most intriguing features they're considering is collaboration. Even upon the first release, you should be able to run a visualization, save it, and pass the URL around for others to comment on. Eventually they hope to let users work together to produce and view interactive animations in real time. They also are looking for ways to filter data on the server side so that less needs to be transmitted over the network.
Development process has a big effect on a project's success, and the WEAVE process is hard to classify -- a bit gawky, in my impression, although they are using agile methods. WEAVE seems like a "get it out there" kind of project rather than a grand-vision kind of project, which is fine and may be the key to success. But it means such compromises as making it easy for agencies to submit spreadsheet content instead of trying to formalize a scheme for accepting well-formatted data.
And despite the leaning toward open software, WEAVE is very much a U.Mass. Lowell Computer Science department project, beholden to the dictates of the university and the research needs of the students. This doesn't mean the software will be bad -- in fact, Grinstein tells me he has built commercial-grade software and built entire companies using student programmers.
WEAVE also has a clearly delineated set of funders, whose priorities will direct development more than user reaction to the visualizations -- or rather, user reactions are important but will be filtered heavily through the funders. But WEAVE has generated a lot of excitement among the public anyway, so I'm sure they'll line up to try it, and in three to six months we'll be able to judge its value.
Related:
- Local government: data supplier
- The story of BrightScope: Data drives the innovation economy
- Open health data: Spurring better decisions and new businesses
Check out WEAVE...