Plotting & Graphs for the web

by Paul Agapow
in programming

tagged plotting, graphs, web-development. visualisation

Or, "there must be simpler way to do graphs in Rails"

In several recent projects, I've needed to produce graphs of various epidemiological & bioinformatic datasets on demand in web applications. Unfortunately, in the world of Ruby and Rails, the choices of toolkits aren't as straightforward as hoped. This is a summary of the available choices and what I found.

Background & assumptions

  • Must support box-and-whisker charts. User requirements.
  • Must be guaranteed to preserve privacy of data, e.g. no rendering on a remote platform. User privacy requirements (PII)
  • Must work in a web environment and Rails
  • Must work with Ruby
  • Should be open and non-encumbered license-wise, e.g. no commercial products, crippleware, limited versions. So that we have full access to the code and may distribute the products without any problem.
  • Should produce image graphs that can be copied and downloaded. Users may want to copy grants for inclusion in reports. This rules out Flash and Javascript-based solutions.
  • Should be an active project and maintained. So there will be fewer problems with incompatibility.
  • Should require a minimal amount of customization. Less work, better code quality.
  • Should work with Ruby 1.8.6. That's what we develop with, although deployment will be with a later version.
  • Would like to avoid wonky or non-standard tech, e.g. anything requiring a flash plugin or browser upgrades. Don't want another user issue to manage.
  • Would like to avoid installation of other requisite libraries, e.g. ImageMagick. Less to install and maintain, less vulnerable points for incompatibility.
  • Would like to avoid commercial solutions, because of difficulties in finding money and being able to share any sofwtare developed freely
  • Would like a Ruby module to wrap any low level details if necessary
  • Note there's a shifting landscape in browser capabilities, with more recent browser version being mucgh more capable than others
  • Note there's a wide body of possible local & remote users, i.e. not a small group we can encourage to install a plugin.
  • Note most users (including most of HPA) still have older versions of Internet Explorer (e.g. v8 or earlier)

Possible general solutions

SVG in the browser

Generate and render SVG for direct display in the browser

  • It's easy to write
  • It looks good and scales well
  • It looks the same everywhere
  • There's some nice toolkits (e.g. Raphael)
  • Can be interactive

But:

  • Before version 9, IE doesn't handle SVG at all
  • Outside technical audiences, no one knows what to do with an SVG file, and few of the general programs (e.g. Powerpoint) can handle them. There are some clever export mechanisms that can convert in the browser but that's extra work.
  • You can use a plugin to get SVG rendering for IE, but a large number of them are being discontinued or neglected. And you have to use a plugin.

Verdict: no good, we have to support IE8.

Canvas

Use the canvas element to draw pictures.

  • Similar comments to SVG

But:

  • Similar problems to SVG
  • The full canvas element is a component of HTML5, which is implemented in a very small number of recent browsers.

Verdict: no good, we need to support legacy browsers

SVG behind the scenes

Use SVG to make pictures and then convert to display format

  • Get ease of use of SVG and downloadable pics
  • Easy to change size / resolution

But:

  • Need a converter. Deal with the install headache of ImageMagick?

Verdict: maybe

Javascript in the browser

Clever drawing solution using JS that make pretty pictures (e.g. Plotr, Raphael)

  • Looks nice
  • Some are interactive

But:

  • Can't save directly
  • Entails working at least a little bit in Javascript, so do we want to bring another language / development mode into the mix

Verdict: probably not

Google Visualization & Charts

Various Google javascript intermediated tools for graphing.

  • Looks nice
  • API fairly straightforward
  • A number of gems for using this

But:

  • Works by calling and sending data to the Google mothership, a big privacy no-no
  • From experience, converting all data into another language layer (JSON) is a bit irritating
  • Unclear if we can save images

Verdict: not at the moment

Call external program serverside

Various and diverse methods that call some commandline thing on the server like MetaPost or GnuPlot, capture result and return to browser.

  • Proven rendering solution
  • Delivers right format

But:

  • Another component in the system to go wrong
  • Have to deliver data to external program somehow
  • Privacy concerns mean we have to careful with any saved files

Verdict: a bit too fiddly, hope to avoid

Fucking Flash

Flash-based solutions.

  • Can look nice
  • Can be interactive

But:

  • Have to ensure users have plugins
  • Don't necessarily have an image export capability

Verdict: no

Specific solutions

Protovis

A server-side library for building graphics and visualizations.

  • Very nice looking
  • There's a lot of smartness in how graphs are built - scaling data to the size of the axes, overlaying charts, etc.
  • Does lots of different types of charts
  • There's a Ruby wrapper for it, RubyVis
  • General drawing toolkit
  • Implementation of protovis
  • Still calls REXML
  • 1.8.7 minmum

But:

  • Really, truly is a toolkit. Just to do a simple barchart, you'll have to write a lot of code. There's no “barchart()” or “scatter_graph()” functions.
  • Produces SVG, so you have the conversion problem.

Highcharts

A free-for-non-commercial javascript based toolkit.

  • Lots of graph types, looks good.
  • Has an export module for saving graphics as bitmaps

But:

  • Have to encode data for Javascript.
  • Ruby gem is for rails 3+
  • Export requires either using public remote service or running your own

Fusioncharts

  • Flash
  • Nice charts
  • "Fusioncharts Free"

amCharts

  • Flash
  • Nice charts
  • "the only limitation of the free version ..."

Open Flash Chart

  • Free
  • It's still flash

Anychart

  • Flash
  • "download trial versions"

Seer

  • Wraps Google visualization API

Highcharts

  • Uses javascript
  • Nice charts & hints
  • Has ruby library
  • "free for non-commercial uses"

Rifgraf

  • Ruby serverside library
  • For timeline / progressive data

Gruff

  • Ruby serverside library
  • Nice looking graphs
  • Require imagemagick
  • No box and whisker plots
  • Active as of September 2010

Scruffy

  • Ruby serverside library
  • Status unclear, dead since 2009? Looks like there's a very recent fork.
  • No box and whisker plots
  • USes SVG internally
  • Says charts can be added

RGPlot

  • Ruby serverside library
  • Uses gnuplot. Seriously?

Prawngraphs

  • Ruby serverside library
  • Render to pdf
  • A little thin on documentation and demos

rchart

  • Ruby serverside library
  • "A port of PChart"
  • Requires gd

statsample

  • Ruby serverside library
  • A large toolkit for analysing and plotting population data
  • Many dependcies or included libraries
  • Calls rubyvis which calls rexml/formatter/default which doesn't exist in Ruby 1.8.6

gerbilcharts

  • Ruby serverside library
  • For time series data

lilygraph

  • Ruby serverside library
  • Lightly documented

SVG::Graph

  • General drawing toolkit
  • Uses SVG
  • Can render SVG & PNG
  • Graph choice is a little thin?
  • Loads, at least
  • Uses Rexml but seems fine with Ruby 1.8.6 version
  • Still have to export to an image via something like RMagick

Processing

  • General visualization toolkit
  • For interactive / compiled visualisations

RMagick

  • General drawing toolkit
  • Draw the images yourself?
  • RMagick has RVG, an SVG like drawing API

Rendersvg

  • General drawing toolkit
  • No examples

Notes

  • REXML and Image/GraghicMagick turned out to be a major problem. Several of the libraries have major (and differing) version dependencies which turns into a big ball of conflicting requirements quickly.
  • REXML cannot be downloaded and installed locally, which would patch protovis

Conclusions

There's a lot of choices out there. But in my case, there's not a lot of good choices. The (what I would hold is reasonable) desire to avoid flash or commercial solutions lops off a good proportion of choices. The (unavoidable due to privacy concerns and users wanting to download graphs as images) need to generate all charts on the server excludes several nice javascript solutions. With what remains, you're confronted by three big problems:

  • The selection of graph types generated is often poor
  • The libraries rely on ImageMagick and (thus) Rmagick, which are a pain to install (e.g. "unexpected segmentation fault at ...")
  • You get caught up in the Ruby dependency and versioning hell, where the sum of libraries required force a narrow window of versions of other libraries, which in turn constrains what you can do elsewhere in the application and makes hosting the application a pain (e.g. one of your apps needs ruby 1.8.7 and these libraries, while another needs 1.9.2 and these other libraries)

In short, it sucks. I've cobbled together some graphs with RubyVis (nice output but needed a lot of code) and done some work with Gruff (simpler), with both needing ImageMagick in the background. This is much harder than it should be.