Quartz - Data Visualization Library (aimed at publication, not interactive display)

Some questions (especially for @viniciusmuller, but anyone else feel free to answer):

1. Is it possible to have ExTypst generate something other than PDFs, namely SVG and PNG? SVG and PNG are useful either to deal with publishers who don’t want PDFs submitted as images, or to embed such images places like word documents

2. Currently, typsts isn’t thaaaat slow to compile, and to my great surprise I managed to install a rust toolchain quite quickly. However, it can still be a stumbling block for some. Would it be possible to use rust_precompiled to make the installation process even easier?

3. Would there be any way of querying typst for sizes of some of the content? I’m thinking in particular of text and math formulas, but I guess what I want is the ability to query the sizes of arbitrary boxes. This could be useful in order to have some interplay between the Elixir frontend (which decides things such as how many ticks there are in an axis) and Typst (which renders the tick labels, and as such can determine whether labels overlap or not); in case the labels do overlap, it would be interesting to have a way of feeding that information to Elixir so that Elixir could pick a smaller number of ticks. I’m not asking for bidirectional communication between typst and Elixir, but would it be possible for Elixir to query the sizes of at least some of the generated boxes so that it would know the optimal number of ticks?

There should be a way to configure plots in order to decide on things such as line width, default font size, etc. Although all of this can be passed manually into functions, or using raw typst script, I think it would be easier if there was a way of saying stuff like “axis labels should be 9pt and bold”. Matplotlib (which is my main inspiration), already provides a way of doing this using a “global” configure, which can be set in the current context. The idiom for doing that in Python is:

with matplotlib.rc_context({key: value, another_key: another_value, ...})
    # code that plots stuff using this style

# the style defined above is no longer valid here

The most direct correspondence to this is to use the process dictionary to define the “global config” in a way that it doesn’t need to be passed down into each function. This could be encapsulated in a function, so we could actually have something like this:

Playfair.with_config(%{my: "new", config: "here"}, fn ->
  # Code that plots things and takes the config from the process dictionary
end)

The main issue with using the process dictionary for configuration is that it stops working if we start spawning processes to draw our stuff. I don’t think this will be very common, though… Although data analysis can be parallelized (and often is!), data vizualization is often very “serial” (and not parallel) in nature. I wonder what more experienced people think about this. The Elixir formatter, for example, uses (used to use? I haven’t looked at the source for some time) the process dictionary to store some configuration options to avoid having to pass them around through all the functions.

Update

I’ve simplified the user-facing UI. Now the user can add add the plot title using simple strings (i.e. there’s now no need to build special “typst content” structures to add text elements, although that remains an option for more advanced use cases). The plotting module (YXPlot) now contains a number of configuration options, which can be set for a given plot by wrapping the plot in the right function call:

# The Playfair.Plot2D.XYPlot module contains pretty much everything
    # related to 2D plots with cartesian coordinates in which the X and Y axes
    # are perpendicular
    alias Playfair.Plot2D.XYPlot
    # Import a special sigil to allow us to write length units in a natural way
    import Playfair.Length, only: [sigil_L: 2]
    alias Playfair.Config

    # Ensure deterministic data
    :rand.seed(:exsplus, {0, 42, 0})
    # norm/2 is a function that returns a random value following
    # a Normal(mu, sigma) distribution
    reaction_times = [
      {"Group A", Enum.map(1..50, fn _ -> norm(500.0, 170.00) end)},
      {"Group B", Enum.map(1..60, fn _ -> norm(400.0, 100.0) end)},
      {"Group C", Enum.map(1..30, fn _ -> norm(790.0, 60.0) end)},
      {"Group D", Enum.map(1..80, fn _ -> norm(500.0, 150.0) end)},
    ]

    options = %{
      # The Ubuntu font ships by default with ExTypst
      text_font: "Ubuntu",
      # By setting the text size, we automatically set the size
      # for most text elements in the plot
      # (i.e. titles, axis labels, tick labels, etc)
      # NOTE: this uses a special sigil that allows us to define
      # lengths that mix different units in a way that's correctly
      # interpreted by the Typst backend.
      text_size: ~L[9pt],
      # We can set the label size specifically, and it will
      # override the `text_size` attribute
      major_tick_label_size: ~L[8pt],
      # By default the text weight is medium...
      text_weight: "medium",
      # But we can overwrite it for default plot elements
      plot_title_weight: "bold",
      axis_label_weight: "bold"
    }

    # Note that we don't need to build special structures to hold our text.
    # We can use normal strings, and Playfair will take care of applying
    # the default styles. By default, strings are escaped, but that
    # can be overriden too
    Config.with_options(options, fn ->
      XYPlot.new()
      |> XYPlot.boxplot("x", "y", reaction_times)
      |> XYPlot.put_title("A. Reaction time according to group")
      |> XYPlot.put_axis_label("y", "Reaction time (ms)")
      |> XYPlot.put_axis_label("x", "Populations")
      # Drop the lines of the "x2" (top) and "y2" (right) axes.
      # These axes are added by default to the XYPlot, but actually
      # any number of axes can be added in any location.
      |> XYPlot.drop_axes_spines(["x2", "y2"])
      |> XYPlot.render_to_pdf_file!("examples/box-plot-example-without-spines.pdf")
    end)

The result is the following figure:

image659×563 32.8 KB

I find that Elixir is missing some SVG-handling generic tools, and using Typst is a really interesting technical approach. I’m wondering though, if we are going to bring in an external tool / language, whether this has any advantage over porting out to MatPlotLib / seaborn, or bringing in VegaLite or ECharts? After all, plotting libraries costs years of heartbeats to build.

This is not the problem. Handling SVG is realy easy, actually. You can write an SVG-writing library in an afternoon, and start drawing complex stuff the next day. The problem is always text and font handling. You need a way to measure the dimensions of text boxes or math formulas. Properly typesetting (and measuring) text with modern fonts is a major undertaking. And do draw anything non-trivial you need access to the metrics of your text boxes. That is something which is very easy to do in typst (although I don’t know of a way of feeding those measurements back into Elixir). Handling text and math formulas is the real bottleneck, and that’s what Typst brings in.

It does have some advantages, yes.

1. Advantages over matplotlib: the handling of formulas in mathplotlib is really not that great. Matplotlib attempts to port the TeX math layout algorithm to Python, with very mixed success. Also, there are some design decisions in Matplotlib with which I disagree (I can detail on that a bit more), anc which in my opinion add a lot of complexity for very little gain (for example, the way matplotlib handles orthogonal axes and the way it handles legends). Text handling in matplotlib is very limit, as one can’t use different font styles in the same text block, unless we have TeX render our text elements (which requires installing TeX, which is a can of worms on its own). Eventually, I got quite experienced in drawing custom stuff on top of matplotlib, and I noticed that the part that takes “years of hardbeats” to build is text handling (which Typst already does for me, and Typst depends on some Rust crates which I believe did take multiple years to build) and parts which I’m not interested in at all, namely all the interactive stuff, which is a distraction for my goals of generating the best static output. The rest is mostly heuristics (with some actual constraint solving) for element positioning, which don’t actyually work that well and which I have to noverride all the time. Also, matplotlib is not trivial to install and requires a working python environment, and python environments are not very easy to set up. The good thing about Typst is that it’s just a “big NIF”, and if we can get it to work with RustPrecompiled, there won’t even be a compilation step

2. Advantages over VegaLite: VegaLite requires a browser-like engine to render the charts into something static for publication purposes. I don’t belive you can do it with NodeJS alone, for example. And even if I can, setting up a NodeJS environment is not that easy. Finally, from my preliminary exploration, VegaLite is not great if you want to draw custom stuff on your plots. I’m the first to admit that as it is now, Playfair can’t draw any custom stuff on the plots, but I can see pretty clearly the steps I’d have to take to draw it.

3. Advantages over ECharts: ECharts seems to have all the sabe advantages and disadvantages of VegaLite.

Handling text and math formulas is the real bottleneck, and that’s what Typst brings in.

Agreed. It’s incredible how difficult simulating word-wrap can be in SVG. But when I say missing generic SVG handling, I mean full specs and not a subset like ChunkySVG; and it should be able to round-trip parse from XML and write. I don’t know how one would do that in Elixir.

matplotlib is not trivial to install and requires a working python environment, and python environments are not very easy to set up.

Again agreed with gusto. I’ve been doing LaTeX with and without Python for a long time, and they really have some problems.

Inspired by your post, I’m looking at how to:

1. get publication quality static SVG
2. with accurately placed math
3. “without other tools”

ECharts has SSR but probably isn’t a winner since AFAIK its renderer doesn’t handle latex-like maths.

What do you think about pgfplot, perhaps in conjunction by type-setting with Tectonic? 1,2 comes from its LaTeX heritage, and 3 seems to be a similar Rust/cargo affair as bringing in Typst.

That’s my whole point. SVG is not the problem, the problem is correct text handling. That’s inherent to the complexity of human languages and typographical rules. You need an actual text-rendering engine for that, and Typst is such a rendering engine (and a very small one at that in terms of binary size and memory use).

Parsing the XML you’ve just written isn’t actually helpful for this. You actually need to query an SVG renderer to get the proper line break locations.

Tectonic seems to be strictly more complex than Typst, although it does keep compatibility with latex, so that would be a plus, I guess?

Hey there! Thanks for the interest in Typst, I see you’re building something really nice with it!

Is it possible to have ExTypst generate something other than PDFs, namely SVG and PNG?

Currently upstream typst does not appear to provide provide SVG output, as there’s an open issue for it. About PNG, it seems that typst supports PNG output and when I get some time I’ll give it a try, but in the mean time if you’re feeling adventurous, a PR would also be welcome!

Would it be possible to use rust_precompiled to make the installation process even easier?

That would be nice, when writing the bindings I didn’t try to add precompiled NIFs because I’m not familiar with them and mostly because I didn’t know if there would be interest from the community in typst. I’ll see how precompiled NIFs work and about adding support for them.

Would there be any way of querying typst for sizes of some of the content?

I think in this case, this is something that needs to be done on the typst side, since we just format a typst document and give it to the typst formatter, which already outputs a PDF binary.
Also, most of their API in rust is private, so that means external code using it cannot access a lot of properties/methods.

Is there a way of storing custom metadata in a PDF file using Typst? One could write a Typst program which would generate objects and store their metrics in PDF metadata. Then, one could parse that metadata out of the PDF using Elixir and get access to it

It seems that you can use the document function to set metadata, but it appears to be only limited to author and title. But if you can serialize/deserialize what you need in string format, I think this approach could work

Yes, I could encone arbitrary data in the title and then somehow extract the data from the PDF. That seems interesting.