HN Theater @HNTheaterMonth

I think notebooks are great for building a presentation / reports of a data exploration, but I agree that they shouldn't represent programs.

Rstudio won't export an R-markdown notebook to PDF without rerunning all cells sequentially. It's great for keeping the code "in check", but for things that takes hours to run, it can be quite annoying.

I do think most of the problems with notebooks go away if you somehow force sequential running before presenting it to other people.

Another really cool take on notebooks are Julia's Pluto Notebooks. Like a spreadsheet, there is no sequence that cells are run in. Everything is updated simultaneously. It's kinda hard to explain, but the JuliacCn presentation on these is absolutely wonderful: https://www.youtube.com/watch?v=IAF8DjrQSSk

I think every new notebook project should take ideas from Julia's Pluto: https://www.youtube.com/watch?v=IAF8DjrQSSk

Working directly with the source files is just one of the niceties it has.

⬐ KMag

Very interesting. Thanks for the pointer. LWN has a nice textual introduction: https://lwn.net/Articles/835930/

⬐ KMag

Thanks again for the pointer. Pluto really seems to nail the major pain points from notebooks: for a quick summary of Julia's Pluto from https://lwn.net/Articles/835930/ :

1. Pluto's on-disk format is genuine executable Julia, with metadata in comments. Code review happens on the actually executed code, so highly regulated/otherwise conservative environments don't have to worry about the single source of truth deviating from what code reviewers see.

2. Pluto disallows multiple declarations of global variables. This prevents problems with getting different results, depending in which snippets within the notebook get executed in which order.

3. Pluto makes sure the notebook's displayed output is always in-sync with its source code; it eliminates the problem of someone getting some result, changing some code, and sharing a notebook with results that don't match the code. It does this by performing dependency analysis across the code fragments and topologically sorting them. It then re-evaluates all dependent code snippets when a given snippet is modified.

Wow. This is the way.

⬐ dnautics

My only complaint about the pluto system is it's use of strange extended ascii characters.

⬐ st1x7

That's a Julia thing, not a Pluto thing. Not a fan personally but I think that's a minority opinion in the Julia community.

⬐ dnautics

I'm fine with Julia's use of unicode for math symbols in general.

The strange ascii characters I'm speaking of here are used to annotate block boundaries in pluto's generated code, they are literally in # comments so they could have used anything.

I think they are the old school ascii box drawing characters. They might live in a separate unicode era these days.

The closest may be Pluto.jl https://www.youtube.com/watch?v=IAF8DjrQSSk

In Pluto plots, calculations get bound to data and when you change that data the effects immediately ripple through.

Personally if I work with table data in Julia, I will use it with something like Table Tool https://apps.apple.com/us/app/table-tool/id1122008420?mt=12

It lets you view and edit a CSV file. Saving and loading CSV files from Julia is very quick anyway. You can use a couple of key strokes to bring back a previous command.

You also got some newer ones such as Tad: https://www.tadviewer.com

The Pluto notebook for Julia does exactly the same. It creates dependencies between code cells and data automatically. If you change any data or code, that code ripples through through the rest of the notebook. So e.g. you could bind a slider to a variable and if that variable is used in a plot, dragging that slider will modify the plot in realtime.

Demonstration of Pluto.jl here:

https://www.youtube.com/watch?v=IAF8DjrQSSk

It is linked in the article, but I would like to draw attention to the wonderful JuliaCon talk presenting Pluto:

https://www.youtube.com/watch?v=IAF8DjrQSSk

It's mentioned in the video [1], it does static analysis of the code to create a graph of dependencies (for example which cell uses a variable defined by another cell), so when you update any cell it will find what cells are affected by the change (the downstream nodes on a directed acyclic graph) and only evals the code on them (instead of running everything). Julia is particularly good for those kind of code analysis since it's a very Lispy language.

It also does a trick of creating new modules to manipulate scope to make deleted variables/import/cells invisible (and therefore free to be garbage collected).

[1] https://youtu.be/IAF8DjrQSSk?t=596

Really happy to see that the good ideas from Observable notebooks are being copied elsewhere. I don't use Julia myself at the moment (although I think it's a beautiful language), but I know some people who will be very happy with this! Also, the playful enthusiasm in the presentation video linked in the description just makes me smile:

https://www.youtube.com/watch?v=IAF8DjrQSSk

⬐ benhurmarcel

Very interesting, thanks.

I don't understand the point of the "reactive" cell order, instead of conventionally doing top to bottom. It seems like it goes against the idea of "the program state being completely described by the code you see".

⬐ uoaei

Reactive notebooks are closer to the idea you mention than other kinds of notebooks, because global state is reflected in the outputs of each cell. The biggest source of bugs when working in Jupyter for instance is that a variable was re-defined somewhere and it's not easy to see that it is changed when you have a cell somewhere above that has the output `foo = x` when the global state claims that foo = y.

⬐ TeMPOraL

In reactive notebooks, the program state is still being completely described by the code you see - it's just that it doesn't have to be in top-down order! Instead of forcing you to manually define everything before use, a reactive notebook just looks at the references you make, forms a dependency graph, and then evaluates it in topological order (and, as optimization, only re-evaluates things that depend on the thing you've just changed).

This leads to a much nicer code organization, particularly when you're writing something resembling an interactive document (e.g. an "explorable explanation"). Much of the documents I do on ObservableHQ has roughly the following structure:

  Title
  Prose
  Visualization
  Interactive UI elements (possibly mixed
    with further visualizations)

  ------------------------------
  All the actual source code
  powering the implementation,
  organized in readability order.

⬐ ddragon

Even if you do it completely top down, it doesn't mean all cells need to update if you change something on the top, so you'll still profit from the dependency graph.

And notebooks are mostly for exploration, and you don't really have a fixed order. You start getting the data, then you run the model, then you go back and change the data in the import... The order of your internal logic (or how you want to explain it to others) isn't necessarily linear, and of course you can just go back and write like a normal program but you lose the chain of changes that led you to the result (in this case the compromise is that your chain is restricted, like you said you can't define the same thing twice, but in exchange the notebook will provide with you the program order for free).

⬐ vanderZwan

So in the context of these notebooks it is actually quite useful to not depend on cell order, because the idea is that notebooks aren't simply programs or scripts, they are (potentially interactive) documents. You can write an article that presents the results of your script, with cells that contain text, interactive widgets and plots at the top, and put the code and data that generates these plots in "appendix" cells below that. Observable, a "JavaScript ancestor" of Pluto.js, has plenty examples of these:

https://observablehq.com

Also, I guess that you haven't used notebook environments like Jupyter before, so a bit of historical context might help. In Jupyter, cells aren't necessarily executed top-to-bottom, they are executed when the user asks it too. The result is then stored in the global state (well, assuming there is a global variable that the data is assigned to). This means that cells that depend on other cells also depend on the order in which those cells were executed. Worse still, if you write your notebook in a sloppy manner, you can end up with a state that you cannot reproduce from the still-remaining code (for example, you can have variables A and B, B is generated from the result of A, then you remove A. Because Jupyter is not reactive this does not update B, so your notebook keeps working just fine... until you decide to edit B). So previously, notebook-like environments made it really easy to introduce bugs like this.

With reactive cells you don't have to think of state. It's kind of like pure functional programming: it removes global side-effects. And note how on a technical level, making cells execute top-to-bottom is really just very a simple way to enforce that cells must executed in order of dependency! So either option insists that this global-state-that-does-not-respect-dependencies is a big problem that should be avoided, they just present different solutions for the problem.

⬐ benhurmarcel

Thanks a lot for the write-up.

They had a fun talk at JuliaCon 2020: https://www.youtube.com/watch?v=IAF8DjrQSSk :)

The game changer for ordinary technical programming folks from JuliaCon 2020 is the introduction of a new "reactive" notebook, Pluto.jl. "Reactive" means that -- unlike Jupyter -- it automatically updates the whole notebook, like a spreadsheet, when you change a value in any cell.

I am a Julia and Jupyter newbie, but I picked up on it easily.

Watch the following video made by Pluto's creators:

https://www.youtube.com/watch?v=IAF8DjrQSSk

Go here for Pluto installation instructions:

https://github.com/fonsp/Pluto.jl

One thing that tripped me up, which is a generic Julia REPL issue: You enter the Julia package manager with "]" on the Julia> command line -- but how do you exit the package manager? Answer: Either Backspace or Ctrl-C.