HN Theater @HNTheaterMonth

What a great article - and the way it's presented is insanely great. It reminds me of a talk given by Bret Victor

https://youtu.be/oUaOucZRlmE?t=1175 (Media for Thinking the Unthinkable)

⬐ throwaway6734

It looks like it's built with the Distill framework. Distill.pub has some other really neat work.

⬐ youssefabdelm

Was just about to say it reminded me of Bret Victor. But someone needs to synthesize these technical articles with the field of General Semantics ( https://www.youtube.com/playlist?list=PLaoJIXlyLvLkMQUtbiTi1... ) to make it even clearer for the curious complete beginner to mathematical/technical thinking/approaches.

A deeper awareness of all the jargon one is using + deep intuitive layman explanation would be valuable.

⬐ 123pie123

that's neat, do you know what the drawing / visualization program is called?

You make a great point about notations becoming more abstract for more advanced use cases, indeed this may be a "natural" process. However I'd like to add nuance with the observation that Bret may not be suggesting that we reverse the trend from abstract notation back to concrete visualizations. Actually I think he is calling for a rich variety of visual/concrete notations and symbolic/abstract notations that work in concert together to express our dynamic concepts, and extending those with the use of the dynamism of the computing medium to add interactive aspects to the notations (ala "Magic Ink").

His talk "Media for thinking the unthinkable" goes into detail on this, especially from minute 10 to minute 17: https://youtu.be/oUaOucZRlmE?t=600

The essay Magic Ink: INFORMATION SOFTWARE AND THE GRAPHICAL INTERFACE also goes into detail about this topic http://worrydream.com/MagicInk/

on one level, yes

on another, they're down right scary and overwhelming, or at least dumbfounding and perplexing

..thinking specifically about The Humane Representation of Thought https://www.youtube.com/watch?v=agOdP2Bmieg I had to stop it half way through.. :-/

see also Seeing Spaces https://www.youtube.com/watch?v=klTjiXjqHrQ and Media for Thinking the Unthinkable https://www.youtube.com/watch?v=oUaOucZRlmE

In principle, yes, but in practice, no. Tools let us think previously unthinkable thoughts. See Bret Victor's Media for Thinking the Unthinkable [1], where he quotes al-Khwarizmi from the ninth century:

What is the square which when taken with ten of its roots will give a sum of thirty nine? Now the roots in the problem before us are ten. Therefore take five, which multiplied by itself gives twenty five, and amount you add to thirty nine to give sixty four. Having taken the square root of this which is eight, subtract from this half the roots, five leaving three. The number three represents one root of this square, which itself, of course, is nine. Nine therefore gives the square.

He's solving x^2+10x = 39 by completing the square.

[1] https://youtu.be/oUaOucZRlmE

⬐ Retra

You don't have to use those words. For instance, for

n >> k >> ln(n) >> 1,

he says "Specifically, we require n much greater than k much greater than log n much greater than one", and says it's unwieldy. Well, if you say it like that, then it is.

"Specifically, we require than the magnitudes of n, k, log n, and one be in decreasing order with significant distance between each."

Not a problem. Just don't try to read notation literally in the order it's written.

This is really significant. At this moment in history, the growth of computer power has made a bunch of important signal-processing and statistical tasks just feasible, so we are seeing things like self-driving cars, superhuman image recognition, and so on. But it's been very difficult to take advantage of the available computational power, because it's in the form of GPUs and clusters. TensorFlow is a library designed to make it easy to do exactly these things, and to scale them with your available computing power, along with libraries of the latest tricks in neural networks, machine learning (which is pretty close to "statistics").

As a bonus, it has built-in automatic differentiation so that you can run gradient descent on any algorithm — which means that you can just write a program to evaluate the goodness of a solution and efficiently iterate it to a local maximum. If you do this enough times, hopefully you'll find a global maximum. There are a variety of other numerical optimization algorithms, but gradient descent is very simple and broadly applicable.

And it runs in IPython (now Jupyter), which is a really amazingly powerful way to do exploratory software development. If you haven't tried Jupyter/IPython, google up a screencast and take a look.

I'm just repeating the stuff that's on the home page in a different form, but this is a really big deal. Most of the most significant software of the next five or ten years is going to be built in TensorFlow or something like it. (Maybe Dan Amelang's Nile; see Bret Victor's amazing video on the dataflow visualization he and Dan built for Nile recently at https://youtu.be/oUaOucZRlmE?t=24m53s, or try the live demo at http://tinlizzie.org/dbjr/high_contrast.html, or download and build the Nile system from https://github.com/damelang/nile.)

Apparently the Googlers think that too, because among the 39 authors of the white paper are several shining stars of systems research, including some who may be up for a Turing Award in the next decade or two.

⬐ elcritch

Agreed, this is huge. I've been thinking along these lines for a while, but more focused on traditional programming systems. The approach of programming via data-flows and visually representing those is going to be enormous in the coming decade, IMHO. Especially in asynchronous and heterogenous programming environments.

Actually, it feels like I've been "scooped" somewhat. However, at the same time it's awesome to find that many of the core assumptions I've been working in formulating for traditional programming systems are actually already being used by other top engineers/researchers at Google. Thanks for posting about the whos-who of systems research. I'll have to check up on their research more!

I think there is a lot of room left to apply this approach to a lot of existing systems outside of machine learning. We're working on releasing a prototype in the coming months at our new startup (vexpr.com) based on applying these same data-flow principles to normal programming projects. Actually this might give a huge boost in presenting the underlying approach to systems engineering to audiences.

⬐ w_t_payne

Yeah ... I'm a big fan of data-flow programming (Been working on my own approach for a little while too ...)

⬐ elcritch

Nice, lol, hope TensorFlow didn't scoop any of your ideas too. ;) Overall, it's exciting to see data-flow paradigms gain popularity.

What kind of dataflow approach are you most interested in? I've been shifting my approach to work within existing systems. One things I've been doing is abusing the intent of web-components and DOM events to create data visualization tools.

Recently my team and I decided to focus down a little more. Have you had success in bridging the gap between esoteric academic papers and industry?

⬐ w_t_payne

Nothing too serious ... mostly to glue embedded machine vision system components together in a simulation environment.

⬐ hmehta

Do you guys know of any theoretical analysis on Data flow vs Control flow software ?

⬐ kragen

This comment is not being answered, not because there is none, but because there is far too much to fit a good summary of it into this margin. It includes most of modern compiler and CPU design, as well as most current HPC work.

⬐ elcritch

That's a fair point overall. It's basically the entirety of the debate between functional languages and imperative languages. Nonetheless, it is still pretty good question as far as someone trying to understand the differences and why it might be useful.

@hmehta, one of my recent favorite videos generally on this topic has been from Curry On conference [1]. It's not strictly data-flow vs control flow, but it does discuss some of the more practical differences in between using monads for structuring data flows vs imperative control.

My take is that monads can be roughly considered as wrapping up imperative control flow along with the data. E.g. wrapping up the if/else choices into a data structure. Currently I actually found studying up on actual category theory easier than all the "easy introduction" things regarding monads. As in, the ideas when presented straightforwardly and technically correct are simpler than we give credence to.

[1]: https://www.youtube.com/watch?v=449j7oKQVkc

⬐ cafebeen

I wouldn't worry too much about being scooped by this--the idea of data flow programming has been around since the 60s. Tensorflow looks like a very nice implementation of those ideas for neural network training, and is potentially useful for more general numerical computing, but to what extent remains to be seen.

⬐ Fede_V

One small note on the paper itself. I love that the paper's author are in alphabetical orders, without any stupid jostling over first/last authorship.

I think that's fantastic and wish academia went that way too (yeah right).

⬐ sytelus

Why is alphabetical order great? It just only means that people with last names starting with z will get penalized every single time because of choosing wrong parents! I would think randomized order would be chosen - especially if you were mathematician.

⬐ infinii

"Choose" parents with a surname starting with A then.

⬐ Swizec

You can also legally change your last name to anything you want at any point in time. If you care enough.

Or use a pseudonym.

⬐ ant6n

Meet Mr. Aaberg.

⬐ finance-geek

Mr. Aaberg has nuthin' on Mr. Aaaron

⬐ mcnawoh

Interestingly, with this sounding like a somewhat realistic Norwegian/Danish name, it should actually show up at the bottom by the sorting rules of those two languages.

The digraph "aa" is an archaic representation of the modern character "å", the last letter of the Norwegian and Danish alphabets, and should be sorted accordingly.

⬐ Steuard

Why "penalized"? If there's no significance attached to author ordering, why should it matter who's listed first?

(I can actually answer my own question to some degree: the "alphabetical authors" convention is also the norm in my own field of high energy physics. Only after I finally landed a tenure-track job and served on another search committee did I realize that folks from other fields of physics would see my lack of first-author publications as a black mark. But in principle, among those who know the convention, order ought to be unimportant, shouldn't it?)

⬐ danieltillett

If only it were so simple. It really depends on the field, but in the biological sciences it really matters where you are in the authorship list. If you are first it is assumed you did most of the work and if you are last you were the head honcho of the group. When you apply for scholarships and grants the committees will only consider the first and last authors as important and everyone else is just filler. It really important to be last when you are trying to get grants as so much emphasis is placed on track record.

⬐ None

None

⬐ amelius

I think that is usual for any large project. Take for instance the papers published by CERN.

Also consider that a single "order" might not make any sense. How do you determine if somebody "contributed" more than anybody else?

⬐ revelation

I realize the jostling going on, but I personally appreciate that I can tell from the order of names who to contact with inquiries.

⬐ Someone

That isn't that simple, as there is a cultural component in the order authors are listed (https://en.m.wikipedia.org/wiki/Academic_authorship#Order_of...)

Also, at CERN, with papers with 1000+ authors, any chance of picking the 'right' name from the list approaches zero, anyways. Such papers typically list a "corresponding author" to whom to send inquiries.

Alphabetic order also gave us the fun Google search phrase "Does G. Aad exist?" (Gives a link to https://jeffollerton.wordpress.com/2014/05/28/does-gaad-exis..., but Google also suggests a correction to your "typo")

⬐ jamessb

This is the convention in mathematics.

From the AMS statement on The Culture of Research and Scholarship in Mathematics: "For this reason, mathematicians traditionally list authors on joint papers in alphabetical order. An analysis of journal articles with at least one U.S. based author shows that nearly half were jointly authored. Of these, more than 75% listed the authors in alphabetical order. In pure mathematics, nearly all joint papers (over 90%) list authors alphabetically."

http://www.ams.org/profession/leaders/culture/CultureStateme...

⬐ Zephyr314

Depending on how long it takes to "evaluate the goodness of a solution" techniques like multi-start gradient decent can rapidly become intractable though, especially in higher dimensions. There are a handful of open source libraries out there that try to tackle this time consuming and expensive black box optimization problem from a more Bayesian approach [1] [2]. There's also a YC company to do this as a service (full disclosure, I'm a co-founder) [3]. Combining these methods with TensorFlow could be very powerful and is one step closer to more automatic ML flows though.

[1]: https://github.com/Yelp/MOE (full disclosure, I co-wrote this)

[2]: https://github.com/hyperopt/hyperopt

[3]: https://sigopt.com/cases/machine_learning

⬐ orasis

Could something like MOE be made to run on TensorFlow?

⬐ Zephyr314

Yes, if you could expose the parameters to MOE from within TensorFlow. We're going to work on an example of showing how this can be done with SigOpt, which has a similar interface.

⬐ argonaut

You've just described Theano. I don't know why you think this is a watershed moment in history.

⬐ tequila_shot

> I didn't see the author list on the white paper.

http://download.tensorflow.org/paper/whitepaper2015.pdf

starts with the Author list.

⬐ tomatioaz

What is Nile?

I can't find it on Google.

⬐ nickpsecurity

It was also built with OMeta:

http://www.tinlizzie.org/ometa/

https://en.wikipedia.org/wiki/OMeta

That tool can be used to solve a lot more problems. REBOL/Red and Racket are also taking DSL approach.

⬐ azeirah

Nile is a language built by Dan Amelang, a researcher @ viewpoints research institute. It is a language made for graphics rendering, it has the goal of having no incidental complexity.

2D vector graphics rendering as an example, takes ~10.000 lines of code using traditional libraries like Cairo (used by Firefox) or Skia (used by Android and Chrome). In Nile, it's 200 lines of code.

Check out this repo, and the pdf inside the readme https://github.com/damelang/nile

⬐ StevenOtto

And check out the Nile Viewer as well! http://subtlegradient.github.io/nile/viz/NileViewer/demo_sha...

⬐ mtw

What about Torch or Theano which allow you to use multiple GPUs and clusters? they also have a wide array of libraries which allows you to extend capabilities (itorch etc.). torch is also very fast, most parts written in C so I don't know if Tensorflow would be really that fast compared to existing librairies.

One thing find I found interesting is the ability to use the software you designed in research directly in production (without having to rewrite the whole thing in java)

That said, I'm excited to see Google release this. Hopefully it would encourage Facebook or Baidu to also release

⬐ dchichkov

As a side note, it feels like the front page picture of interactive computational graph is a bit misleading - looks like the interface is still using Python (or C++), not fancy GUI... But yes, it looks very similar to a great tool from Yoshua Bengio's group - Theano. So, great! Another amazing tool to take away the grind of doing ML projects!

⬐ tianlins

TensorFlow seems to have better multi cpu/gpu support than Theano.

⬐ maccam912

As far as I can tell, this may not be faster than theano/torch, but it doesn't sound like it'll be slower. It takes a similar approach of sending all the big computation stuff out to efficient implementations. The "slow" language code is simply building up a graph of work to be done by the efficient implementations.

⬐ chimtim

Do 10% speedups really matter for its success, if both libraries are ultimately calling into CUDNN? I do not think so.

Right now, Torch/Caffe adoption is really good with lots of research code being released for it (even from within Google DeepMind). So it will be interesting to see how this catches up amongst the research and developer community.