HN Theater @HNTheaterMonth

I saw this talk, Stop Drawing Dead Fish by Brett Victor, a while back. While I don't think it's the tool you're looking for, he does a great job showing why we may want better tools than we have currently and what they might look like. https://vimeo.com/64895205/description

That's only true if you don't consider the context that the code lives in. For example coloration in editors could be considered a visualization of additional dimensions of info (syntax, types) and autocompletes are like a branching "time-like" dimension. Besides that code mostly tries to solve problems that can be represented and show visually.

The need exists, how to turn that into a solution with a useable interface that I'd use for multiple hours a day is beyond me though.

For me Bret Victor's talks point to this problem where we treat code as something we only interact with as text but it can be so much more: https://vimeo.com/36579366 https://vimeo.com/64895205

Agreed, but I think the real problem is that the medium for communicating science is still basically a static PDF. Even if you use fancy microscopy to capture some of this roiling biology, the best you can do is link to a video file in the supplementary section of your paper, which few people will visit. Or to summarize the information in a bar graph.

This is so disappointing in this day and age where everyone at least has a computer, if not an iPad, capable of displaying video and animations in line with text.

And it would be great if BioRender would apply the principles laid out in "Stop Drawing Dead Fish" by Bret Victor: https://vimeo.com/64895205

Bret Victor talks are inspirational:

- Inventing on Principal https://vimeo.com/36579366

- Stop Drawing Dead Fish https://vimeo.com/64895205

⬐ unixhero

Fully agree. You forgot one!

Brev Victor - The Future Of Programming https://vimeo.com/71278954

⬐ mitchtbaum

on one level, yes

on another, they're down right scary and overwhelming, or at least dumbfounding and perplexing

..thinking specifically about The Humane Representation of Thought https://www.youtube.com/watch?v=agOdP2Bmieg I had to stop it half way through.. :-/

see also Seeing Spaces https://www.youtube.com/watch?v=klTjiXjqHrQ and Media for Thinking the Unthinkable https://www.youtube.com/watch?v=oUaOucZRlmE

⬐ yesenadam

Yep, I thought Inventing on Principle was the best computer-related talk I've ever watched. Not easy to describe what it's about! Design, tools, art, UI, visualization, invention, life.

⬐ unixhero

It's like he has caught on to a kind of a zen thing. It is almost as if he isn't even there, while presenting about a technological topic at all. That is just the media for him to get an expanded wisdom across. What that message is is hard to identify. But there is a glance of it at the end of inventing on principle. It's a lot more profound than what the talk seems to be at the surface.

I really wish he would publish more frequently He has inspired me greatly.

⬐ yesenadam

Yeah, I was marvelling half the time at the wisdom, it makes other speakers seem merely knowedgeable, clever, brilliant etc. I missed most of it in my summary before. I love the

this is a cool way to live your life, according to some principle - this is mine; these are some other amazing people who had their own

aspect of it. His creators should be able to see and control what they're doing as they do it principle is awesome, and he's maybe uniquely well-equipped to do something about that. I wanna get the synth he designed!

⬐ jonnydubowsky

What's the synth?

⬐ FelipeCortez

Alesis Ion, Micron and Fusion

http://worrydream.com/Home2007/electronics.html

⬐ nikofeyn

what is curious to me is that that synth is famous for being extremely unfriendly to program. its usability is pretty bad and goes against many of his principles. i love bret victor's thinking and work since he became independent though.

I agree — as someone with as much of a design background as a programming background, seeing something like Bubble described as "visual programming" feels a bit offensive. Yes, there's a bit of meaningful visual structure sprinkled here and there throughout their environment. But it's nothing like what one would imagine if they were left to freely ideate about some hypothetical, ideal "visual programming" environment.

In a post-Bret Victor world, we know what such an environment might look like.

Stop Drawing Dead Fish: https://vimeo.com/64895205 Drawing Dynamic Visualizations: https://vimeo.com/66085662

As a visually-minded developer constantly hunting for such tools, and even working on some myself (in a small way — nothing that anyone here will ever end up using, surely), I can assure you that there's enough smoke that eventually we'll see something like what we desire catch fire.

If I had to guess, I'd say we'll see it come out of the world of video games, not web applications. So don't fret that Bubble looks like a step in the wrong direction. They're not building tools for visual thinkers, they're building tools for MBAs.

> The problem is that in order to play with the mockups and test them, you could create interactions that work with the expected happy path, but are not what the final application logic is supposed to be.

> With a tool like this, you could easily create an interaction that updates the UI immediately. It would look correct, but it would be wrong.

This also happens if you build the component in code, no? I fail to see how this is specific to a visual tool. A professional should take care of all cases, regarding of the medium of the tool.

> That's fine, but I don't count that as an interaction.

That was just the simplest example I came up with in ten seconds. For more complex examples of what interactions you can build purely with a visual tool that then spits code, see Apparatus [1] or Stop drawing dead fish. [2]

I can imagine building components easily in an environment like that. We already could do that in the Flash/ActionScript environment in a limited fashion, but the results were very limited (state could only be represented as a video timeline), and the platform itself was terrible.

[1] http://aprt.us/ [2] https://vimeo.com/64895205

⬐ realharo

>This also happens if you build the component in code, no? I fail to see how this is specific to a visual tool. A professional should take care of all cases, regarding of the medium of the tool.

If you want to do it correctly in the tool and tie the UI update to the "permission granted" response, that event (or part of a store, or however it's implemented) already has to be defined in code (even if it's just a stub) by the time you're making the UI.

So again, this breaks the isolation because the component in the tool now has to know about this.

In contrast, if you're writing the component in code, and find something missing, you can just go "oh, guess I'll just add that to the appropriate place".

I can imagine the tool exporting some kind of interface with a list of things it needs (project-wide, across different components), that the developers would then implement, but with that we're getting pretty far from existing real-world React patterns. It's also starting to approach the difficulty of just learning to code in the first place.

⬐ TuringTest

> That event (or part of a store, or however it's implemented) already has to be defined in code (even if it's just a stub) by the time you're making the UI.

That's a good point. A visual tool for building components will need a testing feature that can keep all those stubs. But isn't that just like Test-Driven Development?

> It's also starting to approach the difficulty of just learning to code in the first place.

Not at all.

In any design, you'll have to think about all the cases thoroughly and systematically; and any rational people can do that. But coding in a general purpose language has the additional requirement of having a mental model of the execution platform and conveying precise instructions; it's not enough to think in terms of the problem being solved. That makes all the difference to people without coding experience.

The syntax of programming languages is particularly tricky and difficult to learn. The people working with Chris Granger, the designer of Light Table, found that users of visual development tools find it incredibly difficult to understand scope. And coding in any modern general PL is all about scope.

You may think of this kind of visual environment as a particular Domain Specific Language. The users are coding in it, but the programming language is not general purpose, which makes it not "coding" as developers understand it.

> I can imagine the tool exporting some kind of interface with a list of things it needs (project-wide, across different components), that the developers would then implement

BTW, the mockup tools I'm talking about already offer that kind of interfaces to define the list of coding requirements generated by the user interface prototype. They cough up that list of requirements as a Word document or web page, though. Wouldn't it be best to generate those as placeholders in code?

I think one thing that made this happen is that this is a presentation made to exercise an interactive software tool that—although it can also be used for one's own experimentation—was mainly written to be used in the presentation. It's a case of making a custom "musical instrument" crafted to play exactly the song you're about to play on it. A lot like Bret Victor's "puppet controller" app in his presentation Stop Drawing Dead Fish (https://vimeo.com/64895205); or the actually-an-instrument "MIDI fighter" created by Shawn Wasabi for his music (https://www.youtube.com/watch?v=qAeybdD5UoQ, https://www.youtube.com/watch?v=Cj2eUpt3MVE).

I see no benefits over TV where best angle is already selected for you (and framing is art in itself), here you have to play the cameraman. How would multiple people (a family) watch this kind of media without getting in each others' way?

And most importantly, they didn't take advantage of biggest feature of platform - interactivity. They are just drawing a dead fish [https://vimeo.com/64895205].

I've had this on my mind in various forms for almost a decade. It's only recently been crystallized from something abstract into much more concrete thanks largely in part to Bret Victor's talks. If you haven't already seen them, I highly recommend "Stop Drawing Dead Fish"[0] and "The Future of Programming"[1] as starting videos. Then just watch all the rest of his talks. :)

[0] https://vimeo.com/64895205

[1] https://vimeo.com/71278954

I'm using "dead fish" in the Bret Victor sense.[0] It's a static tool by nature. By the time you're in a live environment, webpack is gone.

[0] https://vimeo.com/64895205

⬐ jamescostian

Oh. Isn't prepack also a "dead fish" then? Like you said, "It's a static tool by nature. By the time you're in a live environment, [prepack] is gone." There isn't any sort of prepack runtime

⬐ gavinpc

I was assuming that the intention was to eventually provide runtime support. If that's correct, then I think we'd be in agreement about what "static" means?

Babel and all sorts of other compilers can run in the browser. Whereas webpack is mainly concerned with the text that you transmit and could not "by nature" be made to do its job in the browser (because then it would be too late). That's the difference I had in mind.

They're pretty good for carrying around a bunch of custom "controller" interfaces (in the sense of a MIDI "controller") without needing to pack around hundreds of pounds of hardware covered in knobs and keys. DJs like them.

Also see Bret Victor's talk (https://vimeo.com/64895205) where he does digital puppetry using two tablets: one using direct motion input, the other with a knobs-and-sliders UI for a set of behavior controls.

this reminds me of Bret Victor's talk, "stop drawing dead fish", which takes a similar concept of live-performance programming and applies it to the world of animated storytelling:

https://vimeo.com/64895205

Actually more like Stop Drawing Dead Fish by Bret Victor: https://vimeo.com/64895205

He is my hero actually :)

Actually, that presents the code in a visual way while this allows you to work on the thing itself. Similar to Stop Drawing Dead Fish by Bret Victor: https://vimeo.com/64895205

> There is incredible motion design work that is purely accomplished in code. Certain things are faster and far more flexible in that medium.

There are, but not that much. Look at the code of this demo. It's not obvious, it was probably hard to tweak to look nice, not to mention doing any debugging.

> There are plenty of effects I've created with code that would be an absolute nightmare to try to create with a GUI.

That's kind of my point. It isn't that code is better-suited to this job - it's that our GUI tools are mostly stuck in the static world and suck for dynamic / interactive visualizations.

Bret Victor put a lot of thought on how to improve this state of affairs; he also brings convincing arguments that such things are still better-suited for direct, visual manipulation - that coding directly is a crutch we use because of lack of better tools.

Some good talks and essays on this:

http://worrydream.com/MediaForThinkingTheUnthinkable/

https://vimeo.com/66085662

http://worrydream.com/DrawingDynamicVisualizationsTalkAddend...

https://vimeo.com/64895205

⬐ ekzy

Stop Drawing Dead Fish Bret Victor http://worrydream.com/

I encourage watching the entire video, but I linked to the last part which may encourage you to do so

> How will you work with authors to make these books? I would wager that most authors don't know HMTL5/CSS3.

Great point!

So right now there's a great divide between the world of authors and let's say front-end developers. But there is some intersection too and that is where we focus currently. We work closely with new writers (I personally sit with them or collaborate with them over Github) and try to yield their thoughts into telling creatives -- it's a bit different for everyone, depending on the nature of book/individual etc.

In the medium term we intend to provide this as a process with which developers and designers can work alongside ordinary writers to help them create bestsellers. As if the book were an app!

Part of our strategy is also to provide open source tools (See bookiza[1]) and predesigned templates so that writers can start off quickly and not have to worry about HTML/CSS, or the underlying "tech". They can focus on writing their story instead and leave the "tech people" handle their tech. Obviously, different people have different skills and preferences so we'll see if this evolves into a 'book writing framework' sorts or just a, say, predesigned editing/word processing tool or a set of hard rules that people may or may not always follow.

> What would be the incentive for an author to work with you vs. a more traditional publisher?

There are quite a few big ones:

1. Number one: The ability to create a winner.

Imagine a horror story with a webgl interactive showing blood spilled on a glass table? Eww. Heh, I just made it up but why not! :=) You can add visual explanations[3] to the context, like showing a physics experiment to explain how a pendulum works or showing how projectile motion can become an in-orbit flight around a planet with acceleration etc. -- in your book. It will add life to the context of your story.

This simply isn't possible on traditional platforms that are based on artefacts that sit outside of web. In other words, with Bubblin you'll be able to do books that are otherwise impossible to do elsewhere.

2. Second big advantage: 100% Support!

Our books work everywhere -- all devices, all OSes, all viewports[3]. They are responsive and scale well. In theory, your book can reach everyone on the web and not just those who are behind similar but closed ecosystems today.

3. Third, it takes about half the time to get the book out. Guaranteed! I encourage you to try Bookiza and see for yourself.

Why so? Because HTML and CSS!

Compared to writing with an enterprise-y tool like MS Word and then having a bunch of people (editors/friends) back-and-forth it and then converting it to N different ebook formats this one rakes in much less complexity and saves time.

We tap into all the advantages of our existing developer toolchain - using git for versioning, ability to collaborate with co-authors on Github, using markdown or haml or just plain text, utilizing rapid manuscript deployments for editions etc. etc. etc. So quite a few advantages in there!

4. Fourth, the readers don't have to wait for a book to download or worry about disk space. Again this is convenience & simplicity of web. There is no botheration of managing files or their compatibility, format or other issues. Portability is simply in-built. I mean, it just works!

> How big is this market compared to mostly-text ebooks and print (picture) books?

You can always do a text-only book or an images-only magazine on Bubblin. Whatever makes your readers happy!

ebooks market size and opportunity is pretty big even if we subsection it into just interesting set of interactive books. I'll be happy to run by some numbers if you want.

Feel free to reach me at [email protected] :-)

[1] http://bookiza.io

[2] https://vimeo.com/64895205

[3] https://bubbl.in/support

Watch Bret Victor's videos about Learnable Programming[1], in particular Stop Drawing Dead Fish. [2] Only because visual tools in the past were limited to putting properties in boxes doesn't mean that all visual tools have to be that way.

[1] http://worrydream.com/LearnableProgramming/

[2] https://vimeo.com/64895205

⬐ hacker_9

I have seen it, and while Bret Victors 'research' is interesting, none of it actually scales to a practical level.

⬐ TuringTest

Because it is not engineered to industrial levels. It lacks encapsulation and packaging, but those can be added.

Have you seen the tools by Chris Granger (Light Table, Eve)? Although they are not as spectacular as Victor's, those show a lot of promise to become practical tools.

⬐ hacker_9

Bret Victor's ideas lack a lot more than that. Complex systems contain hundreds of sub systems which in turn can contain hundreds of algorithms, which then contain many many variables. All interacting in some way. His ideas simply don't scale to this level (he works with < 10 variables, enough for a nicely packaged demo).

Chris Granger himself stopped working on LightTable because he didn't believe in the project, and if you use it, the live editing gives minimal gains in productivity. As well as being buggy, slow, distracting and potential dangerous when blindly running half complete IO operations.

As for Eve, well.. his first release was a visual programming tool that inflated 3 line sql statements to take up a whole page with, you guessed it, boxes and arrows. Now he is working on a 'card wiki' idea which effectively tries to hide all the program logic behind pictures and hyperlinks. Neither are good ideas, or once again, scale in any meaningful way.

⬐ TuringTest

Why do you think that doesn't scale? Hiding information is the basis for progressive disclosure, which is essential to handling the limitations of short-term human memory. Wikipedia is card-based,* and it is one of the five largest sites in the web. See Cognitive Dimensions[1] for ways to make visual systems scale.

As soon as you add encapsulation and abstraction, the system can scale; it's no different than hiding code logic behind function declarations and object schemas. If you add easy composition, like modern functional languages have, such systems could be more scalable than traditional imperative languages; code outliners like Leo follow the same structural principle, and they're great for programming.

[1] https://en.wikipedia.org/wiki/Cognitive_dimensions_of_notati...

* Ok, Wikipedia contains articles and not code, but my point is that the structure can handle lots of hierarchical content in multiple categories.

⬐ EdiX

> Why do you think that doesn't scale?

Because it doesn't. Watch Bret Victor's last talk about Eve. All those ideas fail when you give them to novices and try to do anything more complex than describing a few static relationships.

Maybe there's a way, but I'm skepitcal.

⬐ TuringTest

Yet end users are able to build fairly complex data structures and conditional processing using spreadsheets, so we know for sure that it can be done.

Of course experimental tools will be severely limited compared with industrial IDE systems that have been polished literally for decades; it will take years before we get these new tools into shape and learn what their best use cases are. But I'm not skeptical, because I've seen it happen before, and the conditions for it are now better than ever.

⬐ EdiX

Spreadsheets IMHO are a solution space that hasn't been sufficiently explored. However they are not particularly visual (especially if you try to do complex things with them), they are very much domain specific they are extremely computationally inefficient, they're very hard to version and they become intractably complex much quicker than equivalent imperative or functional programs.

I've thought about what could be done to improve on their weaknesses (like strong typing, bridging the gap to databases, better table naming), even wrote some code in this direction, but anything I can think of involves either making them less visual, less user friendly or both and doesn't even begin to address the lack of generality

Finally I have to point out that some spreadsheets have been around for longer than many programming languages in current use so they don't actually lack maturity, yet they have seen little improvement in over a decade.

⬐ hacker_9

Wikipedia is also written in English, and made for human consumption. English is infinitely more scalable than any programming language, and the brain is far more capable of understanding than a computer. So comparing the two is unfair to say the least.

Additionally simply adding 'encapsulation and abstraction' does not equal scale. Live programming for instance requires context and re-running of code on every edit (and we could be talking millions of lines of code that need to execute), re-reading of data files, re-running of prior user input and so on. There's a reason it is only implemented for small systems (read, useless practically).

⬐ TuringTest

> English is infinitely more scalable than any programming language, and the brain is far more capable of understanding than a computer. So comparing the two is unfair to say the least.

IMHO programming a large system shouldn't be all that different from writing an encyclopedia, or rather a math textbook; the same combination of informal explanation and formal demonstrations could be the shape that "visual" programs could take. Literate programming is written in English, and it provides much of the same advantages of the encyclopedia or technical book. Systems like IPython / Jupyter work in that direction, and they belong in the family of visual programming (although they use "classic" languages for the formal part, nothing stops them from also including graphical models as part of the code specification).

Programming languages should be made for human consumption, and one of the promises of visual languages is that they use secondary notation to convey the meaning of the program, so that parts that are not visible to the compiler are still relevant.

I believe that programming environments should converge towards a hybrid of programming, using the best representation for each concept - like text (to represent abstract concepts), diagrams (to represent more physical properties - time, data flows, and dependencies between modules), and raw text for the human-readable comments that explain how everything is tied together. If you build a programming environment like that, my comparison is not unfair at all.

>Live programming for instance requires context and re-running of code on every edit (and we could be talking millions of lines of code that need to execute), re-reading of data files, re-running of prior user input and so on. There's a reason it is only implemented for small systems (read, useless practically).

Modern IDEs are capable of doing a lot of that when in debug mode, I see no reason why having a language with a graphical syntax should make any difference - it just places a little more weight on the parser and graphic card.

⬐ the_af

> Programming languages should be made for human consumption

The problem here is the "impedance mistmach" between humans and computers. Wikipedia is not a good example: it is exclusively for humans. If there is a "program" for the computer, it's minimal: "search this", "when I click here, take me to this page" (you'll notice everything more complex than that, even in Wikipedia, requires its own mini-language). This is the kind of trivial systems which are easily encoded by "visual" programming.

The problem is that real computer systems need to be understood by the computer as well as by humans. In order for something to be understood by a computer, it must be specified in formal terms; English -- natural language, actually -- is terrible for this. Once you're working in a complex system with a formal language, the limitations of these "visual" tools become evident.

⬐ TuringTest

> Wikipedia is not a good example: it is exclusively for humans.

Not exclusively. Bots and Wikidata have shown themselves pretty capable of doing complex automated tasks on top of the structured data contained in the encyclopedia. And the wiki platform plus templates is a very general and flexible data structure that makes it easy for humans and software to collaborate.

> In order for something to be understood by a computer, it must be specified in formal terms

This doesn't need to be that way, at least not for 100% of the system. End-user development tools like spreadsheets and Programming By Example can be used to ease the impedance mismatch without requiring the human to specify by hand every single step in a formal language, so that partially ambiguous inputs can be used to specify valid programs.

Only the task of building system software should require the full technical approach; most business applications can benefit from looser approaches - in fact being less strict can be a benefit in the requirements specification phase, just like User-centered design has shown us.

Combine that with logic solvers and deep learning, and in the future we could get some systems that can use very high-level commands to build fairly complex automated behaviors.

⬐ the_af

You have a point about wiki bots, I hadn't thought of that. But isn't the "Wiki platform plus templates" an example of a textual programming language with a precise syntax and rules?

As for the rest, I think it's a pipe dream except for the most basic, "build this out of predetermined blocks" apps.

⬐ TuringTest

Does the Visual Editor in Wikipedia count as a visual tool? :-P

My point is that the structure of a wiki, where each unit of content is a separate building block, can be used to build a visual system in the same way that you can build a textual one; the basic components of storage and APIs are the same, only the thin syntax layer is different. You are no longer dependent on the single stream of raw text following a strict syntax, which is the basis from which compilers generate binary code in traditional languages; you'd be able to build programs by arranging code and data in a network within a semi-structured platform, which is one of the characteristics of visual tools.

You're right that stuff like this has been a pipe dream for at least half a century, since the Mother of All Demos; but in the very recent years I've seen a lot of the required pieces falling into place, so I believe it to be possible.

If we apply the same amount of engineering work and refinements that went form Engelbart's demo to modern desktop systems, we could have a functioning system with this paradigm; the basis are in place for it, and many of us have the vision of the benefits it shall provide.

Of course, part of building a program may require describing some components using a textual representation, where it makes the most sense; but with a wiki-like "outliner" platform, parts of the working system could be programmed using visual code, and be integrated with those other components programmed in traditional languages. Heck, GUI builders and visual tools for database schemas already work that way; the only thing they lack is the capability to be expanded with new components, which this new family of visual tools have.

I once saw block languages described as "What I could have typed out in twenty characters takes me 100 meters of dragging and dropping instead." Pretty much sums up the one major issue with them, doesn't it?

Having said that, I'm convinced there's a way to design an interface around it that creates an acceptable compromise with (most of) the best of both worlds, although I don't know what that interface would look like. Perhaps introducing a cursor and treating the blocks as we would normally treat glyphs in typed languages[0], using keyboard shortcuts to quickly insert and manipulate blocks.

EDIT: Also, what's notable about all these block lanuages is that they're essentially taking writing, and turning the words into things that can be dragged around and manipulated as objects. Although we do to some degree process language in that way, I wonder if it isn't doing things backwards.

If you look at Bret Victor's recent work you can see interesting alternative takes on this where the problem is approached from a concrete example first and then generalised into abstractions[1][2][3].

[0] Err... not as in typed languages, obviously, but as in languages that we type out.

[1] https://vimeo.com/66085662

[2] https://vimeo.com/64895205

[3] https://worrydream.com/DynamicPicturesMotivation/

⬐ TheLoneWolfling

Some IDEs end up being effectively doing this.

⬐ vanderZwan

Right, but in that case the IDE evolved around the existing character-based programming language. What if we try it the other way around? Start by designing an elegant block language and then create a keyboard interface around it.

⬐ TheLoneWolfling

Did it? Look at tokenized FORTH, for instance. Now, admittedly there's not a whole lot of syntax there to begin with, but still.

People tend to shy away from languages that don't have textual representations, and for good reasons. It's harder to share code, among other things.

I think the best approach would be to come up with something that is human-readable and writable, but is optimized for the block approach.

And I'd argue that some languages (lisp?) already fit this.

⬐ TeMPOraL

> And I'd argue that some languages (lisp?) already fit this.

Yup. And with proper editing help like Paredit, you start to actually think in terms of moving tree nodes ("blocks") around instead of typing in text. With that, all you'd have to do is to start drawing colored boxes around the code instead of parens, and you'd have a decent, keyboard-operated "block language".

⬐ sukilot

Right, what you want is a seamless view that shows the block diagram while you type, perhaps by "dissolving" a single block while you type in a way that transiently corrupts the block, and condenses the block when the incremental compile notices that there is once again a valid block.

⬐ TheLoneWolfling

Some editors already do this, by, for example, highlighting pairs of brackets.

Seeing how each visualization adjusts as I change the original dataset is so useful. The technique reminds me of Bret Victor's amazing work.

Ladder of Abstraction Essay: http://worrydream.com/#!2/LadderOfAbstraction

Stop Drawing Dead Fish Video: https://vimeo.com/64895205

This is awesome, thanks for sharing!

Generative Art will take off when it doesn't require expressing the code through conventional programming language syntax. A visual, interactive programming environment like those advocated by Bret Victor [1] would be ideal for artists without a technical background.

NodeBox, mentioned below, have some characteristics of this, although it doesn't seem to allow the artist to create new abstractions without resorting back to Python.

[1] in Stop Drawing Dead Fish. http://vimeo.com/64895205

⬐ delinka

Watching Mr. Victor, he tells us about how we're doing old media with new tools (painting, film.) Then he moves on to ... a puppet show using his new tools. I feel like he started with what was going to be an intriguing thesis ("make something new, not just old media with new tools") and changed to building those new tools.

That criticism aside, he's absolutely correct about needing new tools to prevent the creator from needing to learn a programming language. I'd suggest that this is the case in many fields. I'd go so far as to suggest that some day most of us software creators might be able to create software with little to no code, just dragging together concepts. (I'm aware that such tools have existed and do exist. One could argue that they haven't quite nailed it and thus are not very widespread in use.)

⬐ TuringTest

(I'm aware that such tools have existed and do exist. One could argue that they haven't quite nailed it and thus are not very widespread in use.)

I believe that day may be closer than you expect. I've seen all, and I mean all the components required to build such a practical programming system. They just were scattered around systems for widely different purposes, or buried in decades-old research projects that were too experimental and heavyweight for the machines available back then. (Victor's is certainly the closer to that vision, although it still misses several key components to make it practical and widespread; a close second is the Wolfram Language).

I have in mind a pet project to combine those partial tools in a way that I think would make a practical mostly-general-purpose programming language. Having some background in human-computer interaction, I think I'd be able to identify the most obvious roadblocks.

Unfortunately, I'm afraid my limited skills and time wouldn't allow my to expand the system beyond a simple proof of concept. If you're interested, I think I could compile a small recollection of my ruminations and put it in a presentable form, if only to inspire others and get some feedback on my approach. I don't know though if that would make for a good Hacker News submission, does this site accept links to personal blogs discussing pet projects?

⬐ mej10

I am interested in your article.

And yes, HN absolutely accepts links to personal blogs discussing pet projects.

⬐ TuringTest

Ok, I'll try to put something in writing. Yesterday's link to the Leo editor has inspired me to organize an idea-dump of all my dispersed notes. :-)

⬐ tomek_zemla

I don't think that generative art needs to stop requiring programming skills to be successful. The huge community of www.processing.org artists who code suggests this requirement is not a problem. The whole point of generative are is to use programming language as the tool of visual expression as opposed for example to using UI based tools like Photoshop or Illustrator.

Any art (visual or not) requires creators to use various tools and learning these tools takes a lot of time and afford. And mastering them often takes insane amounts of study and practice. Think of work required to learn to sketch human figure well or chisel it out of wood. Think of years of training required to play a violin concerto. Or to dance a ballet piece. Programming is just a new/differnt tool required for creating generative art.

I would even go as far as stating the opposite. Attempting creation of generative art via interfaces other then conventional programming languages is what would limit it! This is for the same reason that professional programmers haven't adopted (not yet at least) visual programming languages - they are too limited in their expressiveness.

The whole interest in generative art and design is driven by the fact that you can create things that you cannot using Adobe (etc...) tools. All the Photoshop and other image creation and editing applications are very powerful, but still limited by the imagination of the people who made them. Creating images using code lets you escape these limitations. This is what makes it an interesting new visual arts discipline.

And if you study lives of many artists you realize that they can be pretty obsessed and determined to work hard to learn tools and techniques in the process of their practice, so I am not worried - there will be many who will become excellent programmers!

What stops fast adoption of this type of art is the educational institutions insistence on separating artists from engineers. If computer science departments offered and occasional art course and fine art departments intro to programming, the generative arts would possibly become one of the most popular art forms in the new century. It would also help more of multidisciplinary creative thinking on both sides. I certainly agree with the article that the possibilities are immense...

⬐ TuringTest

The main problem is that programming skill requires a very different skill set than the one usually required to build successful art (insight, a sense of composition and proportions, good taste in colors, all that right-brainy stuff). Although a person may have both, it's rare that someone will be proficient in all those.

People with good visual skills may devote intense learning to the tools of their work, but see how the tools you mention (chisels, violin, proprioception for dancing) don't require strong abstract skills like programming does. Some renaissance men were capable of dominating both forms, but science was way simpler then - and they still gravitated toward graphical representations of structure and theory.

If all generative tools are based on "linguistic" programming based on strong syntax requirements, not allowing a more "intuitive" way to build abstractions, it will remain a highly conceptual form of art. This is still valid art, but quite different from the most popular forms of artistry that people associate with the idea of "art".

⬐ jmclean

Respectfully disagree. I think it's easy for people who have by now built an intuitive grasp of code to appreciate how challenging tools like Processing can be for a beginner.

I have a design background and learned front end code (and eventually ruby on rails) by getting progressively more interested in how the things I was designing were built. I feel like I achieved a pretty respectable level of knowledge in what I had been exposed to.

And yet, many aspects of Processing were a huge leap for me. I spent days looking at code examples just to try and grok how 3D works in processing. I get it now (mostly), but it still feels unintuitive. I can certainly imagine a language that would describe 3D behavior, and a whole host of other things, in a way that a visual artist would describe them.

I definitely believe that there is value in artists learning to think with the rigor that code encourages. It's a fascinating cross-pollination. Creativity often springs from encountering the limits of a medium (and one's mastery of the medium).

But look at the excitement this week among people like me about Swift. It's not about whether it was possible before for me to learn Objective C and build a game. It was. But goddamnit, it's such a pain, and I would certainly understand that someone who is starting from zero in terms of CS knowledge would find it impossibly intimidating.

Design is remarkably accessible. If you can pick up a pen and paper, you can do it. Code is getting way closer to that, but let's not have collective Stockholm syndrome. Our tools are a long way away from where they could be.

Honestly, I wish I had the skills to write a language or build the tools like the ones I can imagine myself. But I have confidence that someone will. It's going to be an exciting time for art, and I think it's coming very soon.

⬐ TuringTest

Same here. Do you have a place were you have compiled your ideas for your imagined language? We could cross-pollinate with each other :-)

⬐ jmclean

I do! I have some sketches that I drew up a while back. It's sort of a loose collection of thoughts, though this is inspiring me to think about it with more rigor :)

I imagine it as a combination of an abstraction layer on top of processing and a light IDE. Something like a palette of shapes and objects similar to what's available in photoshop and illustrator. Instead of drawing shapes, though, it would create code snippets for the various objects.

Those objects would be available to modify in an event loop similar to Processing (or Arduino, for that matter). With a strong autocomplete for both object names and the available manipulations for those objects, I feel like you could learn just by playing around.

Live preview like Swift /light table etc would be really powerful. Out of the box support for common things like collision detection would be great too. Color pickers that spit out rgba values? Maybe. Basically a tool palette oriented around building code for visuals.

In terms of the language itself, it probably veers more into personal taste. I'd love to see names of things be less arcane /abbreviated. Dropping some of the brackets and parens would be nice for non programmers too. (Again, taste.) Ruby-like syntax for new objects instead of objective-C-like constructors would be more readable. Basically, the more it could just resemble an English description of what you want to happen, the better.

Processing isn't that far away from this description, but the combination of having to refer to the documentation constantly, no sensible defaults, almost limitless customizability, and having to actually compile the sketch to run means that the initial barrier is really high. It's very difficult to get into flow until you reach a high level of familiarity with the language.

I would love to chat about this further and share ideas with you or anyone who's interested!

⬐ tomek_zemla

You realize that to build such a tool you would really need to develop high level of programming skills, right? ;-)

⬐ TuringTest

It seems there's no way to send private messages here at Y Combinator. Send me a mean of contact to twaway00 at gmail if you wish to continue conversation.

⬐ tomek_zemla

It's a fascinating discussion. Maybe we can meet half way?

> I definitely believe that there is value in artists learning to think with the rigor that code encourages. It's a fascinating cross-pollination...

I agree!

> Design is remarkably accessible...

But this... Well it depends on the point of view. Design can be as intimidating for geeks as programming is for artists.

I have nothing against making programming tools more user or artist friendly. In a sense Photoshop is such a step - it allows many creators make sophisticated images without learning to draw and without learning to program. And certainly many many amazing works were created with it and similar tools.

The point I am trying to make here is that the visual/artistic power or possibilities of generative art and design are very much driven and dependent on the artist's fluency to program. There were many attempts to create programming tools [1] that let you avoid having to slog through typing the code, but seems like none of them would match the expressive possibilities of 'raw' coding and gain any wider adoption. In other words all of them have significant limitations and this would be a step back from the point of view generative artist.

And just to make it clear, I don't think that using any of these higher level tools is wrong or produces works of lesser value. It's just that these works are outside of unique possibilities of generative art driven by traditional coding.

[1] http://blog.interfacevision.com/design/design-visual-progarm...

⬐ TuringTest

Have you seen the "Stop drawing dead fish" and other Bret Victor talks? Those are not high level tools, they allow you to build generative animation and graphs from scratch.

Photoshop-like visual tools are indeed limited to pre-built concepts, but Victor has found a way to make coding possible without textual syntax; that's a powerful idea that may be the basis for a tool allowing artists to program without requiring the programmer's skill to keep the parse-tree-plus-AST in your head while building automations.

Take a look at the topic of End User Development and Programming by Example[1], it's a quite comprehensive research field dealing with ways that programming-like activities can be done without any traditional "raw coding".

[1] http://web.media.mit.edu/~lieber/Your-Wish/

The most important thing to keep in mind is this is for designers not coders -- I haven't looked at this deeply enough to know if it is good yet, but my guess is if you read hacker news comments it isn't for you because, well, you probably code=).

Yes, these sorts of editors often create horrible leaky abstractions. Yes, these editors often confuse product folks into thinking that coding isn't necessary or important or hard. But that isn't its goal -- the goal is to make the common stuff easy to do for people writing / designing simple site content -- fades, positional, simple triggers. There is no reason to write code for this stuff.

If you don't believe me, imagine drawing something without showing the picture. Word documents without wysiwyg. Photo editing without Photoshop. In this video (http://vimeo.com/64895205) Bret Victor brilliantly shows what a powerful interactive editor can do. For the web, no amount of manual css typing can beat what a great editor could do. Whether this is that editor remains to be seen, but the idea of editors is a good one.

⬐ kpapke

Actually I think Webflow can be a useful tool for if you're a developer who wants to learn how to design a prototype. I struggle with visualizing a new site without a mockup or style tile for reference. Webflow's UI is intuitive - everything you do has immediate visual feedback and directly translates into clean CSS/HTML. The previews of media queries affecting content is nice too. So it's very easy to reverse engineer how to make a good design, the same way you would reverse engineer some source code.

But with that said, I am not a paying customer. When I need to make a site then I just sign up for a free trial and export the code to continue the project. Their individual plan doesn't really make sense for me and I think it's too expensive.

Bret Victor's?

http://vimeo.com/64895205

The stupid Discus thing they use does not let me comment(What a terrible UI).

Bret Victor also has many other amazing and thought provoking talks:

Inventing on Principle: http://vimeo.com/36579366

Stop Drawing Dead Fish: http://vimeo.com/64895205

Drawing Dynamic Visualizations: http://vimeo.com/66085662

The Future of Programming: http://vimeo.com/71278954

Then, I'll have to finish my PhD and show the world how it's done. :-)

The essence of the card is that it contains a fixed, small amount of content that can be linked to and reused at different contexts, not that you 'always' have to interact with it in a fixed "3x4 inches" format. If you try out the Smallest Federated Wiki below, you can drag-and-drop cards within a page, but the page doesn't seem to built from cards (although it is). Pages can be created by composing cards and giving them lightweight styles (such as removing the visual frames around them, or showing them in list formats).

If it helps you, this is how I picture a system programmable by the end user through cards:

- The structure is roughly like in the 10/GUI concept demo:[1]

* Information is organized in non-overlapping plaques.

* Plaques can be shown folded (cards), unfolded (pages) or minimized (links).

* Plaques are hyper-linked; clicking on links open related plaques like in the Smallest Federated Wiki. [2]

- Users can create new cards (this is what's missing in current card-based systems, which are read-only):

* Creating a card's structure is like creating a dialog in a traditional visual GUI builder, by composing small elements in a larger layout.

* The card's behavior is programmed with the interactive comic-strip operations that Bret Victor demoed recently in the "Stop Drawing Dead Fish" demo. [3]

Does such system make sense? Can you picture it?

--

[1] http://www.gizmag.com/10gui-multi-touch-interface/13104/

[2] http://fed.wiki.org/view/welcome-visitors/hsi.fed.wiki.org/h...

[3] http://vimeo.com/64895205

⬐ taeric

The 10gui demo is somewhat neat, though it does harken to traditional wacom devices in my mind. The GUI part is the neatest, and I'm just not sure how I feel about moving actions which are almost strictly conceptual now, choosing applications, and moving them to a physical component. Well, at least more into a physical concept. (That is, right now picking an application to run is more conceptual. You state the application name you want to launch. In the physical analogy, you have to find the application you want to launch.) I fully grant that this is territory that most guis already breach. (I'm much more heavily into the command line than I would guess most are.)

But, back to the idea of a card containing a fixed amount of content. This seems no different than the semantic idea of RSS or even HTML. Items have dates, links, titles, and descriptions. "Unfolding" is simply following the link to the full item. Seen this way, I am not at all sure how what you are describing is much different than an RSS UI. (Indeed, if you drop the idea of moving and editing items, and move the orientation to vertical, this is the old google reader. Or, the current gnus. :) The only difference is "opening related plaques" becomes "launching relevant related application.")

I argue that twitter gained success because they effectively dropped the link/description part and set a hard limit on how large the title could be. Probably more importantly, they provide the hosting service for everyone.

More directly, twitter has seen success largely by limiting the interactions and capabilities available to users. This would leave the interface we are talking about largely more confusing than twitter, without being as powerful as just giving access to the raw data. More confusing for the less interested audience, and less capable for the more interested audience.

All of this is to say that I am not at all against you doing this. Indeed, I welcome the advances that may come. Sadly, I still can not picture it myself. Instead, I see the ashes and poor reflections of previous offerings.

⬐ TuringTest

Well, of course cards are semantically equal to RSS feeds or HTML tags - the card is a presentation concept for the idea of a "container", after all; programmatically, they can be treated the same as any other information block.

What changes is that they're now visible to the end user, while their structure previously could only be accessed by the system's developer. Now they can be published through an API, or directly manipulated from a GUI; that's a change in focus - the same operations can be done on the content, but different people can perform them if they are published as atomic cards instead of embedded in larger, whole documents.

Semantic-aware interfaces are less based on applications and more on tasks. These interfaces work just like with the old CLI, where instead of loading data into an app (where available commands are selected and composed by the developer), you have a information flowing through a "pipe" of independent commands selected by the user from all commands available in the system. Environments like KDE Plasma Active [1] (still app-based, but with "Activities" that represent tasks), or the old Cannon Cat and its spiritual succesor Archy (which was a graphical version of Emacs, sort of) [2], are examples of this model of interaction. These are not new concepts, really, but they've never been fully realized in graphical interface (afaik only the Canon Cat and Emacs follow them, and both are text-based).

Yes, Google Reader is a good example of the kind of content aggregation and sharing that I'm pursuing with my ideal system (and many people found it useful, so there's an audience for it). Google Now is another system which has chosen the card metaphor to show information according to context. But Reader was not programmable, as it had fixed styling that couldn't be changed by the end-user; and Google Now has fixed rules created by the provider, not the user. I believe that the card is a natural data structure for people that don't have a programming background to create simple procedures, that would allow users to create new behaviors for these tools.

You've given me some thoughts to chew with the "more confusing than twitter, less powerful than raw data"; I'm certainly aiming for "more powerful than twitter", and I'll have to ponder how to achieve that without making it too confusing. I have some ideas, but it's interesting that you identified this problem.

--

[1] http://www.linuxjournal.com/content/plasma-active-new-approa... [2] http://en.wikipedia.org/wiki/Archy

⬐ taeric

So, I think I was using the term "metaphor" where you are using "presentation concept." As such, I think we essentially agree here. I just don't have the vision to see how this graphical based concept will work. This sounds like efforts to move programming out of text based exercises and into visual ones. Or, modifying the AST of a program instead of the text. I clearly think this has some merit, but the largest success that I can think of for this idea is LISP, which fools everyone looking at it into thinking they are looking at text.

I definitely look forward to anything that is coming our way. Seriously best of luck on your endeavors here! I have already greatly appreciated the links you've given me.

⬐ TuringTest

Thanks! Yes, I see a trend of modern programming languages towards LISP features. Also visual environments are having a resurgence, now that computers are powerful enough to support "Intellisense"-like autodiscovery and suggestions without grinding the system. I expect future user interfaces to allow users to modify ASTs, and fool them into thinking that they're looking at blogs and wikis. :-)

Bret Victor, who has been inspired[1] by Doug Engelbart, has showcased plenty of innovative ways on design/user interaction on his presentations/writtings:

* Magic Ink[2]

* Stop Drawing Dead Fish[3]

* Drawing Dynamic Visualizations[4]

* Media for Thinking the Unthinkable[5]

* LearnableProgramming[6]

* Inventing on Principle[7]

[1] http://worrydream.com/Engelbart/

[2] http://worrydream.com/#!/MagicInk

[3] http://vimeo.com/64895205

[4] http://vimeo.com/66085662

[5] http://vimeo.com/67076984

[6] http://worrydream.com/LearnableProgramming/

[7] http://vimeo.com/36579366

Watch his presentations, they are incredibly inspiring and impressive.

https://vimeo.com/36579366 https://vimeo.com/64895205 https://vimeo.com/66085662

All are required viewing for the HN crowd.

Where's our version of "The Mother of all Demos" ?

Bret Victor, who has been inspired[1] by Doug Engelbart, has showcased plenty of innovations on his presentations/writtings:

* Magic Ink[2]

* Stop Drawing Dead Fish[3]

* Drawing Dynamic Visualizations[4]

* Media for Thinking the Unthinkable[5]

* LearnableProgramming[6]

* Inventing on Principle[7]

[1] http://worrydream.com/Engelbart/

[2] http://worrydream.com/#!/MagicInk

[3] http://vimeo.com/64895205

[4] Drawing Dynamic Visualizations

[5] http://vimeo.com/67076984

[6] http://worrydream.com/LearnableProgramming/

[7] http://vimeo.com/36579366

Awesome links, bookmarked thx!

I found out about Kyle Gabler as the score to the final animation composition (50:00) in Bret Victors fantastic "Stop Drawing Dead Fish" talk: https://vimeo.com/64895205

Could someone please explain to me why these tools are not commercially available?

This is the desktop version of the app Bret Victor created: http://vimeo.com/64895205

The tools would make my live so much easier, I can not start to express how much science needs this. Why did no startup jump on implementing them, like "Light Table" [1]? Alternatively, why did Victor not release it commercially? He would not get rich, but I guess he does not need to earn money anymore. His tools could benefit humanity, from cancer research to safer cars.

I really wonder, what the reason is why Mr. Victor never published any of his tools. It makes me sad to see such genius applications and all I can do is look at them.

[1] http://www.chris-granger.com/2012/04/12/light-table---a-new-...

EDIT: Thank you @williamcotton for watching the summary (I didn't) Mr. Victor mentioned he will make it available on github, his account is: https://github.com/worrydream This is great news.

⬐ grinich

There's a big difference going from a demoable app to production-ready software.

So much so, that it's exceptionally rare for a single individual to do both.

The worst managers don't see this, and mistake a well-made prototype for a finished app.

But as Elon Musk has said, "The first step is to establish that something is possible; then probability will occur."

:)

⬐ seltzered_

This. it's really hard to ship, it's even harder to ensure all the marketing/support/commerce stuff you really don't want to do as an interaction developer is also running.

Personal case examples:

1) Recently announced a plugin mentioned by some top designers - basically brings some new interaction ideas to graphic design tools. It's taken about 4-ish months of evenings to make a functioning prototype. Then another couple months to design the website/do user interviews/etc. Despite the flattering mentions, I still freak out a bit that not enough people will buy it when I launch.

2) Made a desktop app that had more of an 'interaction-first' approach, and inspired by folks behind Balsamiq. After 2 years of doing code, hiring a coworker, etc. It was 80% done, customers couldn't wait for me to ship, but some just-hired manager didn't get it and canned the project, along with my employment. Just downloaded what the old stuff they still ship, it has 2012 copyright info.

Also, it hasn't been encouraging when going to conferences like microconf, where the focus and advice leaned towards "find that b2b niche and make a saas app, you're gonna regret otherwise"

⬐ williamcotton

He says at the end of the video that the tool will be made available on Github sometime soon.

⬐ dkersten

I'm also interested in knowing why. A lot of his demonstrations are really cool and I'd pay a dollar or two just to play around with them as they are, so I imagine people would pay a lot if they could buy full featured packages.

⬐ bullfightonmars

Hrm... It looks like these talks are pretty recent. It seems like he is still exploring interaction and visualization and is creating some pretty fantastic work. That doesn't mean he is ready to or has something to sell.

⬐ lobotryas

You can always email Bret yourself and ask - his email contact is available at worrydream.com.

⬐ GuiA

HCI guy here, who gave a fair share of demos at conferences etc. to illustrate new concepts. Those demos are usually held together with duct tape, and would require TREMENDOUS work to turn into production ready apps.

Additionally, HCI researchers (group in which I'll include Bret Victor, even though he doesn't publish at the main conferences etc., sadly) are more interested in creating those examples to illustrate their research, but not in building commercial stuff (which is a very different type of work).

I actually was at Victor's talk "Stop Drawing Dead Fish", and at the end an audience member asked if he could release it. He said that if he released it, no one would get any use out of it, because it was really built only with that talk in mind. (which, of course, is not surprising– that's the nature of his work)

> ' He would not get rich, but I guess he does not need to earn money anymore.'

You're greatly overestimating things here, my friend :)

⬐ seanmcdirmid

I'm sure Bret would call himself a designer rather than an HCI researcher, and its true that many HCI researchers sneak design ideas into HCI conferences (Are we designers? Are we scientists? Let's just through in some pointless vigor in what is otherwise a good design paper!). Sorry, this is just one of my pet peeves, and its a problem we have in my own field (PL) also.

I agree most of our prototypes are held together by duct tape and do not represent ideas that are ready to go into production. The paper/idea is the artifact, not the prototype/demo, which exists to promote the idea. Ten years from now (more or less) the best ideas will be integrated into products by some entrepreneur who has the tenacity to make it all work for reals. E.g. we hope this happens with LightTable.

⬐ ColinWright

Older submission (by minutes): https://news.ycombinator.com/item?id=5634662

⬐ ColinWright

Barely older submission: https://news.ycombinator.com/item?id=5634662

⬐ stefanix

Not opposed to Bret's ideas but also feel he has never looked into how artists use code and design interactions. Many like programming environments that are visual (like max/msp) with immediate feedback while other prefer the austerity of code. There is something very satisfying when going from code straight to simulation. It often feels more like materializing ideas than noodling around with guis.

Canonical example: https://www.youtube.com/watch?v=hQhoe_hJIZ8

⬐ jbrennan

It's true some artists can already work with interactive media through code, but I think his greater point might be that we're selling every other artist short if they've got to express themselves in code instead of geometrically.

So while there are some who are currently exceptional to that rule, we're still leaving out a ton of potential art by limiting interactive art to code.

⬐ meemoo

tl;dw:

* "Drawing dead fish" refers to using a dynamic medium (the computer) to make static images and unchanging animation.

* Games represent the promise of dynamic media: things on the screen have behaviors.

* Dynamic behaviors are generally defined by code. We should break that association and design tools that allow more direct, visual manipulation of behaviors.

* Code thinking is like algebra. Tools demonstrated define behaviors more like geometric construction.

* Key takeaway: dissatisfaction with Photoshop, After Effects, and other digital art tools that don't take advantage of the media.

⬐ ColinWright

There are other submissions of this - you might want to check them out in case discussion arises there first. This one is the first, the others are here:

https://news.ycombinator.com/item?id=5634668

https://news.ycombinator.com/item?id=5634677

⬐ meemoo

This reminded me of many hours exploring The Geometer's Sketchpad before my coding days (and before that, a straight edge and compass). Ooh just found http://geogebraweb.appspot.com/app.html

I like being able to code in different paradigms and switch between them.

⬐ JacksonGariety

This is probably one of the most amazing talks I've ever heard. At its core it lets you understand just how unexplored the computer is as a medium.