HN Theater @HNTheaterMonth

> It feels like there's an unexplored dimension of abstraction here. Something where generics, interfaces and higher order functions are too static and low level primitives. Something where it's really difficult to pinpoint what exactly is the repeated pattern here and how to exploit it. Instead of generic framework that can do everything, I'd like an efficient way do something specific.

This bit reminds me a lot of a concept Aaron Hsu mentions in his fantastic talk on anti-patterns in the Iversonian languages[0], a paradigm he terms "Idioms over Libraries". The idea being that because of the expressive power of APL's primitives programmers often opt for utilizing easily recalled snippets of symbols over the importing of library code to implement common functionality. This facet of array-lang culture can be seen with sites like APL Cart[1], which serve as indexes of these frequently used snippets.

I think a lot of people miss that this is one of the major advantages induced by the terseness of these languages (it's not just about code-golfing for the implicit joy of laconicism). By allowing for the direct expression of algorithms in the same context they are being used, APL and its kin enable one to tweak and optimize these idioms on a case-by-case basis: precisely the kind of specificity the author of this article is asking for.

[0]: https://youtu.be/v7Mt0GYHU9A?t=1027

[1]: https://aplcart.info/

Comment from a year ago(https://news.ycombinator.com/item?id=22562344):

>A story, since in retrospect I think it's worth telling. Some years ago I was at the SF Clojure meetup. Arthur Whitney's daughter worked at the sponsoring company, and he agreed to come tell us about K.

>In retrospect, I don't think we gave him the welcome he deserved. No one was rude or anything, but it seemed there was a disconnect: Arthur was keen to show off how fast K (and Kdb) was, over zillions and zillions of rows.

>But the thing that the Clojurists were all searching for (that got them into Clojure in the first place) was expressivity. Is Clojure fast? I don't know, generally the problems I face come down to avoiding balls of mud rather than performance bottlenecks. And I think that was true of most there.

>So Arthur got something of an underwhelming reception. I remember someone asking "Does K have the ability to self-modify, a la Lisp macros?" When Arthur said no, you could see most people in the room just sort of mentally shrug and move on.

>And this was too bad. Because recently I've been playing around with J (another APL descendant) and been very impressed by some expressivity/readability benefits. Some small things that have very big effects on the codebase you actually end up with.

>The first thing is the avoidance of abstraction. To use a Twitterism:

>Broke: Just write your code and don't divide it into functions, creating one long main method

>Woke: Divide your code up, naming parts that get reused

>Bespoke: If your code is made up of really really short things, it ends up being shorter than the names you would use, so you can just write the thing rather than your name for it. An analogy would be: there is no human-comprehensible way to communicate the idea of "picosecond" in less time than an actual picosecond.

>The other thing I didn't expect was the benefit of multiple dispatch being baked into e v e r y t h i n g. In Clojure I might write (map + a b) to add each index together; in J I could just write a+b.

>This is neat stuff! Best practices for keeping complexity down in APL's tend to be the opposite of what they are in other languages. Aaron Hsu gave a talk about this: https://www.youtube.com/watch?v=v7Mt0GYHU9A

>It's too bad! Arthur came to tell us about speed---there's a reason it's used on giant datasets in finance, where performance translates directly into cash---but I wish we'd had the presence of mind to ask more about experience of writing K.

>So, Arthur, if you're reading this: Sorry everyone seemed kinda bored in SF a few years ago when you kindly came to present. We missed out!

i've sometimes thought that APL could mesh really well with spreadsheets.

https://www.youtube.com/watch?v=v7Mt0GYHU9A https://www.youtube.com/watch?v=UltnvW83_CQ

I think I found a nice example: https://youtu.be/v7Mt0GYHU9A?t=1237

⬐ kick

Sorry for seeing this so late! I didn't have any in particular in mind, but it comes up pretty frequently throughout, I believe.

A story, since in retrospect I think it's worth telling.

Some years ago I was at the SF Clojure meetup. Arthur Whitney's daughter worked at the sponsoring company, and he agreed to come tell us about K.

In retrospect, I don't think we gave him the welcome he deserved. No one was rude or anything, but it seemed there was a disconnect: Arthur was keen to show off how fast K (and Kdb) was, over zillions and zillions of rows.

But the thing that the Clojurists were all searching for (that got them into Clojure in the first place) was expressivity. Is Clojure fast? I don't know, generally the problems I face come down to avoiding balls of mud rather than performance bottlenecks. And I think that was true of most there.

So Arthur got something of an underwhelming reception. I remember someone asking "Does K have the ability to self-modify, a la Lisp macros?" When Arthur said no, you could see most people in the room just sort of mentally shrug and move on.

And this was too bad. Because recently I've been playing around with J (another APL descendant) and been very impressed by some expressivity/readability benefits. Some small things that have very big effects on the codebase you actually end up with.

The first thing is the avoidance of abstraction. To use a Twitterism:

Broke: Just write your code and don't divide it into functions, creating one long main method

Woke: Divide your code up, naming parts that get reused

Bespoke: If your code is made up of really really short things, it ends up being shorter than the names you would use, so you can just write the thing rather than your name for it. An analogy would be: there is no human-comprehensible way to communicate the idea of "picosecond" in less time than an actual picosecond.

The other thing I didn't expect was the benefit of multiple dispatch being baked into e v e r y t h i n g. In Clojure I might write (map + a b) to add each index together; in J I could just write a+b.

This is neat stuff! Best practices for keeping complexity down in APL's tend to be the opposite of what they are in other languages. Aaron Hsu gave a talk about this: https://www.youtube.com/watch?v=v7Mt0GYHU9A

It's too bad! Arthur came to tell us about speed---there's a reason it's used on giant datasets in finance, where performance translates directly into cash---but I wish we'd had the presence of mind to ask more about experience of writing K.

So, Arthur, if you're reading this: Sorry everyone seemed kinda bored in SF a few years ago when you kindly came to present. We missed out!

⬐ idoh

I was at that meetup, and I do remember the Clojure crowd not being all that impressed. But it definitely came across that Arthur Whitney was a type of programming wizard. He had some offhand remark that he was running his laptop off of an OS he made with K, and was editing with an editor he made in K, but somehow what still working on getting a sound driver to work.

⬐ kick

kOS! Some of the people who helped work on that hang around news.YC!

There's some pretty interesting stuff on it. The Impending kOS paper has probably introduced more people to APL and k than just about anything at this point.

http://archive.vector.org.uk/art10501320

⬐ eternalban

looks like it was discussed: https://news.ycombinator.com/item?id=20250401

⬐ kick

There have been a bunch of discussions on it, yeah. It's a fun subject!

⬐ RodgerTheGreat

It was directly the reason I wrote oK. I wasn't satisfied with learning k4 when I knew that somewhere out there Arthur was tinkering with k5, and the only way to have it was to build it...

⬐ wvlia5

Thanks for doing it, I played with your code, and it was inspiring.

⬐ 7thaccount

What are the big advancements between the major K versions?

⬐ beagle3

K1 was never available outside of Morgan Stanley AFAIK, and I’ve never seen any docs of it. So ...

APL -> A+ : opinionated (vector indices always start at 0, for example) and “electric” GUI (deserves a whole post to describe)

A+ -> K2 : significantly simplified - only ascii, no multidimensional arrays (uses nested vectors instead), much smaller vocabulary but just as effective in the basics department (e.g. the Whitney where/replicate monastic operator replaces 6 or 7 APL operators), added simple dictionaries, workspaces, dropped a lot of math from language. Simpler electric GUI. Time and date native types (as floating point)

K2 -> K3 mostly dropped GUI afaik. Added integral time and date.

k3 -> K4/kdb+: simplifies language more, comprehensive dictionary, TSDB rolled deeply into language (was previously a product written in K), native 64 bit support.

K4->K6: still more simplification, I think k6 is the kOS language. Never left dev state.

K6->k7->k9: continuous simplification, now at shakti rather than kx/firstderivatives - and with much more community involvement.

⬐ hu3

Concise! Thanks.

If I were to explore K today, where do I start and what version would I pick?

⬐ chrispsn

Q/KDB+ for something production ready now.

k9 (Shakti) for a simpler product that's in the active language design phase and with not as much integrations - yet.

ngn/k for an actively developed version that's free and targets k6-era (for now).

⬐ nivertech

Does either k9 (Shakti) or ngn/k have built-in timeseries database / qSQL[1], table / keyed table datatypes or something comparable?

--

[1] qSQL is actually a DSL implemented in K on top of kdb functional queries

⬐ chrispsn

k9: yes to all.

⬐ idoh

Does anyone know pricing on k9? Like could a small software shop or bootstrapped business use it?

⬐ chrispsn

I'm sure they'd be open to discussing that with you! Try Fintan: [email protected]

⬐ beagle3

I see the one letter email is spreading..... I’m actually surprised Arthur can live with such a long and verbose email address - 12 whole letters is almost enough for him to write an music player or something.

⬐ gricardo99

I had to google "ngn/k" [1]. Interesting project, and also interesting what google returned:

   1 Nigerian naira / Boltzmann constant =
   1.96791577 × 1020 m-2 kg-1 s2 K U.S. dollars

1 - https://bitbucket.org/ngn/k

⬐ QuesnayJr

I'm curious about the Whitney where/replicate monastic operator, and how it replaces 6 or 7 APL operators. My Google-fu was not up to the task.

⬐ beagle3

“Replaces” in the sense that you would use it instead, and not feel like you are missing something, not in the sense that it is equivalent char-per-char.

APL has a few expand/compress operators that take a boolean vector; all of these in K use monadic & (where/replicate), either on the at-index e.g. +/a[&x>3] which is read “sum over a where x is greater than 3”, or at the amend-index, e.g. @[&7 9 15;i:&x>3;:;7-x@|i] which is read “in a vector with seven 0s, followed by nine 1s and fifteen 2s, in the indices where x is greater than 3 (call those indices i), place 7 minus those values from the same indices in reverse order.

Obviously contrived examples, but in APL you’d have to compress/expand a couple of times instead. (Sorry, don’t remember the operator names, my APL is rusty, last wrote a real program in 1992....)

⬐ mlochbaum

NARS2000, Dyalog APL, and dzaima/APL all have the Where function now. See https://aplwiki.com/wiki/Indices.

⬐ beagle3

Thanks! Should probably brush up on my APL, as I have last really used APL2 on a mainframe nearly 3 decades ago.

But I have gotten lazy and too comfortable with K.... oh well.

⬐ miltondts

I'm not familiar with J but it also appears to have something similar. However the resulting code seems a bit more convoluted than the K version. +/ (I. a>3) { a

⬐ TheRealDevonMcC

More idiomatic J might look like this: +/a#~a>3 .

⬐ magicalhippo

> Aaron Hsu gave a talk about this

In it he mentioned that abstraction is bad, which was also mentioned in the previous K discussion. I still really struggle with that belief.

I find abstractions essential to mentally manage the complexities of the overall solution. For example I just implemented some REST calls at work, and when coding a GET or a POST call I don't want to think about the gritty details of how to conjure up an OAuth token from the multiple supported ways just so I can stuff it into the HTTP headers. I far prefer to know that the passed OAuth token provider abstraction (interface instance) will provide a suitable token if asked nicely.

Though perhaps he must mean something else than what I mean by abstraction. After all, the APL sample code he showed is abstracting away quite a lot by way of the various operators and whatnot.

⬐ hcarvalhoalves

I believe our industry overuses the term "abstraction", and incorrectly. You're probably right that he meant something different by it.

In your example, the OAuth token provider is concrete, and not having to know the details of OAuth would be an example of encapsulation.

Abstraction would be deriving general rules (of authentication, perhaps?) by drawing from the OAuth example, and by induction being able to generalize for anything not OAuth specific.

⬐ magicalhippo

So wouldn't a further abstraction be to introduce an auth provider that returns a set of HTTP header values to be used, or something along those lines?

I could then implement the OAuth auth provider based on the OAuth token providers interface I got now. And this would allow me to not think about how exactly the auth is done at all, just that it is done somehow.

Isn't this the layers of abstractions mentioned? If so, how is this bad?

⬐ willtim

There are many good points about KDB/Q the product (most of all simplicity) and it certainly was well-engineered. But K is not a modern language and its performance is only impressive as an interpreted language; and these gains come at a significant cost (various limits on expression sizes / variable counts / branch sizes, no closures, poor error messages). It's only really "fast" on bulk operations, much like Python.

If you want to see what a modern K or Q might look like (with static types and an embedded JIT compiler!), checkout the excellent Hobbes project from Morgan Stanley: https://github.com/Morgan-Stanley/hobbes

⬐ TurboHaskal

Well, this is one of the reasons why I left the Clojure community (along with the complete disregard for performance and the excessive personal branding with low effort, 101 blog posts). It really feels like a cult sometimes. They already program with the perfect language that Hickey designed for them, so why bother with anything else?

⬐ bladecatcher

Great point re: the “Bespoke” style of programming. It takes some unlearning and getting used to tacit programming [0] and trains [1] to appreciate this.

I’ve worked on fairly complex systems using kdb/q that are a joy to debug because the codebase is a handful of files. q is admittedly a more simplified and “mainstream” cousin of J, but allows a fair bit of expressiveness. The powerful data store (kdb) and q-sql that comes along with it makes it one of the most concise data platforms ever. Too bad it’s not free for commercial use!

[0] https://www.jsoftware.com/help/jforc/tacit_programs.htm

[1] https://www.jsoftware.com/help/learning/09.htm

⬐ jonahx

How does it compare to Jdb?

⬐ kazinator

> so you can just write the thing rather than your name for it

I do not find this convincing. Names, especially good names, can convey additional meaning, which is helpful even if the names are longer than what they stand for in terms of raw character count.

One obvious example is that I would often rather see a seven-letter manifest constant than the two-digit decimal literal it represents.

In a similar vein, I would rather have a structure with named fields, than array indices or offsets. employee.department is objectively better than e[13]; a compiler can generate the equivalent of the latter from the former.

In calculations that have many nodes in the syntax tree, it's helpful to break up the tree and give intermediate values names, which remains true even if the names are longer than the intermediate calculations.

A name may longer than what it denotes in raw character count, but in spite of that it may less complex due to its flat structure which only indicates "same or not". If a 13 character identifier stands for some 9 character syntax that denotes a tree with 7 nodes, the identifier is certainly simpler.

Lastly, different expressions that occur in a program can sometimes be very similar to each other. If we do not use names, then these differences can be hard to find. It might seem that identifiers which are longer than what they denote make the problem worse, but that is not necessarily so. Identifiers which are clearly different from each other can be selected for denoting forms that are very similar to each other. If instead of ((<]):) and ((<[):) I write foolyfoo and barlybar, that will be less error prone.

⬐ smabie

I used to think the exact same things, but after using kdb+/q for AoC2019, I felt like I had a revelation. When you get rid of all abstraction, your code size is reduced so dramatically that abstraction is no longer needed. My solutions were maybe 10-100x shorter than everyone else's (except some guy's k code I found) and to this day, I can still mostly remember them, character for character (much like how an equation is easier to remember than a traditional line of code).

Being able to fit all of your code in your head at once is a truly magical feeling.

⬐ chrispsn

BTW, some examples of AoC 19 answers in k:

ngn's; targets ngn/k (dude's a machine): https://bitbucket.org/ngn/k/src/master/aoc19/a.k

Mine; progressively porting from k7 to k9. Not as many answers and probably worse coding than ngn's, but perhaps more approachable for newcomers. https://github.com/chrispsn/aoc2019

llelf's; targets k7. Pretty slick. https://github.com/llelf/advent2019/blob/master/sol.k

⬐ kazinator

This obfuscated "magic eye" C program also produces a magical feeling!

http://www0.us.ioccc.org/2001/herrmann2.c

This is a program which generates stereograms. It not only does that, but its source code is one; if you stare past it into the distance, you will see the letters IOCCC appear floating in the foreground.

Accompanying text: http://www0.us.ioccc.org/2001/herrmann2.hint

If only the industry would pay us for having magical feelings.

⬐ kazinator

You should give definitions a chance, because they point to a way to make your code even smaller. Consider that dictionary compression, like LZ77 and LZSS works by encoding redundancy using symbols that index into a table.

Even APL/J/K programs with repeated patterns in them are further compressible. Occurrences of any repeated string more than two bytes long can be replaced by a 16 bit symbol.

Speaking of which, why don't you just learn the data structure and algorithm of a compression program like gzip inside out? Then you can write compressed code in any language. Others can easily work with your code by uncompressing it and then compressing it again when their changes are applied. Or vice versa; you can work with other people's code by compressing it first, then doing your changes in the more convenient, ergonomic compressed format. Maybe some checkout/commit hooks can do that transparently.

Compression will ferret out gratuitous frivolities like 17 copies of remoteDatabaseConnectionHandle, and turn them into a convenient, tiny code.

Think of how much smaller it can be if you go beyond reducing tokens to character. You've now got Huffman and dictionary coding at your disposal, and possibly more.

⬐ alimw

You might be right that good code needs names. But since in a terse language like J or K it may take longer to think up a good name than to write the code, it's handy that one doesn't have to have names for everything when prototyping.

⬐ rgoulter

What's the cost of reading it?

Just use comments to roughly convey what a line of code does, and then for more precise understanding, take the time to read through it?

⬐ sriku

The "write the thing" attitude perhaps comes to some extent from (human) languages that have that as a natural word invention mechanism - such as German, Finnish, Sanskrit. In Sanskrit, etymologies seem to often go down to single syllables.

I quite often miss that in programming languages. To some extent, functional languages let you construct functions that compose like that and we also see that happen with CSS with frameworks like Semantic-ui and Bulma. But it would be fun to have or experiment with it at the identifier level.

Edit: I generally prefer to use "abstraction" to refer specifically to "beta abstraction" and not the "make named functions" kind ... Which is more about encapsulation. Beta abstraction always results in something more general and abstractions are about leaving out detail.

⬐ philosopher1234

Each method has its own innate complexity, the complexity of "whats this method do? when do I use it? Why would I use this method instead of a different method in this class? Now I have to dive into this method to read its implementation which pulls me out of the reading context I was in before"

Also, your point about the meaning beyond the code can be served by a comment, which drops the constraint of "don't make it too long" and "write in camel case", and also allows inline code reading!

There's a time and a place for everything. I reccomend "A Philosophy of Software Design" for a book that elaborates on the idea of "less and longer methods tend to be better than many concise methods".

What you want to make short/simple is the API, rather than the implementation!

⬐ sbergot

A function has a well defined scope, parameters and output. A comment does not. A function name describes an intention. Within the scope of this function, someone can check whether the intention is respected by the implementation. If this function calls other functions, just assume that they are working as their name suggests. If that is not the case, fix those other functions later.

This has been my overall approach to maintaining code. To such extent that I design things around the possible names I can find. Some call that DDD. I call that respecting my team mates. Without writing and naming function, this is not possible.

Languages such as J and K are fine for single developers & prototypes but really bad for bigger projects.

⬐ nivertech

100% agree

https://news.ycombinator.com/edit?id=22565985

⬐ xelxebar

Do you have trouble discerning "cobble" vs "coddle"? Maybe; how about with context? "We cobbled together some crazy hack," vs. "We coddle our infants to show love and affection."

Arguably this comparison is quite similar to your "((<]):) and ((<[):)" example. Certainly the relationship between "]" and "[" is the same as between "b" and "d". Kids first learning to write often do mix up "d" and "b". The difference certainly seems subtle and arbitrary in the first stages of study.

Do you have much experience in APL/J/K?

As I see it, the point is that these languages offer some really spectacular ergonmics and expressibility _once one has gained fluency_ and in a way not possible elsewhere.

This is really surprising! Perhaps that's why there's a lot of initial skepticism.

In the space of all possible programming languages, it certainly seems like we've explored a pretty darn good chunk. There are thousands of languages trying out myriad different possibilities. We have everything from C to Java, from Haskell to Scheme, from Forth to Bash, and this still doesn't do justice covering all the paradigms and methodologies we programmers have dreamed up. Is it really possible that there's some random corner of language space that offers something more than localized and incremental benefits? Is it really true that there's some wacky scheme for writing code that is way shorter to write, pleasant to read, while also running fast?

The answer is yes! And that's super cool! The catch is that it's a really weird family of languages, but if you spend the time to learn them well, I suspect that you will also see what we are talking about.

⬐ kazinator

With context, I can understand "We co___ed together some crazy hack" and "We c___le our infants to show love and affection."

These sentences almost give definitions to the partially obliterated words, which is why we can guess what those words are. We can form plausible definitions, and then work backwards through our vocabulary by pattern matching.

(I would probably guess "cuddle" for the second one; I have negative associations with "coddle", since it's mainly used in describing situations when ineffective adults are supported economically, and protected from the consequences of their bad decisions.)

The strings in your APL/J/K are not existing vocabulary words that you're re-iterating with a context that gives their definitions. (Some of them may be idioms that appear in other programs, including other people's programs, but others are local inventions in the program.)

If you obliterate parts of them with underscores and then hand that to the next programmer, they are screwed.

Cobble and coddle are confusable. If we use cobble as a parameter name in some C function, and later refer to it as coddle elsewhere, only the compiler will notice. (Provided it's not the case that they are both defined, and have compatible type.)

  extern int coddle;

  // ...

  void frob(int cobble)
  {
     // [ snip 300 lines ]
     if (some_condition || (another_thing(foo) &&
                            some_condition(foo, coddle++))
        do_that();
     // [ snip 400 lines ]
  }

My eyes will likely will not spot that the "coddle++" isn't "cobble++", and the compiler has nothing to say. Doubly so if I'm not aware of the existence of the coddle global.

I remember copious instances in the past when my brain decided to hide a difference like that, and then "cached" the decision so that it remained hidden even after staring at the code. Once you believe that coddle++ is incrementing the cobble parameter, your brain may not let your eyes take a fresh look at it.

Very recently, I was tripped up for a while by a sentence in some documentation that talked about PTP (precision time protocol) and P2P (peer-to-peer) in the same sentence. It didn't make sense until a fifth re-reading with a coffee break in the middle.

⬐ xelxebar

Not sure what to tell you if you're convinced I'm lying.

The latter half of your argument I find strange. It is arguning against APL/J/K using a C example. Mimicking this tactic, I could argue that C is completely unreadable because it indents things at so many different levels. If my x86 assembly did that it would be a huge mess and hard to follow.

The languages are just so different that the syntax of one offers properties completely alien to the other, "What is this weird x=2 syntax? I just want 2 in my rax register and C doesn't let me do that?!"

Part of the issue with APL/J/K is that to understand some of the coolness, you really do just need to trust the evangelizers and drink the koolaid. Maybe you'll end up thinking we're all just loonies, but if you give it an honest try, I suspect something will click and you'll see the point we're trying to make.

⬐ kazinator

> using a C example

I see; it's okay for you to use the English language with sentences involving cobble and coddle, but not okay for me bring that back around into a computing context, using a language other than APL, J or K?

I used C because it's a lingua franca of computing, in which it is natural to use identifiers like cobble and coddle.

We can use Python, if you like, or JavaScript, Pascal, Lisp, Haskell, Basic, ... or a Makefile, shell script, HTML, CSS, Sendmail config file, LaTeX, ...

Thea argument is that coddle and cobble are, in fact, plausibly confusable as identifiers in computing, where we don't have the right kinds of linguistic context to spot a mix-up.

If I have <div class="cobble"> in some HTML, and the CSS has a .coddle { ... } selector, I can see not being able to spot the problem.

Furthermore, I have a point in that a sequence of symbols in your APL program which is particular to that program and not a common idiom is not comparable to a word like "cobble". It's more like "cmzblx" versus "cmzdlx".

Because (at least local) identifiers can usually be freely chosen by the coder, I can avoid this kind of problem. I can easily change "cobble" to "horse" (which doesn't look like "coddle") throughout a source file and be confident that it still means the same thing, because the letters are not operators that encode meaning.

> to understand some of the coolness, you really do just need to trust the evangelizers

They're too late; I might have been fooled as a 16-year-old in 1987.

> What is this weird x=2 syntax? I just want 2 in my rax register and C doesn't let me do that?!

When I first saw a BASIC program with assignments to some variable X, I thought they were a system of equations for the machine to solve. I parsed the syntax right, though; it was just the semantics that was off.

⬐ xelxebar

Hrm. Let's step back for a moment. Your responses read to me as defensive and it is not my intent at all to attack your position.

If you do not want to deal with APL/J/K, that's perfectly fine. Your interests and goals are very likely different than mine. All I wish to communicate is my experience, "Hey, this language seems really weird and counterintitive and completely non-sensical on the suraface, but I spent some time actually learning it and it turns out that these first impressions were wrong!"

You know how modern mathematical notation is really compact compared to the "more readable" prose that used to be used in the past? The funny part is that the prose is a lot harder to parse than the modern notation, provided you already know the latter. APL/J/K is a lot like executable math notation in that way.

What about you? Is there some core point you are trying to get across that I/we seem to be missing?

> I see; it's okay for you to use the English language with sentences involving cobble and coddle...

So my answer is that, yes, my analogy with English is relevant, because it attempts to communicate to you what it feels like to read unfamiliar J code in practice.

Your analogy with C is making a different point, that opaque identifiers with similar spellings and similar contexts are confusable. I agree.

⬐ beagle3

That’s an excellent comment. The only thing I have to add is that is’s not some newfangled concept that’s bad but we’ll only figure out why in a couple of years: APL has been around for over 60 years on various forms - it is about as old as Fortran and Lisp.

In fact, APL was designed for paper, as a well-defined notation for algorithms in math and CS papers (which are still either sloppy with algorithm semantics or spend a lot of space specifying their non standard notation to this day). But since it was well defined, it was actually executable, and when computers were good enough, an interpreter was developed at IBM. That’s why APL has so many weird symbols (it’s a paper notation for math, at heart). K is a practical simplified version/dialect that uses only ascii (as does J, which is essentially full APL with ascii)

One of the main complaints against APL is that they are write only. They take longer and more effort to read per character, granted, but per “semantic information”, for any reasonable definition, APl/J/K are way faster to read for someone proficient (and yes, I take proficiency for granted - the minute details in any language between Boolean short circuit, non local control transfer, integer/float conversions etc make proficiency essential)

⬐ agumonkey

about naming, h.baker said that combinators were lambda calc without the need for names. summarize a feeling i had when programming. i love sets of combinators to plug rather than languages.

⬐ nivertech

really really short things, it ends up being shorter than the names you would use, so you can just write the thing rather than your name for it

I inherited a kdb/q codebase like this, the problem with it, is that when you making a change, you need to find all these little code snippets and change them.

You need to rely on TDD to make it work, and this doesn't really scale to large projects.

Much better strategy if you don't want to name functions, is to use their hash.

But still conventional approach of choosing meaningful (longer than 1-2 letter) identifiers and writing comments works much better.

Also the same expression in q can represent several different things, i.e. a symbol might be a name of the column in the table, a key in the dictionary, an enum, a string, a constant, a global variable, etc. Without naming it, after a week you will no longer remember the original meaning.

To add salt on injury - you are limited on the number of global constants (really variables) you can use, so it's not always possible to name everything.

Always excited to see APL/J content on the top page.

Happily, this coincides with a brand new release of the J interpreter, namely J901. It's so new that the homepage[0] still claims 901 is in beta. The install page[1], however, does have up to date information.

If you would like to taste a bit of the difference before deciding to jump in, Aaron Hsu has a really interesting talk[2] that compares APL and "other" languages via the lens of Human-Computer Interfaces.

I am just beginning my journey into J, but so far it has been extremely rewarding. Head on over to #jsoftware on Freenode. The channel is super small, but has some really friendly and helpful people on it!

[0]:https://jsoftware.com/indexno.html [1]:https://code.jsoftware.com/wiki/System/Installation [2]:https://www.youtube.com/watch?v=v7Mt0GYHU9A

⬐ gmfawcett

I'm a J newbie, but I've been using J901 to do the Advent of Code challenge this year. I'm surprised at how pleasant it's been to work with. Some of the AoC solutions have been so straightforward that I wonder if the AoC author used an array language to design the puzzles in the first place. :)

It's not all roses! The error messages can be quite inscrutable sometimes, and I can't seem to get the J901 debugging tools working (except for old-fashioned print-debugging). But overall J is a charming (if quirky) language, and well suited for a bit of low-risk exploratory programming.

⬐ darkkindness

Remarkable, I've also been learning J901 for AoC. I'm curious, what resources have you been using to learn?

⬐ gmfawcett

Hey, small world! :) I hope that you're enjoying it, too.

Mostly I've just been using the articles and resources on the J Wiki. But I found it useful to skim through the free book, "Mastering Dyalog APL" [1] -- of course the languages aren't identical, but they're similar enough that many of the APL examples were instructive.

I also have the NuVoc page on speed-dial. :) I have definitely not memorized all of those crazy verbs yet!

[1] https://www.dyalog.com/uploads/documents/MasteringDyalogAPL....

[2] https://code.jsoftware.com/wiki/NuVoc

⬐ xelxebar

For whatever it's worth, there's also the older Voc[0] documentation which NuVoc is a rewrite and expansion on. The former is a heck of a lot terser and dryer, but I prefer it as a compact reference. The Voc reads more like a graduate math text, where each example serves to illustrate (often non-obvious) uses or interactions with other primitives. This means it requires careful reading, which I personally find helps me learn better.

The NuVoc, on the other hand, has a lot more in the way of straightforward examples along with extensive expository prose and hyperlinking.

J has somewhat peculiar terminology. This probably helps to avoid the pitfalls of overloading more familiar terms with J-nuanced meaning. Anyway, to make any headway with eithe Voc or NuVoc, one should at least be familiar with the Absolutely Essential Terminology[1].

For me, personally, that's about all the documentation I use. The mailing lists[2] are quite active, and some of the early founders of APL and J even interact extensively with beginers! Highly recommended.

There are also a lot of Essays[3] that illustrate high quality J code via solving specific problems. The book "At Play With J"[4] is also a really fun read, even if you have to keep NuVoc by your side.

Then, of course, there is the IRC channel.

Those have been my primary sources of learning. Hope it helps!

[0]:https://www.jsoftware.com/help/dictionary/vocabul.htm

[1]:https://code.jsoftware.com/wiki/Vocabulary/AET

[2]:https://code.jsoftware.com/wiki/System/Forums

[3]:https://code.jsoftware.com/wiki/Essays

[4]:https://code.jsoftware.com/wiki/At_Play_With_J

⬐ pmoriarty

I remember a year or two ago I was struggling with an AoC problem, and the best I could come up with in Python was a one or two hundred line program. Then I saw an answer for it in about 20 characters of J. That was impressive, though I do still wonder how readable the language is in practical every day programming, as opposed to toy problems in a programming contest.

⬐ icen

I can't speak to j, but using q, its cousin, at work looks quite a lot more like the graph APL posted above. It's spread out over several lines, and heavily commented.

Most problems are also much simpler, and don't require any heavy one-liners.

⬐ kick

For readers who are unaware of what q is, it's a language on top of Arthur Whitney's k, made to be accessible to your average human, and mostly for SQL queries.

I don't think Arthur is as passionate about it as he is k, and he barely has anything posted on it.

It's great in some ways but usually lags behind k for people familiar with array languages.

⬐ kick

J is incredibly readable. For example, here's an (old) implementation of 2048 in it that's fairly small:

https://github.com/tikkanz/2048

⬐ gmfawcett

True. I'm not sure how I'd feel about maintaining a 2,000+ line J codebase. It definitely feels like a "programming-in-the-small" kind of language.

It is impressive how much logic you can fit into so few characters. Even other terse languages (e.g. Haskell) seem wordy in comparison. Of course this is especially true when the problem has a fairly natural array representation, and that might not be true for many real-world problems.

I haven't tried it, but I feel that a literate-programming style might work well for J. The code itself is very dense, so I think it might work well to have a long-ish, plain-text explanatory document with bits of J interspersed throughout it.

⬐ chongli

Perhaps the approach to maintaining a large codebase of terse code (such as J) is to keep the files very small and narrow in focus. This could end up very similar to a Unix userland code tree.

Of course, lots of documentation could also help. Maybe a style similar to R/Python notebooks would work.

⬐ gmfawcett

That's an interesting point. I wonder if the small, focused approach is (or was) common in professional APL/J/K development.

It turns out that there is a Jupyter notebook implementation for J [1]. I might play with that! I suspect that you're right, notebooks would be a good fit for terse languages.

[1] https://code.jsoftware.com/wiki/Guides/Jupyter

It's really no more difficult than learning recursion, pointers, memory allocation, type theory, functional programming, iterative programming, OOP, or any other of the myriad techniques that we use for computing. As I mention in my talk on APL design patterns, it's largely a matter of what we have been trained to understand, and there is some evidence to suggest that APL is actually arranged in a manner more natural for the human faculty than most computing languages:

https://www.youtube.com/watch?v=v7Mt0GYHU9A&feature=youtu.be

⬐ qwerty456127

> It's really no more difficult than learning recursion, pointers, memory allocation, type theory, functional programming, iterative programming, OOP, or any other of the myriad techniques

You don't really need anything of this to solve a problem with Python and Pandas.

I always feel like I wish I'd solve it with APL (because it's beautiful) but end-up just getting the thing done this way :-]

Thank you for the lecture, perhaps it is going to inspire me to actually learn to solve real problems with APL.

⬐ arcfide

Except that you do. Python is a form of OOP/Iterative programming, involves memory allocation, pointers, references, and the like. Pandas requires the skilled application of a general purpose language coupled with a domain-specific vocabulary for your problem space.

I maintain that APL isn't fundamentally more difficult. If there is something that you have as an example that you think is just too "hard" to do in APL that is very "easy" in Python and Pandas, please do let me know via email ([email protected]) as I would love to make sure APL has a good answer to such things (I want to make APL accessible).

Since you are talking about statistics and data analysis type things, you may find TamStat of interest:

http://www.tamstat.com/

Additionally, don't forget that Dyalog APL also has a full Chart/Graphics system (two, actually), and a large suite of idiomatic expressions for computing many classic analysis problems. There's a number of other integrations that people seem to be unaware of often enough.

I very seriously would like to receive from you one example of something you have done that you find too difficult in APL, because if it really is, then we can do better and I'd like to make sure that we do.

⬐ 7thaccount

Just seeing this. I'm not the original poster that you replied to, but their point is somewhat valid as is yours.

Python certainly requires one to learn iterative programming which is definitely not a walk in the park. I learned it over a decade ago, but vividly recall how non-intuitive it was at the time (global versus local scope had me scratching my head once upon a time). Then there is lists, tuples, dictionaries, pass by reference, iterators, file I/O...just soo much to learn.

One interesting thing is that I used OO methods in Python for ages before understanding how to write and use classes. To me, a = list.sort() was just a weird way to read a = sort(list). You can almost write Python entirely in an imperative manner and for short scripts I almost always do.

Learning all the OO aspects of Java/C# and all the fundamentals of FP (monads, currying... etc) is much harder for me than APL which is basically just learn what these symbols do and the very straightforward way to combine them and you're done. The only harder part of APL is somehow learning to replace imperative programming with array operations. Sure it is easy to do some things like sum up an array (+/ array), but that is also easy in Python (list.sum()). The idioms as APL people call them are not at all obvious to me, nor is the path of enlightenment outside of stop talking and go write some APL haha.

Maybe look at a different programming paradigm then? :)

I remember trying to learn APL and then watching https://www.youtube.com/watch?v=v7Mt0GYHU9A and was amused that there is this world where programming is just symbols :D

So maybe look at i.e. something apl-like, or maybe ml or haskell or erlang, where so many things you can do with functions and composing them, or even Prolog, where everything could be thought of as a set of rules/constraints to be solved (I think? haven't programmed in Prolog since uni :) )

⬐ geocar

APL is great.

If you haven't seen them, the late John Scholes did a number of absolutely beautiful demonstrations programming in APL[1], [2], [3].

[1]: https://www.youtube.com/watch?v=DsZdfnlh_d0

[2]: https://www.youtube.com/watch?v=DmT80OseAGs

[3]: https://www.youtube.com/watch?v=a9xAKttWgP4

⬐ jodrellblank

For anyone looking to get into APL, Adám at Dyalog posts a lot of answers on the CodeGolf and Programming Puzzles StackExchange site often with expanded/explained code, and he hangs in the "APL Orchard" chatroom on SO. His profile says "always excited to [talk about] APL" and that seems to be genuine. - https://codegolf.meta.stackexchange.com/users/43319/ad%C3%A1...

The most public APL dialect using the classic symbols (not J/K/Q/etc) seems to be Dyalog APL - they offer one of their older versions 14 free for non-commercial use (Windows), and they offer their current version 17 free for non-commercial use if you give them your full details (maybe Linux as well?). There is a book "Mastering Dyalog APL" by Bernard Legrand from ~2008 which is available as a PDF with companion files from: https://www.dyalog.com/mastering-dyalog-apl.htm although they have added to the language since that book was released.

There are other APL interpreters - GNU APL, NARS2000, ngn/apl, dzaima/apl (Java/Android) in various states of development and license conditions, and the developers of the last two are also sometimes in the APL Orchard chatroom on StackExchange.

Online there's https://tryapl.org/ run by Dyalog, and https://tio.run/# which has three APL interpreters.

The classic 1970s book "APL\360 an Interactive Approach" by Gilman and Rose is online as a PDF here: http://www.softwarepreservation.org/projects/apl/Books/ with other books as well. That's before the {} syntax for anonymous functions was added to the language, and other developments.

Videos:

Jay Foad while he was at Dyalog, talking through some Advent of Code puzzles with APL: https://dyalog.tv/Webinar/?v=Q_vgSN6rza0 (might want to skip the introduction of explaining the Advent of Code puzzle format).

Aaron Hsu with Dhaval Dalal and Morten Kromberg, talking through some problems using APL https://www.youtube.com/watch?v=Gsj_7tFtODk

And anything that you get from YouTube searching for Aaron Hsu, he has several talks / presentations from high level ideas to working with trees represented using arrays.

Qualification: I was introduced to FP a number of years ago and have finally reached a point where I feel that I have a deep understanding of FP in general. In my mind, Haskell is the "best" language out there ("best" in that it provides the best overall gain when all tradeoffs are considered).

I'm no longer confident that Haskell is the "best" language anymore. I was recently introduced to a language that might be even better than Haskell: Dyalog APL. (https://www.dyalog.com/). Try it out yourself (https://tryapl.org/)

To explain why, I recommend watching Aaron Hsu's videos on Dyalog APL:

- Does APL need a Type System? (https://www.youtube.com/watch?v=z8MVKianh54) - This is actually a very interesting question that is more nuanced than upon first glance.

- Design Patterns and Anti-Patterns in APL (https://www.youtube.com/watch?v=v7Mt0GYHU9A) - The main takeaway from this video for me was the principle of "Idioms over libraries."

- Higher Performance Tree-Wrangling, the APL way (https://www.youtube.com/watch?v=hzPd3umu78g) - Or how to model Trees without using pointers by using "Inverted Tables." (An inverted table is a table where the columns are the rows and the rows are the columns)

My favorite (so far) is Aaron Hsu https://www.youtube.com/watch?v=v7Mt0GYHU9A

because it pokes hard at OOP culture.

⬐ 6thaccount2

Yea. His discussions on HN (user arcfide) are well worth reading.

Some users thought his compiler was a farce before he posted the stats. Yea it's just a few pages, but the magic is getting to just a few pages. It apparently took 4M loc of added and then deleted code to get there (iirc it was a huge number).

⬐ agumonkey

he made a few videos about apl too, parallel tree processing.. I like his subjects of discourse. I wish I could work (or intern~) with him.

All of the languages you mentioned are watered-down and inferior versions of the programming models invented by APL and J.

By that I don't mean that APL/J are unilaterally better -- there are plenty of practical reasons to choose R/Matlab/numpy -- but they are unquestionably more beautiful and visionary. They hang together at a deep level in a way those languages do not. They are also much harder to learn, and much more fun.

On the readability of APL (terrific video): https://www.youtube.com/watch?v=v7Mt0GYHU9A

Alan Perlis on its beauty: https://www.jsoftware.com/papers/perlis78.htm

Two talks with these slides:

1. https://www.youtube.com/watch?v=9xCJ3BCIudI (Dyalog17: Patterns and Anti-patterns in APL: Escaping the Beginner's Plateau)

2. https://www.youtube.com/watch?v=v7Mt0GYHU9A (Design Patterns vs Anti pattern in APL by Aaron W Hsu at FnConf17)