HN Theater @HNTheaterMonth

Justine Tunney's APE (actually portable executables) exploits, er, makes use of, the fact that a shebang isn't required to run a shell script.

The single executable file generated by the Cosmopolitan linker creates an insane polyglot binary executable that runs on 6 operating systems as well as on bare metal (i.e. bootable as its own OS); in addition, it's a valid Zip archive, and you can modify files inside it using any zip utility... fascinating stuff. On Linux, it bootstraps execution initially as a shell script, but the shell script writes the "ELF" file header and executes it (I'm probably doing a horrible job of explaining though).

https://justine.lol/ape.html

If you're interested in learning more, this talk from 2021 is fascinating: https://www.youtube.com/watch?v=1ZTRb-2DZGs. It covers Redbean, a portable, run-anywhere web server built with the APE toolchain; since then Redbean 2 has been released, which adds a lot more features. Here's a link to Redbean: https://redbean.dev/

⬐ easrng

If you want a simpler polyglot, here's a sh/bat to get you started:

  REM=;echo hello from sh;exit#
  echo hello from cmd

Author here. Thanks! The other projects I'm known for here on HN are:

https://news.ycombinator.com/item?id=26271117 Redbean (2011) is a single file distributable web server. Here's a video of me giving a talk about it https://youtu.be/1ZTRb-2DZGs

https://news.ycombinator.com/item?id=24256883 αcτµαlly pδrταblε εxεcµταblε (2020) lets you build c/c++/fortran/etc. code so that it runs on seven operating systems

https://news.ycombinator.com/item?id=13769727 Operation Rosehub (2017) where I helped save a lot of people from an Apache security bug similar to Log4j issue today

I like doing these projects because they all fit together so well into a consistent story. For example, a lot of these past projects were what helped to make SectorLISP better. Thanks to APE you now have a 20kb quickly deployable and embeddable LISP interpreter. Thanks to Redbean we're going to have an online pastebin service that'll let you publish SectorLISP gists in a few days, for example: https://lisp.pub/1

Thank you Hacker News community for being so supportive and encouraging.

⬐ deknos

Hey, i have a question regarding your projects! and before that, i am a huge fan of redbean, this is so cool! as long as the world has also people like you with that cool stuff in it, there's still some fun and hope in the world.

but for the question: for your projects, do you have like a stable buildplatform (like nixos, minimal debian stable or sth like that) for which you publish your build instructions for your project?

⬐ jart

I used to do a lot of work on Bazel. I wrote its downloader code for example. https://github.com/bazelbuild/bazel/commit/ed7ced0018dc5c5eb... Bazel is nice, but these days I'm running a small scrappy operation, so I just use GNU Make. https://github.com/jart/cosmopolitan/blob/7064d736e3ded15087...

I like Make since it's able to build a repository with 17k .o files, 80 .a archives, and 661 .com executables from scratch in under a minute on one personal computer (if the kernel page cache is warm). I wrote a couple small helper commands to make the make config more manageable, like package.com, mkdeps.com, compile.com, ar.com, zipobj.com, and runit.com.

The reason why Make works for me, is because I think the root cause of needing things like Autoconf and Cmake is because most projects need to depend on seven different C libraries. I decided that, rather than focusing on writing a better build config, I'd rather use an unfancy build system and instead devote my energy towards having a single C library that runs on all seven of the platforms I'm targeting. I owe a lot of thanks to projects like musl, dlmalloc, dtoa, llvm, etc. from whom I borrowed source code. That enabled me to abstract portability at the libc level, rather than punting to #ifdefs and configs.

⬐ waynesonfire

Beautifully executed. I aspire to solve my problems with such grace. It's difficult and i fail often but not without trying.

⬐ Ericson2314

How do I get you interested in the Nix ecosystem? It's a great sandbox for all sorts of portability and bootstrapping experiments.

⬐ MonkeyClub

Awesome work, and very enjoyable to read, kudos!

> αcτµαlly pδrταblε εxεcµταblε

You mean ακτυαλλγ πορταβλε εχεκυταβλε?

⬐ haakonhr

Shouldn't it be: πορταμπλε (since β actually is a "v" sound so μπ is used for our "b")

⬐ MonkeyClub

Was going for visual resemblance, but certainly!

⬐ Datagenerator

Congratulations on all your progress especially creating the worlds tiniest programming language in the world with GC!

redbean is a pure fork() web server and according to ab it is much faster than nginx. I feel like we've come full circle with history. Apache started off with fork() since it offers the strongest security versus other i/o models. The problem is that code bloat makes fork() go slower so as Apache ballooned over the years there was this race to the bottom in terms of i/o models, caused by tragedy of the commons. https://youtu.be/1ZTRb-2DZGs?t=761

Messages per second, latency, footprint, name it. Particularly if it's gzip encoded. See https://youtu.be/1ZTRb-2DZGs?t=717 I'm working on giving redbean the fastest https serving too. Recently I've been helping to make the strongest elliptic curves go 3x faster.

⬐ javajosh

You seem to have written a glorious series of interlocking hacks which yields a small, tight, simple, fast, portable, useful nugget of functionality in a single file. Bravo.

I really like it and will attempt to use it for something real. However, I fear that even this bit of magic doesn't address the central problem of our time, which is software distribution. I believe that the web has solved that problem, and although the web is currently abused by central power, and webapps tend to be thin, animated protocol viewers, it doesn't have to be that way. You've created/discovered a local (maybe global, given real-world limits) minima of what a binary executable can be, but this only finds the minima of the pain of traditional software distribution, but doesn't eliminate it.

The real path forward, if I might be so bold, is to make a browser on top of cosmopolitan/redbean, and bring TBL's original dream of a singular client+server http/html runtime to modern fruition - but with additional superpowers that cosmo brings which I don't think TBL anticipated. No doubt some enterprising souls are already working to get Bellard's QuickJS into redbean to mimic node. Then you need window/drawing context, and the rest of the browser, including layout, could be done in (presumably equally tight) JS. Have you given any thought to exposing those drawing syscalls directly instead of delegating to the browser? And if you haven't and are interested, may I suggest Java's AWT v. SWT as an interesting case study in "where the indirection should go".

⬐ jart

QuickJS is already ported to Cosmo so adding it to redbean is only a matter of time. If we wanted a browser we could always port the one the SerenityOS guy built. It'd be the best thing since OpenSSH. Don't look to me to do it though. I don't do anything unless I can do it better than all the existing alternatives out there. I can build a better web server. I can build a better executable format. I don't think I can build a better Chrome. Most of the platforms I target don't even have GUIs.

Data control and privacy are two of the most important things. I wrote a popular open source project where I've sought to improve that. Would you be interested in taking a quick look and offering candid feedback? See https://justine.lol/redbean/ and discussion thread https://news.ycombinator.com/item?id=26271117 and interview https://youtu.be/1ZTRb-2DZGs and polyglot executable format https://justine.lol/ape.html Someone like you could help me know if I'm doing it in the best possible way, since studying the influence of groups like cygnus was what helped me learn the hacking stunts that allowed me to pull it off in the first place.

⬐ 1vuio0pswjnm7

Kinda sad to hear that small, offline programs to run on various OS was viewed as some sort of challenging problem for Google (or anyone). Offiline interoperability was intentionally made difficult. Too many people, many of whom do not know the history, are just way off-track. IMO, this project should not be called "mad science". It should be standard procedure and what many people are working on. Small, portable, fast, resource-efficient, offline programs. Instead they are working on all the monstrosities that she mentions. God bless her for admitting how slow Python is by comparison. And how large Go binaries are. She says she studied "old code" to make her discoveries. For years I have believed this is what is takes to make progress and I avoid all the "new" stuff that gets incessantly promoted to today's developers. IMO, no one "forgot" the things she discovered, others are just constantly trying to bury them. Outside of HN, as well as on HN, there is strong distaste for anything "old". Instead of being thankful for what we have to build on, that has withstood the test of time, I see steady resentment for it because it is not "modern". This "modern" stuff is all throwaway, it is not built-to-last.

When salaries are paid by advertising, then the only content that matters is "popular" content. Unpopular stuff is not valued. That includes unpopular even though technically superior technology.

⬐ 1vuio0pswjnm7

What gives me pause about this project is the author's apparent satisfaction with having a Linux-only development environment. With all the portability techniques discovered, portable Makefiles and a portable build system seems to have been ignored. NetBSD's toolchain and NetBSD itself can be built on a variety of OS. Yet this library can only be built on Linux. That does not sound right. Only supporting "NetBSD 9.1" when the code for older kernels is so easily available is also a bit odd.

⬐ dundarious

A lot of impressive and intricate work has gone into getting their project to this stage. I think especially something like only supporting running on NetBSD >= 9.1 can be forgiven, but even the build OS requirements seem quite justified for now (Linux, macOS, and Windows supported, the latter two require the GCC ELF toolchain though).

⬐ BBC-vs-neolibs

A port of a compiler, for instance TCC, seems to be the next logical step. TCC could be a portable executable.

⬐ jart

Author here. The Apple and Microsoft linkers make that difficult. The GNU and LLVM compilers usually choose to conform to those difficult conventions when compiled for those platforms. You can help us fix that, because we're working on chibicc, which is like TCC but 10x better. https://github.com/jart/cosmopolitan/blob/master/third_party... It's not there yet. It can be built as an actually portable executable. I already wrote an assembler for it. Soon we'll have a C linker for it too, and thus a whole toolchain in one file. The tradeoff is it doesn't optimize. But it supports C11 and most GNU extensions. So when we're OK having velocity in development anywhere with code that only goes half as fast, but compiles 5x as fast, this will be the way to go.

⬐ 1vuio0pswjnm7

Is there an option for BSD users to use their own toolchain instead of chibicc or the included version of GCC. TCC, as impressive as it was, was never made to be portable. Portability was apparently not a goal. After all these years, it is still a WIP in pkgsrc. Correct me if I am wrong but with a chibicc solution cosmopolitan will still be "Linux-centric" because anyone compiling on Linux will have the option to use an optimising compiler instead of chibicc.

⬐ jart

It's build-once run-anywhere. I build my software on Linux. I'm the primary contributor so that just happens to be what's best supported. Cosmopolitan is a labor of love that I made to make developers happy. I'm sure over time it'll evolve into build-anywhere run-anywhere. You can help me will that future into existence. You yourself said that something like redbean should become the new normal. When I studied the old code, I saw its merits, and I've used my popularity to help change what popular means.

⬐ 1vuio0pswjnm7

I support what you're trying to do 100%. (Except all the ANSI escape code and Unicode stuff. I disable UTF-8 wherever I can.) IMO, "popular" is overrrated. What actually works the best, e.g., fastest, smallest, most portable, lowest resource usage, is more interesting. I want to compile the modified runtime crt.S and ape.S on NetBSD. Maybe with /emul/linux it's possible. I've been using shell scripts without shebang lines for decades (because NetBSD's default shell is my preferred interactive shell and I hate typing). It was funny to see that described as "forgotten".

⬐ jart

I'm pretty sure you can just use the cosmopolitan binaries. Right before the v0.2 release I verified that the release binaries worked with the LLVM that came with FreeBSD 12 on the stock system, without any Linux emulation. (Sadly a regression in ape.lds got introduced at some point since then, that broke LLVM support). So if you use the v0.2 release ape.o release binaries, you should probably be OK.

If you're a fan of NetBSD's shell, you'll be pleased to hear that it's now more bug free thanks to Cosmopolitan. I found a bug with it and reported it to the dev team months ago. In many ways I'm proud of the fact that the work I'm doing is helping to support the BSD communities. It's not perfect yet. But I think a project like this is going to attract further interest to the platform. Particularly for production use cases.

⬐ 1vuio0pswjnm7

Not really a fan of using binaries I cannot myself compile on NetBSD (i.e., non-portable code that only compiles on Windows/MacOS/Linux). No doubt the provided binaries will work, but I want to compile crt.S and ape.S myself on a NetBSD host.

Have not looked at it closely but is the "unbourne" shell more or less a copy of NetBSD's Almquist shell. Would be neat to use an Almquist-derived shell with command history on Linux.

From what I have seen of other OS, attracting "further interest" is not always a good thing. Depends on whose interest it attracts. The negative side effects of popularity, to me, are not always worth it. The Linux kernel is said to get an additional 1.5 million LOC per year. Of course it can be trimmed down (even smaller than NetBSD's), but still. The fewer cooks in the kitchen the better, IMO.

⬐ pxc

The history you mention sounds pretty interesting to me. Where can/should I read (or hear) about it?

⬐ kickscondor

Amazing! I loved seeing the binary dump slides of the executable headers and ZIP portions narrated. This is the true hacker spirit incarnate.

⬐ SPascareli13

Not to be confused with RedbeanPHP.

⬐ teddyh

Nor with “Red Bean Software” <https://red-bean.com/about.html>.

⬐ tssva

Nor with "Red Beans and Rice" <https://en.wikipedia.org/wiki/Red_beans_and_rice>

⬐ d136o

mad science

⬐ beepbooptheory

that guy has like 5,000 GitHub repos and I wish he'd see my PR about his timidity wasm library !

⬐ Henrry9610

Jalkers Milon

⬐ dang

Some related past discussions:

Redbean 0.4 - https://news.ycombinator.com/item?id=27001961 - May 2021 (13 comments)

Cosmopolitan Libc: your build-once run-anywhere C library - https://news.ycombinator.com/item?id=26277521 - Feb 2021 (11 comments)

Actually Portable Executable - https://news.ycombinator.com/item?id=26273960 - Feb 2021 (162 comments)

Show HN: Redbean – Single-file distributable web server - https://news.ycombinator.com/item?id=26271117 - Feb 2021 (249 comments)

How Fat Does a Fat Binary Need to Be? - https://news.ycombinator.com/item?id=26103769 - Feb 2021 (67 comments)

Cosmopolitan Libc: build-once run-anywhere C library - https://news.ycombinator.com/item?id=25556286 - Dec 2020 (166 comments)

αcτµαlly pδrταblε εxεcµταblε - https://news.ycombinator.com/item?id=24256883 - Aug 2020 (286 comments)

⬐ capableweb

Do you keep something like topic pages in your personal wiki or how do you keep track of these? Just really quick and efficient searching when you remember something has been discussed before? Maybe even some Google Chrome extension to insert story title + link + date + # of comments from submission links into the comment field?

⬐ dang

Nope, I just use HN search (hn.algolia.com) but I have software that makes it faster and does the formatting - yes, it's a browser extension that I've been using for moderation and adding to for, gosh, coming up on 9 years this fall. See https://news.ycombinator.com/item?id=27236708 and the other links there for more information if interested.

I'm going to make it so any HN links to past threads will render as "title - link - month-year - # of comments" or something similar.

One of these years I'll publish the browser extension too, and the Arc-over-JS implementation it's written in.

⬐ dingdingdang

I came into this thread thinking I was going to cross-post the earlier Cosmopolitan/APE related posts, but dang, someone beat me to it, both on the username AND content front?! Thanks though, saved me quite bit of googling.

⬐ nighthawk454

dang is an admin here on HN :) often when there's been significant past discussions, you'll find these posts up top. quite helpful!