HN Theater @HNTheaterMonth

Why not visidata?

https://www.visidata.org/ https://www.youtube.com/watch?v=N1CBDTgGtOU

(It does much, much more than pretty printing, but no reason you can't use it for that.)

⬐ qwertyuiop_

This is why I love HN. Never knew this existed. It has become my favorite tool in the past 5 mins I installed it. Also reminds me of Mainframe programs that I encountered in the past. I wish we had more tools like this instead of electron mouse click based apps for people who prefer speed and keyboard.

⬐ d4rkp4ttern

Visidata is fantastic, it is a lifesaver for viewing and getting stats on tabular data. It also handles json and others

⬐ certifiedloud

Visidata is a great interactive tool. TV seems like it would be better when scripting, or in one-liners.

⬐ dotancohen

For scripting I would use grep and cut, maybe awk. For scripting with CSV files, at least in my experience, you usually want specific columns from specific lines.

If TV had a switch for specifying only certain columns, that would make the job much easier.

⬐ flusteredBias

sounds like you are looking for xsv. I like that tool a lot for selecting specific columns.

⬐ flusteredBias

I love visidata! But when I want to just glance at a csv file I reach for tv (I used to use `column` which is more of a tv competitor than visidata). This is for a couple reasons.

1. tv gives a quick summary of the count of rows and columns

2. tv shows all columns at the bottom that don't fit in the terminal. With vd I have to scroll on wide data.

3. tv guides the eye to missing data better with NA highlights

4. tv has sigfig logic that is better. I work with files where the decimal dust can become long. Those unnecessary characters pushes remaining columns off the screen. This means the user would need scroll over to see additional columns. I generally think it is better to avoid additional key presses if possible.

5. tv is fast for large files. It does not have to read and format all of the data like vd. tv is focused more on looking at the file and not operating on file. It does not have to do as much as vd. That helps tv with what it is uniquely good at. "Do one thing and do it well"

It does not matter if your file is really wide (lots of columns) or really long -- tv will give the user a compact useful pretty print of the data. Why not use vd as a TUI spreadsheet and tv for glancing at csv files. They are both great tools in my eyes with different purposes.

⬐ saulpw

Hey there, VisiData author here. Nice work with tv! I'm sure it's more useful than VisiData for certain use cases. I just want to clear some things up since there are a few misconceptions here (which will happen if you don't use VisiData a lot):

1. In VisiData, The number of rows is always shown in the lower right, and you can see the number of columns with either Ctrl+G or a list of the columns with Shift+C. Or Shift+I for the list of columns with summary statistics (mode/distinct/errors/etc). This is an extra keystroke, but the amount of data you can get with that keystroke more than justifies it.

5. VisiData will instantly open and show any file it can, and continue to load the rest until it's done or you press Ctrl+C (or quit). Everything in VisiData is lazily evaluated, so it's not actually doing any more work than tv when you view the first page of rows, and then you can see the next few pages of rows with only one keystroke (PgDn, as opposed to having to edit a command and rerun it). Fewer keypresses ftw!

A lot of people think VisiData is a TUI spreadsheet, but vd is not a "spreadsheet" in the classic sense, as it's not cell-based. Its primary use-case is exploring and wrangling tabular data. It just turns out that this is what a lot of people are doing with their spreadsheets, but they have to bend over backwards to get Excel/whatever to play nice with their data's structure. By the same token, if you try to do little single-cell formulas in VisiData, it's going to be quite difficult.

For people who like static binaries and only need to view a few rows of CSV files, or produce part of a larger report in a pipeline, tv could be a better fit than VisiData, especially if it continues to be maintained. I'm always excited to see new data tools in the terminal space!

⬐ flusteredBias

https://github.com/alexhallam/tv/pull/58

I added VisiData in my README and represented it in a positive light in the description. Again, just wanted to apologize for my mistake.

#better-together

⬐ flusteredBias

Oo, I am sorry. I see I misrepresented VisiData. I apologize. Thank you for the corrections.

I have a lot of respect your work. Let me know if I can make it up to you. I would be happy to point people to VisiData in my README as a recommendation of a tool that is built to explore and wrangle tabular data.

Also, thanks for the compliment! Like you, I like seeing more data tools in the terminal.

visdata is pretty mind-blowing...at least, that's how i felt when i saw the demo the first time: https://www.youtube.com/watch?v=N1CBDTgGtOU

If you aren't familiar with Visidata, this video gives a pretty good idea of what it can do: https://youtu.be/N1CBDTgGtOU

It seems like making a living on shell tools can be difficult to impossible. The author Saul is using Patreon to help fund his work on this tool. I love this tool, and hope others can help support him too. Maybe this "duo-repo" approach will allow him to do this full time.

Another great tool for data exploration is visidata! This is a good example of what it can do. https://www.youtube.com/watch?v=N1CBDTgGtOU

⬐ sam_bristow

Visidata is great. My main problem is that I use it so infrequently so it's hard to build up the muscle memory to use it really effectively.

Not sure about saving but, more importantly for me, it can read sqlite DBs, which has been a godsend whenever I need to take a quick glance at some random DB just to see what kind of tables it holds and what does the data look like. I know my SELECTs but I can't, for the life of me, remember what are the commands to check schemas and DB metadata on sqlite ("Was it .describe? Maybe .schema? Or perhaps \t? Gah, let's check the help once again..."), so knowing that visidata gets me there much faster feels good.

> Also looks powerful in it's own right

If you aren't convinced of that yet, I'd simply suggest to take a look at this lightning demo (~5 mins): https://www.youtube.com/watch?v=N1CBDTgGtOU

The "video demo" is pretty buried in the site, here it is for your enjoyment. Like most viz, seeing it in action makes it much clearer than reading about it.

https://youtu.be/N1CBDTgGtOU

The section "TUI : Text-based User Interfaces" resonates with me strongly, and is somewhat relevant to the WTF thread[1] from yesterday, in particular this subthread[2] about GUI vs TUI.

Compare the statements:

> terminal protocols are ghastly display server protocols in disguise

vs

> > At what point do we just say why not 'install a modern GUI'?

> I have yet to see a modern GUI that didn't put pretty little animations a higher priority than being modular and configurable and having as much or as little data density as I desire

> Command lines[3], on the other hand, do it natively if not intuitively.

Which brings up few interesting observations:

* Terminal vs graphical is almost completely orthogonal and an implementation detail. There are plenty of WIMPy style[4] TUI/ncurses applications, and also various forms of commandlines/repls that are natively graphical.

* I'd argue that there are few other categories that are less often recognized, namely "keystroke-driven" and "web style" (nomenclature is mine and not fully though out).

* The characteristic attributes of "keystroke-driven" is full interactivity (in contrast to command-line) but where typically there are no (or minimal) interactive elements on-screen. Some examples off my head would be VisiData[5], tdo[6], fman[7]. These types of applications tend to have very high data densities (what the commenter above desired)

* "Web style" applications is more ambiguous (and debatable) category, but I think there is a distinction to be made between more traditional relatively rigid widget and form based UIs and the more free-form (and usually more heavily stylized) "modern" UIs that have been heavily inspired by (and/or implemented with) web design and technology.

* There is currently fairly big chasm between graphical applications and TUI applications, which I believe to be because there isn't really established libraries that would be effectively "graphical ncurses". I do believe that it is a chasm that deserves to be bridged. There is no real reason why developers wanting to create small, fast, powerful tools should be constrained to archaic character cell grid by default.

[1] https://news.ycombinator.com/item?id=17195618 [2] https://news.ycombinator.com/item?id=17196031 [3] editors note: Considering the context, I suspect inetknght meant terminal applications here when they wrote "Command lines" [4] https://en.wikipedia.org/wiki/WIMP_(computing) [5] http://visidata.org/ | Demo: https://www.youtube.com/watch?v=N1CBDTgGtOU [6] https://mstijak.github.io/tdo/ [7] https://fman.io/

in retrospect I probably should have made this a blog post instead of shoehorning it as a comment here, but oh well.

⬐ crazyloglad

Author here - do write it up as a blog post, the discussion should be had and a bit more systematically if you don't mind me saying so, and I have quite a lot of things more to chime in with.

Another relevant point for thought that was provided to me as a response to the original article is the design and thoughts behind https://www.eblong.com/zarf/glk - in hindsight it is more than reasonable that the interactive-fiction crowd had been working in the problem space but not attempting to generalize it.

Vis a vis animations, those can still serve a certain point of they are set up for cognition rather than mere aesthetics. A simple example is switching workspaces. Some of the people I worked with refrained from having tag-sets, or tiling with workspace switching and kept a surprising amount of windows around yet knowing about compartmentalising into workspaces. The argument was a problem with finding 'where all the windows went' on a switch, and that feeling, anecdotally, went away when the workspace switching animation indicated a direction, left-to-right or right-to-left depending on if the workspace index went up or down. That is the mental grouping was not in 'specific windows -> workspace number or identifier' but a spatial place, even if not visible.

While I agree between the gui/tui chasm (and often try to use text-dominant or similar phrasing - though in my head there is no real distinction between graphical and text - considering rendering emoji from a font is equivalent or darn well close to blitting tiles from a tilemap), I consider it more of a gammut where you sortof have to draw a line in the sand. The glyph+attributes in cell addressing model is much less complex than 'per pixel' even when accounting for BiDi/shaping/ligatures. Just don't mix the two which is the norm now, drawing a popup with text characters when it could've been... a friggin popup is a tragedy, and not even a good one.

That said, and this touches upon the bigger idea partly hidden behind the many lines - there is a need for what could've been the 'gestalt' of the mobile "app" idea had it not drifted towards such an opportunistic, exploitative and dark-pattern filled anti-user clusterfuck, and that is the notion of a 'one thing, and well' graphical applications. I think we need both, one biased towards the graphical, one towards the textual and that they can come from the same process, just that some kind of system enforced barriers are in place so the meta-ui (window manager) doesn't degenerate into a hierarchy of nested WMs fighting to block one another's view of the system (browser to window manager to tmux to terminal to shell to virtual terminals) sortof all fight to be the same underspecified 'thing'.

I document more than I disseminate, and the article belongs more in the former category, but for what I'm sortof feeling are concerns you'd enjoy in the context of the TUI discussion are two coming articles (monitor the rss, I rarely touch the browser) "rethinking desktop configuration: my UI is a filesystem" and "leveraging the display server API to bootstrap better debugging" (titles being subject to change).

⬐ aidos

I almost didn't bother because it was a video of someone messing with a csv file, but I'm glad I took a few minutes out to watch that.

For everyone who hasn't watched it yet, from what I can tell, this is a clip of someone using a terminal tool they've made for hacking around with tablular data. It looks awesome for exploring data in a partially visual way, driven by commands. I'll often end up doing a bunch of stuff like he demonstrates directly in python — pull out the data, filter, find patterns, calculate new columns. You could probably do much of that with Pandas, but this definitely looks like it would be worth a play.

⬐ miles

Thanks for your comment - I wouldn't have clicked on the video link otherwise, and it was well worth it.

For others who usually skip videos, here is a link which skips the intro, getting your time investment to under 4 minutes:

https://www.youtube.com/watch?v=N1CBDTgGtOU&t=0m52s

⬐ SteveJS

Saul is the guy who uncovered the crossword plagiarism scandal http://fivethirtyeight.com/features/a-plagiarism-scandal-is-... VisiData is his vision of interactive data exploration at the text terminal.

⬐ stuartaxelowen

VisiData is indispensable for data engineering tasks. I've been `head -n 1000 mybigfile.csv | grep somerandomthing | etc | etc`ing for so long, it's ability to give you fast summaries and exploration is such a breath of fresh air!

⬐ TAForObvReasons

http://visidata.org/ is the page for the tool and https://github.com/saulpw/visidata is the repo

⬐ eikenberry

> removed support for .visidatarc in the current directory and via XDG; only $HOME/.visidatarc is used now;

That was an odd choice considering everything is moving the other way as a standard. I wonder what the authors reasoning behind this decision was.

⬐ derimagia

https://github.com/saulpw/visidata/issues/93 looks like it wasn't implemented correctly ti begin with. Sad that it wasn't fix though.

⬐ andor

Reducing complexity

https://github.com/saulpw/visidata/issues/93

⬐ None

None

⬐ dang

A thread about the product release is at https://news.ycombinator.com/item?id=16515310. We merged some comments from that thread into this one because this one is currently ranked higher. Also the demo seems to be communicating more about what the project does than the project page.

⬐ sgt

Apparently the guy took an entire year's sabbatical just to work on this tool. Pretty impressive stuff.

⬐ Glench

Going to plug my own research project which does some things like this in a GUI: https://www.youtube.com/watch?v=y1Ca5czOY7Q

⬐ diimdeep

You are cool http://glench.com/LaserSocks/

⬐ Glench

Thanks!

⬐ reallymental

Why on earth are blaring horns the first thing on your video?

It's an immediate 'close tab' motion for a lot of people.

Cut that part out and you'll have a lot more eyeballs on your product.

⬐ Glench

hahaha

⬐ codezero

If you watch the rest of the video I think you’ll find they fit well with their personality and the tone of the demo. I recommend the butts visualization as evidence. Sounds like he’s also looking for anyone willing to glend a hand in testing it out.

⬐ lowsenberg

Very neat, especially useful for peaking into (SQLite) databases and getting a quick overview of the contents. Thank you for your efforts!

⬐ est

I always read about how awesome VisCalc/Lotus1-2-3 is but never been able to use one.

How does VisiData compare to them?

⬐ ngould

Congrats, this is awesome!

⬐ flas9sd

since v0.96 it can take input directly from stdin. I very much appreciated that feature as now you can pipe output from mysql (--batch -e)/sqlite and the likes. Would be interesting to use it as a pager inside the mysql prompt.

⬐ _pmf_

Obligatory motivational video of Martin Shkreli using Excel: https://news.ycombinator.com/item?id=12448545

⬐ dedosk

How does this handle plotting huge dataset? E.g. 500000 points?

Is Windows version somewhat on the roadmap? E.g. For mingw-bash or cygwin?

⬐ simonhughes22

What does this give you that pandas lacks? Or is it more geared towards people that like to use the terminal?

⬐ azag0

I don't really see why those two should be compared? Pandas is a Python library for programmatic work with tabular data. Visidata gives you a live view of a file storing tabular data.

⬐ simonhughes22

Thanks for clarifying, that makes sense

⬐ ginsmar

Wonderful tool. Thank you!

⬐ scarnie

Brilliant work!

⬐ None

None

⬐ IshKebab

Pretty cool but ... why not use a GUI? It would make it 100 times easier to use. He even says at one point "this is all just in the terminal by the way" like that's a good thing!

⬐ llao

What they showed in the lightning demo is so much better done with fast keystrokes than moving a mouse around. I can't wait to try this tool, it will save me days of time per year.

⬐ IshKebab

I doubt it is faster if you include learning time. And anyway GUI programs have shortcuts.

⬐ zokier

GUI does not imply mouse-based.

⬐ Parcissons

because a gui crystallizes the project beneath. Either that or the gui is broken. Or it just slows everything down- because the gui needs a update for every release cycle.

And people avoid to submit commits to such a project, because oh dread- you have to update the gui too.

Its more time demanding to learn, but if you invest that time- you get a faster ui for this.

⬐ bsg75

Because I’m remoted into another machine in a terminal. Because I’m already coding in a terminal. Because if I want to use a GUI I can use Excel.

⬐ lucb1e

I am in favor of it being terminal-based, but imho you're giving all the wrong reasons.

If working on a remote machine were really an issue, you'd just mount the remote filesystem or something. That's a small convenience but not a real reason.

That you're already coding in a terminal, didn't disable alt-tab.

Spreadsheets (or to use the brand name, Excel) have a very different focus. You speak as if spreadsheets are just as good as, in this case, Visidata, but the whole point is that this is better. Else, why bother developing and using it? You can indeed just open up your favorite spreadsheet processor.

I think the real reason is that it's universal and convenient. You can use the multitasking features of your favorite shell, and it attracts a certain audience which indeed wants vim-style hotkeys and shortcuts. It also doesn't need a platform to create GUIs on top of, like Electron.

⬐ c22

There's room for both reasons. Sometimes when you're working with massive sets of tabular data you don't want to be doing it over a network mounted drive.

⬐ jsvine

I've been using VisiData for about nine months now, and have found it to be tremendously useful. These days, it's typically the first first tool I reach for when examining/exploring a new dataset.

In fact, I've been liking it so much that I wrote a tutorial for first-time users: https://jsvine.github.io/intro-to-visidata/

(Feedback very much welcome / appreciated.)

⬐ blunte

One of the challenges with powerful tools is learning them. This is why many new people avoid things like Emacs, Vim, etc.

Some hand-holding and guiding is very welcome... so thanks!

⬐ VectorLock

>(That convention, like several others in VisiData, is borrowed from the vim text editor.)

Thats all I needed to hear.

⬐ scarnie

Great work

⬐ lucb1e

This is really great in helping me understand what I can do with it and how to do it.

The lightning talk is a really great sales pitch, but it's unclear how to actually replicate it. Another video of his, a recommended one after the lightning talk, again loses me a little and goes very fast. I could pause and try to understand... but it's not just about how to do it, but also about knowing that I should be looking to do certain things in the first place.

Your tutorial both gives a nice overview of what features exist, and explains very clearly how to do things. Thanks for taking the time!