HN Theater @HNTheaterMonth

For an alternative perspective, allow me to present some classic talks from Raymond Hettinger.

First, some generic tips on how to write Python code that's pleasant to work with: https://youtu.be/OSGv2VnC0go

And then this one digs more specifically into how not to write Python like it's Java, starting at 12:40 (with talking about why leading up to that): https://youtu.be/wf-BqAjZb8M

Raymond Hettinger has a good section on this, tracing it back Knuth's discourse on structured programming in the face of 'goto'. He suggests calling the keyword 'nobreak', rather than 'else'.

Inside any loop are a conditional and a jump. In pseudocode:

    if not <loop end condition> then
        <do loop things>
        <jump to top of loop>
    else //loop is done
        <do things in the case that the loop has terminated naturally>

If we 'break' in the body of the loop for some reason, we will never hit the 'else' in this chunk of code. As Mr. Hettinger explains, this is obvious to anyone reading Knuth or coming from a 'goto' style of control flow. This is not an insult, but an observation. (Un)Fortunately, structured programming is the absolute norm now, and we learn looping constructs directly, rather than learning 'goto' and then building to looping constructs. Especially in a language with rich iteration protocols, such as Python, it is very much unapparent that the looping constructs are fancy wrappers around 'goto'.

Link to the talk: https://youtu.be/OSGv2VnC0go?t=948

These are a few videos which present Pythonic style. They do not form a proper curriculum, but they are example-heavy and give a good insight to a very Pythonic mind.

1. Beyond PEP 8 https://www.youtube.com/watch?v=wf-BqAjZb8M

2. Transforming Code into Beautiful, Idiomatic Python https://www.youtube.com/watch?v=OSGv2VnC0go

Raymond Hettinger's talks are great. I know it's not a succint blog post, but they are out there are watching a video is very approachable.

Maybe https://www.youtube.com/watch?v=T-TwcmT6Rcw (Dataclasses! We could cheekily say Python gets better at something Rust does - dataclasses makes Python better at records.)

And https://www.youtube.com/watch?v=S_ipdVNSFlo

This talk (2013): https://www.youtube.com/watch?v=OSGv2VnC0go Unfortunately this video is now a bit out of touch with modern Python.

Another (2013) classic: https://www.youtube.com/watch?v=HTLu2DFOdTg it is very well known, and it has the very memorable advice: what's a class that only has one method? That should be a function!

⬐ fnord123

> Dataclasses! We could cheekily say Python gets better at something Rust does - dataclasses makes Python better at records

It's not clear if you mean python got better at storing records than previous python or if it got better at storing records than rust.

To be clear, dataclasses make python better at storing records than what was previously available in python but still not as good as Rust (serde, immutable defaults, no perf penalty with immutability).

⬐ dec0dedab0de

Another by Raymond Hettinger that is fantastic.

Beyond PEP 8 -- Best practices for beautiful intelligible code - PyCon 2015

https://www.youtube.com/watch?v=wf-BqAjZb8M

⬐ kzrdude

This is the best one :)

Your mention of simple and clean code makes me think of the excellent talk "Transforming Code into Beautiful, Idiomatic Python".

Although it relates to Python programming, if the ideas and principles (and thoughtfulness and pace) from it could be applied to much more written code, then (I think) we as an industry and all our users would be in a better place.

[1] - https://www.youtube.com/watch?v=OSGv2VnC0go

⬐ Chris_Newton

Possibly worth noting that the presentation there is a few years old now and appears to be using Python 2 for the examples, so some of the code wouldn’t be exactly the same in Python 3 even if the ideas still make sense.

With that caveat, watching almost anything that Raymond Hettinger presents is positively correlated with improving programmer skill.

⬐ jdormit

Great presentation! I didn't know about izip or ChainMap.

⬐ bspammer

FYI: izip is simply zip in Python 3, it's an iterator by default just like xrange -> range.

Here’s why (Hettinger):

> In 1991, “else” was the obvious choice because of the traditional way compilers implemented while-loops: pastebin.com/tY35CTJ4

That is, “if I haven’t finished the loop, GOTO the body.”

He says if he had a time machine he could tell Guido in the future we’re all using structured programming so no one will find “else” intuitive; call it “nobreak” instead: https://m.youtube.com/watch?v=OSGv2VnC0go#

Guido says if he had a time machine he would not have included the feature at all: https://mail.python.org/pipermail/python-ideas/2009-October/...

This whole thing was Knuth’s idea, not Guido’s: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103...

⬐ evandijk70

If Guido would not have included the feature at all, why did he not remove it from Python 3? That would have been the perfect opportunity.

⬐ figgis

Python 3 had it's first release in 2008, that thread was from 2009. Probably didn't even really think about it before then. (the more you know)

⬐ qwerty456127

> He says if he had a time machine he could tell Guido in the future we’re all using structured programming

Sounds like if people were not already coding C, C++ and Pascal when Python was invented...

⬐ pmontra

> Guido says if he had a time machine he would not have included the feature at all

So deprecate it with a runtime warning and remove it after some years.

I wonder if there are some statistics about its usage. I guess it's very low.

⬐ noobermin

Why remove a feature that could be useful? That makes no sense.

Just because it isn't named in a way people find intuitive doesn't mean it isn't useful.

⬐ kevinavery

Unintuitive features that "could be useful" sounds like the antithesis of good design.

⬐ fileeditview

And it sounds like a common problem in the software industry.

⬐ coldtea

>Why remove a feature that could be useful? That makes no sense.

Cleaning up a language "makes no sense"?

Besides "could be useful" can be said for anything. Features should prove their usefulness (and in this case Guido regrets even adding it), we don't just keep them around in the off chance that they might be. That's hoarding.

⬐ Retra

You want a major version bump on very widely used software just to remove something that causes a minor confusion on the rare occasion it is used? In what world does that sound like a good idea?

It's one thing to make a change like this along with a bunch of other breaking changes, but there is no way this justifies breakage by itself.

⬐ coldtea

>You want a major version bump on very widely used software just to remove something that causes a minor confusion on the rare occasion it is used?

No, I want a deprecation warning, and an eventual removal when for 4.0 lands.

>In what world does that sound like a good idea?

In a world where we don't pile crap upon crap forever, like with POSIX, X-Windows, or C++.

⬐ jononor

There are reason those systems are still widespread in use today, strong compatibility is one of them.

⬐ coldtea

There are also reasons those systems have tons of accidental complexity that frustrate users and wastes endless workhours. Strong compatibility is one of them.

⬐ oblio

> Why remove a feature that could be useful? That makes no sense.

Everything has a cost. In maintenance, in documentation, in support, in limiting the design space or constraining further development for other reasons (performance, for example).

Nothing is truly free. Everything has to be balanced regarding costs and benefits.

⬐ nemoniac

Here's the first sentence of the spec of the Scheme programming language:

"Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary."

⬐ kazinator

That grand statement is made possible by banishing needed Scheme features into numerous SRFI documents, and into implementations.

⬐ goatlover

It's interesting how Lisp languages get criticized for lack of maintainable because of their power to let you essentially add new features, but then every popular modern language with the exception of Go piles on new features over time.

⬐ junke

Go is still relatively young.

⬐ nerdwaller

I find it hard to argue it’s (the else block) any more useful than just using the `else` block’s logic in the try block.

    try:
        import uvloop
        asyncio.set_event_loop_policy(uvloop.EventLoopPolicy)
    except ImportError:
        pass

Does the handler for if the try succeeds need to be in an else, separated from the success logic?

⬐ figgis

You can actually use else for try statements. Doesn't answer question but it's something I learned relatively recently and may help some others.

    try:
        import uvloop
        asyncio.set_event_loop_policy(uvloop.EventLoopPolicy)
    except ImportError:
        pass
    else:
        print("well that didn't work")

⬐ pmontra

I'm not sure about what the message in that print() refers to (else fires if the import succeeded) so here's another example

    $ python3
    >>> try:
          1/0
        except ZeroDivisionError:
          print("exception")
        else:
          print("surprise")
        
    exception
    >>> try:
          1/1
        except ZeroDivisionError:
          print("exception")
        else:
          print("normal")
        
    1.0
    normal

The choice of the name else is unfortunate because it's activated under the same circumstances of a then branch of an "if". Maybe its name should really be "then". That would also look like a .then of a JavaScript Promise, which would make sense nowadays and was obviously impossible to foresee back in 1991. However the name "else" makes sense if it refers to "except": it's what happens when no "except" fire. I still don't understand why it should be "else" for a "for" that succeeded: it should be a "then" there.

⬐ figgis

yeah you are right, it should be

print("well that's done")

⬐ saurik

In the case of a for loop, you are modeling "a for which did not break". The cases where you want that block are typically when you are looping through a list trying to find something, so "if I found it I broke, else do this default logic".

⬐ pmontra

That's the intended use case but it works also when there is no break

    $ python3
    >>> for i in [1,2,3]:
          print(i)
        else:
          print("completed")
        
    1
    2
    3
    completed

That should be a "then", but admittedly a very useless one.

I agree that for the intended use case "else" is a better choice. It's like the else of the if that failed to find 2 in this code:

    >>> for i in [1,2,3]:
          if i == 2:
            break
        else:
          print("we didn't find 2")
        

Still "else" doesn't hint much at what's going on, especially if the body of the loop is long. Maybe "nobreak" would make it immediately understandable to everybody? However in the best Python tradition I would make it very explicit, remove the feature from the language and use boolean flags. They are less compact but easier to understand than a feature that (probably) very few people know about.

    >>> we_found_2 = False # this is a valid assertion now 
    >>> for i in [1,2,3]:
          if i == 2:
            we_found_2 = True
            break
        if not we_found_2:
          print("we didn't find 2")

That's also the only way to do it if there are multiple break in the loop on different conditions.

⬐ pmontra

By the way, try also has an else clause that behaves like the one in if https://docs.python.org/3/reference/compound_stmts.html

⬐ nerdwaller

I guess I wasn’t clear, because my point is generally that moving the logic to an else block in any supported control structures (if, for, try, ...) doesn’t necessarily make it clearer or cleaner.

⬐ dragonwriter

It behaves more like the main branch of “if” unless you construe “try” as “try to produce an exception”; outside of “if”, Python’s “else” pretty consistently implies exactly the wrong picture of normal flow.

⬐ abecedarius

Generally you can't be sure that the second statement does not also raise ImportError, where you do not want to catch it.

⬐ tomtimtall

Because readability counts, and id at the core of Python.

⬐ some_account

Goto is also useful... See why now?

⬐ reificator

This comment contains more snark than content, but I'll reply anyway:

Goto is considered harmful, but it's important to recognize the context of that decision rather than parroting the idea. It's not an obvious idea and if the discussion had happened today with modern languages[0] instead it's not at all obvious the same conclusions would be reached.

This write-up does a pretty fair job of translating the original paper to something more modern without a lot of bias. http://david.tribble.com/text/goto.html

[0]: Modern languages of course learned from this paper, so that's not a fair comparison. In fact golang contains a goto, but it behaves well according to Dijkstra's concerns and occasionally results in more readable code. I would be shocked if Dijkstra took any issue with go's implementation.

⬐ jb1991

> Why remove a feature that could be useful? That makes no sense.

It makes sense in the way that a language which is strongly biased to readability and consistent expectations makes sense. It is also how a language which minimizes programmer error makes sense.

For languages that refuse to remove anything, and thus suffer from unnecessary challenges, see C++.

⬐ larschdk

Several things have been removed from C++, including auto_ptr, exception specifications, unary/binary_functions, trigraphs.

⬐ hellofunk

Most legacy cpp features and language semantics will never be removed, despite what many prominent voices in the community think.

⬐ jlebar

I think the extreme difficulty and indeed controversy that the committee stared down when removing trigraphs, which are a useless feature that the vast majority of C++ programmers will never once encounter, speaks well to GP's point that C++ is very conservative in how it removes features. Maybe not JS-level careful. But still.

⬐ anyzen

Of course, JS has a problem in this regard that while it looks like a language, it is also a (de facto) assembler for the browsers. So disabling anything in JS will break lots of sites which will never be fixed.

However as far as programming is concerned, many things have been deprecated through transpilers and linters (for example "==" throws a warning in ESlint).

⬐ mayankkaizen

Actually I wouldn't like it deprecated, not because I use it often or like it very much. In fact, I myself found this feature counterintuitive at first brush.

Deprecating any feature without any strong reason makes me irritateted and I don't find any strong reason for doing so.

⬐ karmakaze

The thing to do here is deprecate 'for-else' in favor of 'for-default' which is much clearer like in a case block.

'for-else' never needs to be removed but the code cleanup will happen over time.

⬐ gvx

I whipped up something to test that: https://gist.github.com/gvx/667efcd263b18d313801c1b1ffa41509

Running that on my personal projects folder gives me the following result:

  for elses 60
  for loops 1815
  # modules 598
  # skipped 2

⬐ pmontra

Thanks. I run your program on a couple of projects of a customer. I'm not the only developer.

    for elses 0
    for loops 428
    # modules 673
    # skipped 26

    for elses 99
    for loops 6058
    # modules 2016
    # skipped 286

⬐ boomlinde

Runtime warning on what channel without breaking compatibility?

⬐ verandaguy_alt

You could print the warning out to STDERR when the Python runtime finishes parsing and before it executes your script.

This will likely break some people's workflows, but honestly, it's the de-facto way of notifying users of a deprecation in many Python frameworks (Gunicorn did this recently with gunicorn-paster vs gunicorn --paste).

They could also include this in a "What's new in Python (4?)" since this is unlikely to be introduced in a minor version bump.

⬐ boomlinde

I'm not categorically against deprecation warnings. Isn't gunicorn a command line tool? I can respect such warnings in a tool or even sometimes a library, but not in a language interpreter. The user should as closely as possible be in full control of the streams in those cases, and any standard error output that isn't an exception (i.e. either deliberate or resulting from a programmer error) should stay the hell off.

The proper way to handle this is of course to introduce a new semver major version. No need for deprecation warnings because incompatibility is expected.

⬐ ProblemFactory

> So deprecate it with a runtime warning and remove it after some years.

After the Python3 brouhaha, I think people are very cautious about making backwards incompatible changes to the language, even with a deprecation warning.

If it was done together with the rest of Python3 changes then it could have been approved, but not any more.

As Guido suggests, adding the warning to pylint and other style checkers, but keeping compatibility is much easier.

I've never heard of Raymond Hettinger before but I'm currently watching one of his videos [1] and it's amazing. Thanks for sharing your list, I'll definitely look up some more of these people.

[1] https://www.youtube.com/watch?v=OSGv2VnC0go

⬐ Lordarminius

All I can say is wow! Any equivalent videos of ruby programmers ? Just found this by Sandi Metz for anyone interested: https://www.youtube.com/watch?v=PJjHfa5yxlU

good list!

links for Raymond Hettinger ones

* https://gist.github.com/JeffPaine/6213790

* https://www.youtube.com/watch?v=OSGv2VnC0go

Check out Raymond Hettinger's talks on YouTube. Here's one to get you started: https://youtu.be/OSGv2VnC0go

⬐ alfiedotwtf

Thanks, adding to my watch list!

⬐ vram22

Another good Python video is Ned Batchelder's "Loop like a Native". Has lots of examples of non-idiomatic vs. idiomatic ways of looping in Python. Google a bit before reading, there may be more than one video, or a blog post and a video. Check for the latest one etc.

Must watch: https://www.youtube.com/watch?v=OSGv2VnC0go

I believe that the list comprehension will be faster than filter, but as always, any time you replace readable code with unreadable code for performance reasons, you damn well better time it.

⬐ m_mueller

That's actually the talk I was thinking about, but I guess I forgot how exactly Raymond described generator expressions there. Thanks for linking it again!

⬐ viraptor

~4.5 times faster with comprehension.

    In [11]: timeit.timeit('''list(filter(lambda x : ('widgets' in x), mixed_widgets))[0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000)
    Out[11]: 2.6956532129988773

    In [12]: timeit.timeit('''[x for x in mixed_widgets if 'widgets' in x][0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000)
    Out[12]: 0.5911771030077944

But not generating the list at all is still going to be faster (with bigger gains for bigger data)

    In [13]: timeit.timeit('''next(x for x in mixed_widgets if 'widgets' in x)['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000)
    Out[13]: 0.3324074839911191

⬐ m_mueller

I guess my next question is then - why is anyone using filter/map instead of comprehensions or even generator expressions? Familiarity when coming from FP?

⬐ pmontra

Coming from Ruby it's easier to use map, because it's what Ruby's standard library offers.

However comprehensions are not that harder when one finally decides to understand how they work. Not as readable as map() IMHO. Example:

Ruby

    [1, 2, 3].map {|x| x*x} # object.method(args)

vs Python

    [x*x for x in [1, 2, 3]]

where we have the function first, then the definition of the variable, then the data. This is the opposite of the object.method OO notation and using a variable before defining it is not what we usually do. But it's almost the usual mathematical notation "for i in set do f(i)" with the function at the beginning.

Not a big deal.

About a problem raised in a comment of the post (which is from 2009): this is Guido (2009) about the lack of tail call optimization in Python http://neopythonic.blogspot.it/2009/04/tail-recursion-elimin...

⬐ viraptor

In my experience filter/map are used by people who just don't know about comprehensions, or are not used to having them available. It takes some time to start using them where properly.

⬐ clusmore

Well, for one I believe they pre-date comprehensions. Having them as functions is also occasionally useful for partial application, e.g.

  from functools import partial
  to_strings = partial(map, str)
  # vs
  def to_strings(seq):
      return (str(elem) for elem in seq)

⬐ EvilTerran

Or this:

    def to_strings(seq):
        return map(str, seq)

Generally, when the operation I'm applying to each element happens to already be a named function, I find "map(f, seq)" preferable to "(f(x) for x in seq)".

⬐ m_mueller

I can see it together with partial, yes, that's when it can become a bit cleaner. Another reason why I use map is when I want to use multiprocessing or multithreading (with IO heavy functions). But on a fine grained level of code I find it really hurts readability compared to comprehensions.

1) There's certainly not "one way" to iterate in Python[1], and they're not even consistently named (xrange is the iterator version of range, but izip is the iterator version of zip).

2) You can use Python in the browser[2][3][4 sort of].

1. https://youtu.be/OSGv2VnC0go?t=3m20s

2. http://transcrypt.org

3. http://pyjs.org

4. http://www.rapydscript.com

⬐ djsumdog

From "import this"

> There should be one-- and preferably only one --obvious way to do it.

Key words are preferably and obviously. Of course there are going to be multiple ways to do everything, but it'd be nice if the best practice is pretty obvious.

> Although that way may not be obvious at first unless you're Dutch.

If you were underwhelmed by this blog post have a look at:

Transforming code into Beautiful, Idiomatic Python by Raymond Hettinger at PyCon 2013

https://speakerdeck.com/pyconslides/transforming-code-into-b... and https://www.youtube.com/watch?v=OSGv2VnC0go&noredirect=1

Topics include: 'looping' with iterators to avoid creating new lists, dictionaries, named tuples and more

⬐ glitchdout

Sorry for the offtopic but why did you add `&noredirect=1` to the end of that youtube url?

⬐ shackenberg

sorry, was only an accident.

⬐ Vaskivo

And, if possible, see every video by Raymond Hettinger. He is a great teacher.

Sincerely, Transforming Code into Beautiful, Idiomatic Python – by Raymond Hettinger... http://youtu.be/OSGv2VnC0go

⬐ scep12

I was lucky to watch this video while first learning the language. Every beginner (coming from another language) should watch this to understand the idioms of Python.

⬐ __luca

Ya, me too. It's also a very funny talk. :)

Might be worth while looking at the video below for more Pythonic gems like this one.

https://www.youtube.com/watch?v=OSGv2VnC0go

Glad you posted this. I came to the comments to post the same thing[1] because thinking of it as "notfound" makes remembering (and comprehending it) easier. :)

[1] https://www.youtube.com/watch?v=OSGv2VnC0go#t=17m12s

⬐ agrover

Ah, I misremembered, "nobreak", not "notfound".

Check out Raymond Hettingers Transforming Code into Beautiful, Idiomatic Python talk on youtube [1].

Great talk on avoiding some of the common pitfalls new python developers step in. Exposes some nice language features.

[1]: https://www.youtube.com/watch?v=OSGv2VnC0go

⬐ gshubert17

And the slides are here:

https://speakerdeck.com/pyconslides/transforming-code-into-b...

Great suggestion. Just searched for him on youtube and a couple of excellent looking talks popped up. Halfway through and this one seems great so far: https://www.youtube.com/watch?v=OSGv2VnC0go

Really enjoy his style as well :)

⬐ RyanMcGreal

Slides: https://speakerdeck.com/pyconslides/transforming-code-into-b...