HN Theater @HNTheaterMonth

This article is more realistic than most ML posts, but it’s clear the author is not a practitioner.

> More handwriting data will make a hand-writing recognizer better, and more gas turbine data will also make a system that predicts failures in gas turbines better, but the one doesn't help with the other. Data isn’t fungible.

Transfer learning is one of the most interesting aress of machine learning. The focus is on taking learnings for one task, and applying them directly for another. More directly, Jeff Dean from Google had a fascinating talk about using these techniques to create a single super-model that combines learnings from thousands of tasks to accomplish new things quickly. [1]

[1] https://youtu.be/HcStlHGpjN8

⬐ Eridrus

Nothing we know how to do can transfer data from handwriting data to improve gas turbine failure detection or vice versa. ImageNet fine tuning isn't at all relevant here.

⬐ outworlder

> This article is more realistic than most ML posts, but it’s clear the author is not a practitioner.

No so fast.

Genuine question: how close is the research on 'transfer learning' to something that can be readily used to solve business problems today?

If you can't fire up tensorflow and the like and use it to solve a real problem, or if only the likes of Google are able to successfully apply it, then the author would be correct.

⬐ claytonjy

You _can_ fire up TF to solve real problems without being Google.

Transfer learning is _the_ way to do image classification for most kinds of images in 2018, and is covered heavily in most classes. In the fast.ai class, you use transfer learning in the very first lesson to build a dog/cat classifier. Takes less than an hour to get to 97+% accuracy with no prior knowledge of deep learning.

⬐ jacquesm

But, the transfer takes place on a network that has already been trained with lots of dogs and cats and has been taught to differentiate different kinds of dogs from lots of other objects and different kinds of cats from other objects.

Getting a useful dog/cat classifier out of something that has been trained to differentiate between different kinds of boats instead of different kinds of mammals would be closer to what the OP aimed at.

⬐ claytonjy

I agree that using an ImageNet-trained model to classify a new set of subclasses should be easy, and is. Subsequent lessons show how to adapt the same approach to distinguishing dog breeds (more specific), and for identifying types of terrain in satellite images, which bear much less resemblance to anything in ImageNet.

That last one sounds pretty similar to your second sentence. Given what we know about transfer learning and CNN's, if we had a massive boat dataset, I bet it could be re-purposed to do pretty well at cat/dog.

⬐ jacquesm

> Given what we know about transfer learning and CNN's, if we had a massive boat dataset, I bet it could be re-purposed to do pretty well at cat/dog.

That's worth proving / disproving.

⬐ mercutio2

It sounds like you’re saying that transfer learning is helpful for image classification, which seems like an uncontentious position.

Are you really arguing that you think transfer learning would be useful from handwriting models to turbine failure models?

Using techniques that are successful with image classification as an example and generalizing to other domains that don’t look much like imaging seems like a stretch to me.

But perhaps I’ve missed some more convincing examples of the state of the art in transfer learning.

⬐ claytonjy

That's a good point, as far as I know there's no examples of cross-domain learning. There's new work in NLP for cross-task transfer learning, but that's as close as it gets at the moment.

It's hard to imagine there's anything to learn from handwriting images that could apply to turbine failure; a much broader kind of multi-task model than anything well see for awhile.

⬐ Dzugaru

The argument is still false. You can very well get an advantage from vast amounts of data in similar domains. And more importantly you can have ML insights not possible without it. What if ImageNet was not open to the public? Would we get an AlexNet breakthrough?

⬐ brisance

It's not that the author is incorrect, it's just that the argument he used was a straw man. i.e. they are unrelated domains. However the metrics that are used to qualitatively measure the performance of each model's performance on their respective problem domain may be useful for research. For example, we know that deep neural nets are currently state-of-the-art for image recognition, and so if our problem involves a similar image recognition problem, we might be wise to start off with a neural net. We don't have to start from scratch; we can use "transfer learning" to get some base weights (bottleneck features) going and refine our model from there.

The point is, there is no panacea, no "ultimate algorithm" for any and every problem, and yet the author demands this of machine learning in that section of his writing.

FYI Jeff Dean gave a lecture at YC on AI last summer:

https://www.youtube.com/watch?v=HcStlHGpjN8

Jeff Dean talks about AutoML using RL and in the paper " Neural Architecture Search with Reinforcement Learning" it also talks about this.

Also it seems different from more traditional hyperparameter optimization because it makes novel cells. So the structure of the network isn't limited to our existing library of layers/cells.

https://arxiv.org/abs/1611.01578 https://youtu.be/HcStlHGpjN8?t=2073

⬐ nl

"Novel Cells" are combinations of existing operators.

It's entirely true that these are combinations that humans haven't (and probably wouldn't) come up with.

I don't want to underplay this. "It's similar to hyperparameter search" makes it sound like it isn't interesting or novel, which is untrue. I completely believe it is a revolutionary way to build software (so much so that I quit my job, raised funding and are working on a similar space of problems).

But it isn't doing something like inventing a new math operations similar to the other operators which humans put together to form cells/layers. It is rearranging and choosing those operators in new ways.

⬐ michaelgreen

Okay I see what you're saying and I completely agree.

⬐ nl

You maybe interested in their most recent paper and blog post from today: https://research.googleblog.com/2018/03/using-evolutionary-a...

⬐ michaelgreen

Wow thanks, reading it now (:

⬐ michaelangerman

I enjoyed this talk.