HN Theater @HNTheaterMonth

Have you seen 3Blue1Brown video about it?

https://www.youtube.com/watch?v=aircAruvnKk

⬐ PoignardAzur

Yeah, I watched the series a few years ago, and it's still an example of the same problem.

The videos give you a good vibe for what a NN network does, but they still stop at "if you can get the gradients using math wizardry, then you can train your network and do tons of cool stuff!"

Meanwhile, if I had to write a neural network trainer from the ground up (even a very slow CPU one), I have no idea how I'd do it.

Like, I get it! A neural network is a bunch of vector<float>! You change the weights in a way that is similar to finding the low point of a slope! Yes, every damn ML video uses the same metaphors! My question is, how do I actually change the values of the goddamn floats?!

I dunno. Maybe I need to watch the videos again and something will click.

EDIT: Okay, so the last video in that series is way closer to what I was looking for.

⬐ hallgrim

I can recommend Andrew Trask’s tutorials and even his book on this. The (e-)book doesn’t go too in-depth with more complex NN applications, but you start out implementing your own NN from scratch using only numpy, which I thought was helpful to get into the topic.

Here's a 4-part video series by 3blue1brown : https://www.youtube.com/watch?v=aircAruvnKk Explains the big picture in an intuitive way

If you want an extensive answer, I recommend the neural network playlist on 3Blue1Brown: https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQ...

For a quick answer, watch this segment: https://www.youtube.com/watch?v=IHZwWFHWa-w&feature=youtu.be...

I can partly understand some this based on an EE101-level control theory, a High School level model of how neurons work, and 3blue1brown's intro to neural nets[1]. However, I have a strong personal interest in developing a much deeper understanding of dopamine neurons and their role in Executive Function.

Can anyone recommend a good curriculum which can take a random web developer from "Knows what a myelinated axon, a sigmoid function, and a feedback loop are" to having a solid enough background to dive into the research on this?

[1] https://www.youtube.com/watch?v=aircAruvnKk

⬐ scribu

Well, there's a major called Behavioral Neuroscience [1], which sounds like what you're after.

Or you could do a whole undergraduate program just on neuroanatomy. [2]

There's also this subfield called Cognitive Neuroscience [3]

(I'm not an expert either.)

[1] https://psychology.nova.edu/undergraduate/behavioral-neurosc...

[2] https://neuro.ucr.edu/courses/nrsc200a

[3] https://en.wikipedia.org/wiki/Cognitive_neuroscience

⬐ shmageggy

Others mentioned the neural side, but I think that's ancillary to the main idea of the paper which is about RL. David Silver's lectures on YouTube are excellent: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html

⬐ westoncb

Kind of an odd title and book, but I think it may do a good job of what you're looking for:

"Principles of Neural Design": https://www.amazon.com/Principles-Neural-Design-MIT-Press/dp...

⬐ afarrell

That looks like exactly the sort of thing I'm after.

3blue1brown videos are probably the best equivalent. Maybe a little basic? But really, all his math videos are just ridiculously good quality.

https://www.youtube.com/watch?v=aircAruvnKk

⬐ sci_c0

Key source material behind the video:

http://neuralnetworksanddeeplearning.com/index.html

The Stanford course is pretty good. https://youtu.be/vT1JzLTH4G4

For something quicker & less applied, maybe the 3b1b series on neural networks: https://youtu.be/aircAruvnKk

Yes, while I haven't done anything with ML myself, I heard via the 3Blue1Brown video on Deep Learning[0] that the sigmoid function isn't really that used anymore. But I figured allowing that being able to tweak the dynamic range could make it a good fit for the rescaling and recentering approach.

You know what, I'll just go ahead and post a link to this article on the Unum google group, perhaps someone there can add some thoughts[1].

[0] https://www.youtube.com/watch?v=aircAruvnKk&t=17m

[1]https://groups.google.com/forum/#!topic/unum-computing/RHrQU...

⬐ dnautics

Sigmoid is indeed hardly used as layer-to-layer transfer functions.

But I gurantee you > 20% of the world is activating sigmoid functions in ML apps every day.

⬐ vanderZwan

Ah, thank you for that correction. Guess the video was accidentally misleading (they technically didn't say anything about other ML approaches, but overgeneralising like I did isn't that big of a leap)

https://youtu.be/aircAruvnKk

3Blue1Brown’s playlist on neural networks. Like other 3Blue1Brown videos, the visualisations and explanations of the math/concepts are so amazing, they give you the intuition behind the neural networks naturally.

I'm in the same boat. For long time, I was interested in AI but at the same time intimidated by math. I'm relatively comfortable with discrete mathematics and classical algorithms and at the same time calculus and linear algebra is completely foreign to me. Also, I do not accept way to learn ML without good understanding of core principles behind it. So math is a must.

A few months ago, I stumbled upon very amazing YouTube Channel 3Blue1Brown which explains math in very accessible way and at the same time I got feeling that I finally started understanding core ideas behind linear algebra and calculus.

Just recently he published 4 videos about deep neural networks:

https://www.youtube.com/watch?v=aircAruvnKk

https://www.youtube.com/watch?v=IHZwWFHWa-w

https://www.youtube.com/watch?v=Ilg3gGewQ5U

https://www.youtube.com/watch?v=tIeHLnjs5U8

So my fear of ML was gone away and I'm very excited to explore whole new world for neural networks and other things like support vector machines etc

⬐ bootcat

I have also used Mathematical monk, who was simple and good in introducing basic concepts and tools related to ML. https://www.youtube.com/user/mathematicalmonk

⬐ darethas

I also recommend taking up computer graphics for honing your skills in linear algebra. Graphics are essentially applied linear algebra.

⬐ markatkinson

I came here to write a similar comment. Really make sure to watch the playlists in the correct order on the above YouTube channel.

⬐ kregasaurusrex

Having watched the third one out of sequence, seeing the first two and then watching the third again helped me get a good understanding of the fundamentals. 3blue1brown as a narrator does a excellent job of allowing a rather tricky subject be more approachable, and inspired me to buy a course to allow a deeper dive into the math behind ML+NNs.

⬐ lerax

Nice to some one pointing the fundamentals. A good understanding about probabilist models is good too. After getting into too the basic math knowledge, I suggest this: https://classroom.udacity.com/courses/ud730

⬐ sn9

Regarding linear algebra, I highly recommend Klein's Coding the Matrix which uses Python to teach linear algebra.

I believe it was developed for Brown's linear algebra course for CS undergrads.

⬐ sova

Hi! That's wonderful. What' a support vector machine used for?

⬐ pletnes

Classification and regression. Given examples, predict labels or values for new data. SVM used to be more «hot» than neural nets and are still very useful.

⬐ hwu2whag

Wow thanks for this resource!

⬐ skytreader

Worth noting that 3Blue1Brown also did a series on linear algebra which is eye-opening to say the least. Playlist at:

https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2x...

Even if you think you grok matrices, have a go at the first few videos of that playlist, if just for the visualization. It really helped me see what matrices (and operations on matrices) represent!

⬐ tmaly

I just watched the first video. Thank for sharing.

⬐ _xhok

3Blue1Brown is a treasure. The production value is excellent, and he's great at taking seemingly uninteresting ideas and painting a beautiful picture to connect them in twenty minutes. I used to go through a video before falling asleep each night.

This was very timely for me and for anyone else learning, here are the first few videos of the series:

https://www.youtube.com/watch?v=aircAruvnKk

https://www.youtube.com/watch?v=IHZwWFHWa-w

https://www.youtube.com/watch?v=Ilg3gGewQ5U (Original video)

https://www.youtube.com/watch?v=tIeHLnjs5U8

3blue1brown is currently doing a terrific series on how neural networks work, which nicely compliments this blog post. https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQ...

⬐ egwynn

Man 3B1B is fantastic. I don’t think I’ve ever seen so much great math explained so clearly and beautifully as in his videos. Keep up the good work!

⬐ zardo

He did a lot of the lectures for this course, with 3d visuals of course.

https://www.khanacademy.org/math/multivariable-calculus

⬐ intjk

It's him that made me understand linear algebra. College? No fucking clue what a matrix was. Now? a clear, intuitive understanding. I'm still struggling with concepts like trace, but he provided a base from which to I'm able to climb myself.

⬐ sillysaurus3

A trick for visualizing higher dimensions: https://www.youtube.com/watch?v=zwAD6dRSVyI

Only 383k subscribers. Hmm. Remember to subscribe, comment, and smash that like button.

Real Engineering is another great channel. And of course Veritasium and Numberphile.

⬐ dahart

That's a cool video, I'm subscribing. It's surprising that the embedded sphere has an unbounded radius. He didn't mention in the video that this problem is due to, and is a sort of dual or inverse to the fact that the volume of an N-dimensional sphere goes to zero as N goes to infinity. That hurt my head a little the first time I learned about it!

⬐ SemiTom

Interesting perspective on speed up neural networks https://semiengineering.com/speeding-up-neural-networks/

⬐ v3gas

Linear regression.

⬐ desertrider12

Unless you use a nonlinear activation function like the sigmoid, as shown in the video.

⬐ uoaei

It's a high-dimensional correlation machine. In other words, it's an attempt to learn "how to recognize patterns" by learning how to represent each one as orthogonally as possible to each other one. This happens at each layer, and how many layers you need depends on how "mixed up" the transformed space is with respect to the appropriate labels following each linear transformation. Once they are suitably linearly separable following the feedforward pass through the network, you need one more layer to identify how the pattern maps to the output space.

Another way to think of it is that each layer learns a maximally efficient compression scheme for translating the data at the input layer to that at the output layer. Each layer learns a high-dimensional representation of the output that uses minimum bits for maximum information reconstruction capacity. There was a great talk given recently by Naftali Tishby where he explains this in great detail.[1]

Having the math is great to know how it works on a granular level. I've found that also explaining it in such holistic terms serves a great purpose by fitting "linear algebra + calculus" into an understanding of NNs that is greater than the sum of their parts.

[1] https://www.youtube.com/watch?v=bLqJHjXihK8