HN Theater @HNTheaterMonth

⬐ hncommenter13

This is obviously very impressive, and far better than I could ever do myself.

But I must say it bears more than a passing resemblence to the version below, first uploaded to YouTube in 2009. Including the specific appearance of the robot and the graphic design of its eyes: https://www.youtube.com/watch?v=0mWHVvKb1hM

Perhaps worth acknowledging that prior effort? (I have no relationship, just feels like that effort--with 2009 technology--deserves some credit.)

⬐ nik12795

Thanks! Actually had a similar comment on my video:

> Yes! I didn't realize this other robot existed until I was partially through the project and was doing more research. I loved their form factor so I used it for Jon Bot Jovi. IIRC, they struggled to beat TTFATF (they got 60-70%?) because their note detection software struggled to decipher long blocks of notes tightly together

⬐ jmfldn

This is awesome. I was heavily involved with designing the audio and gameplay for DJ Hero, the sequel to Guitar Hero. I'd love to see you do this for DJ Hero next!

⬐ Lammy

Off-topic, but I feel like one of the few people who really really loved* this game, so thank you! It was interesting to see companies like FSG and SEGA try to make a DJ game without getting sued by Konami, and I don’t know of any other games besides CRACKIN’DJ and DJHero that employ the track switching/fading mechanic.

[*] And I would have 1000/1000 achievement points except for the one mix (Beats & Pieces) I was never able to five-star on Expert difficulty :argh:

⬐ nik12795

Wow that’s sweet! Glad you liked it. What was the process like to design the audio/gameplay? Sounds like a great gig

⬐ jmfldn

Hey there, nice to meet some fans! Yeah it did really well critically and we shifted a lot of units but the marketing investment killed the profit as we spent on it like it was going to be the next Guitar Hero. It was always going to be more niche than that so this was its achilles heel. Activision loved what we did almost too much and killed the profitability. When I started to see Eminem and Jay Z turning up to promo events I thought "wow" / "oh crap how much did this cost us?!"

Working on it was amazing. My job was remixing music for a video game and turning it into gameplay so what was not to love! I prototyped some of the earliest mixes in the game so helped to get the project through it's earliest milestones. I met a lot of famous people and got to remix Daft Punk, DJ Shadow and a few other heroes of mine. My favorite moment was opening the Ableton project sent to us by Daft Punk containing their stems. The music nerd / fanboy in me couldn't believe I got to play around with these stems that very few people ever had access to. They basically don't work with anyone as a rule so the fact that they collaborated on our project was insane. We had their likenesses in the game and the big pyramid stage. Was surreal!

It was a great gig and I feel we produced a cult classic game that stood the test of time. The financials just didn't stack up.

⬐ vuciv1

Wow, thanks for sharing. Very cool story :)

You put together a lot of fun hours in my childhood

⬐ adrianh

Thanks for sharing this — that is really cool!

What are you up to now? Still in the music/tech space?

⬐ jmfldn

Actually not any more. I'm a software engineer these days in the energy sector so nowhere near as glamorous! Gaming industry is a young man's game so I decided to move on and scratch my programming itch instead! I didn't fancy being a game programmer.

Music is back to being a hobby again. Was just a 3-4 year thing that I went into it professionally.

⬐ nik12795

Wow man that's an awesome story. Sounds like it was quite the ride. Thanks for sharing!

⬐ GuB-42

DJ Hero looks much harder. On screen you need to follow the crossfader line, detect the notes and scratch arrows, all of them on top of each others. On the control side you need to control the crossfader, turntable and buttons, and the buttons move with the turntable. Plus the effect knob, but IIRC, it isn't required, but it is important for scoring. But since the Guitar Hero robot doesn't use the star power, I think it can be overlooked.

Maybe an easier target would be BeatMania IIDX. It is actually a ridiculously hard game when played by a human, but the screen is very clean and you just have to press 7 button and move the turntable back and forth (or use the auto-scratch if that's a problem).

⬐ jmfldn

Yeah DJ Hero would be insanely hard to do, the machine vision part would be tricker for all the reasons you suggested. Simulating the scratch dynamics from the peripheral would be tricky too since it's not just a binary switch as far as the game goes but has certain tolerances which you'd need to calibrate. Effects dial you could ignore but I guess it would be easy enough to send a signal emulating that.

Agree that Beatmania would be amazing as a challenge and could build on the Jon Bot Jovi design!

⬐ nik12795

Wasn't too familiar with DJ Hero. Took a look... dang. That would be an interesting and challenging project!

⬐ jugg1es

DJ hero ended up sitting in my closet. It just wasn't nearly as fun to play as the guitar was. The game itself was good but the hardware didn't do it for me.

⬐ Baeocystin

Random fan here: DJ Hero was my absolute favorite of that era of music games. You did a stellar job figuring out how to make it really feel like you were doing the thing, and not just hitting buttons in time. Thanks for a great experience. :)

⬐ kevinherron

What a trip... I also built a Guitar Hero "bot" some 13 years ago.

Mine was not mechanical like this. Instead I spliced into the guitar with a microcontroller that I controlled via serial port from my laptop. I wrote software that analyzed recordings of the note chart, saved them to a file, and could play them back over the serial port by telling my microcontroller program what buttons to press.

I don't have any pictures of the thing any more, but some of the YouTube videos live on: https://www.youtube.com/watch?v=n1YE8Wwo2rs

⬐ LocalH

I remember TomHannu

⬐ nik12795

Wow how crazy is that! That’s really great. Obviously I had the privilege to leverage modern tech with CV, but amazing that you were able to accomplish it 13 years ago!

⬐ kevinherron

Your version is so much cooler. It’s more like what I had in my head but wasn’t able to accomplish :)

⬐ nik12795

Thanks!! It was super challenging but it was a fun time and the payoff of it shredding was well worth it

⬐ blamazon

Great job on this video. You took an interesting concept, implemented a solution, and conveyed it to the audience in a concise way that was respectful of our time. I love to see folks flexing their creative brains in this way across software and hardware. Video editing and production too. Subscribed and rang the bell.

One nitpick on presentation--I am a youtube addict and the first thing I notice on any vlog-style video is mic quality. I don't have words to describe the audiology of it all but I instantly know when someone sounds 'pro' vs. 'amateur'. This video and the others you've posted recently, have narration that sounds 'amateur' to my youtube addled mind. I don't know anything about microphones and stuff, but if you can tackle that I think it will noticeably elevate your videos! Cheers.

⬐ nik12795

Thanks for taking the time to write that. It means a lot to me and I'm really happy you're enjoying the videos.

Good call on the microphone. I'm still testing some things out so will keep that in mind as I produce the next few videos.

Thanks for subscribing and looking forward to hearing what you think of the next video!

⬐ maratc

Absolutely amazing. I probably spent more hours playing Rock Band than all other games combined.

One thing was a bit unclear: is Raspberry Pi doing the blob detection, or is it running on the Macbook?

Also, why only 97%?

⬐ stefanmichael

a couple of nuances make the remaining 3% a bit more difficult to get, two that I noticed is that the bot is bad at holding sustain notes and also tends to make mistakes during long hammer-on sections because it hits two notes at the same time instead of realizing that they are individual notes.

the second one is probably easy to fix by spreading the notes out a bit (hyperspeed mode) but the first one is a bit more annoying to program, probably

⬐ maratc

Not holding sustain is not a failure per se in the game. If you've hit the note but did not sustain it, you just get less points.

⬐ nik12795

Agreed the sustained notes would be a bit of a programming annoyance, but if I had spent more time, I think I could have cracked it. I did get some MVP working for it but it was throwing some other things off, so I abandoned it in an effort to maximize accuracy.

⬐ maratc

I think it might be easy implementing sustain on any and all notes until the next note needs to be hit. So instead of press-release-wait you do press-wait-release.

⬐ nik12795

The Raspberry Pi is doing the blob detection!

In terms of 97% and why not 100%:

> There wasn't any limitation from the mechanical perspective. The AI did a nearly perfect job of recognizing notes. I don't know exactly why it missed some notes, but my hypothesis is that it was the speed of notes in quick succession. If I do a follow up video with Jon Bot Jovi, I'll dig into it more!

⬐ qwertox

Code on screen at https://youtu.be/htk6eXxpSNA?t=23

`if y < prevValue or prevValue is None:`

should be

`if prevValue is None or y < prevValue:`

because the first one will make a comparison for `y < None`, while the second one won't. `y < None` will always return `False` while `prevValue is None` will then always return `True`, making the entire statement `True` if `prevValue` is `None`.

It is not a bug but it's something which I would correct. In the second one `y < prevValue` would not get evaluated if `prevValue` is `None` and it's not really clear what `y < None` means.

--

Great work which reminded me of the Ace of Spades-Robots. It also shows how insanely good humans can be at playing guitars.

⬐ Yajirobe

that's a condescending comment

⬐ qwertox

My apologies to nik12795 if he felt that way, it certainly wasn't meant to be condescending at all.

It's really impressive what he archived there, not only the image recognition, considering the high speed of certain parts of the video, but also the hardware implementation.

It was my mistake to only focus so much on the code-thing. Thanks for pointing it out.

⬐ nik12795

Appreciate you looking through the code and offering that suggestion. I'll take a closer look ;)

Also was not aware of the Ace of Spades robots. Just watched the video and it's epic!

⬐ kregasaurusrex

Very cool mixture of software and hardware for your project! I've been wanting to build a Go-playing robot for the longest time, hoping the cost of 6DoF robotic arms comes down in the near future. Do you have future tweaks in mind for improving your score, like increasing the signal of the notes when they're first starting to appear on the screen?

⬐ nik12795

A Go-playing robot would be sweet!

I think the biggest improvement for score would be implementing logic to hold down long notes and then implement star power at the ideal time. That would make a dramatic difference!

⬐ jbverschoor

One minor nit pick: It only strums one direction

⬐ ziml77

That's how I used to play. For some reason strumming in both directions really threw my timing off. Damn did that hurt my arm for fast sections though (if it got too painful I would change to strumming both ways simply because the alternative of slowing down would end up with me missing more notes).

⬐ jugg1es

I'm not sure it is physically possible for a human to strum only one direction on some of those songs on 'expert'. Our muscles just don't twitch that fast more than a few times in a row.

⬐ kodt

Also it isn’t necessary to strum for every note in Guitar Hero.

⬐ foobar33333

Yes but if you miss a single note you end up with a long chain of missed notes. The hammer ons are only really useful for the really fast sections where you can't physically strum fast enough for that section.

⬐ nik12795

I did think of ways to have alternative strumming, but there's something hilarious about it just mashing the strum button as fast as it can. And since the solenoid was able to handle it, I kept it as is :)

⬐ frankbreetz

The project is obviously super awesome. I am incapable of such a feat. What worries me is that if the AI CV algorithioms can't catch all the guitar chords, how are they ever going to see all the people crossing the street? 97% is pretty bad when it comes to identifying people in the crosswalk

⬐ Robotbeat

I really think sensor fusion is how we get there, like LIDAR or imaging radar on top of a really, REALLY good multispectral machine vision system.

It’s similarly disconcerting how bad I am (and other humans) at spotting people crossing the street at night if they’re wearing dark clothes. I gotta imagine there are analogous corner cases for machine vision, and a sensor fusion approach should help solve that.

⬐ foobar33333

And now you see why self driving cars aren't single developer hobby projects released in to the wild.

Also you may be overestimating human vision. I doubt normal drivers have 97% accuracy spotting all hazards on the road. We get by mostly on the fact that we keep hazards off the road whenever possible.

⬐ phalangion

That is a fair concern, but also keep in mind that this is one guy’s project. And with even more effort and debugging, I’m sure he could get it to 100%.

⬐ Baeocystin

I love this kind of silly/fun stuff. Thanks for sharing your project with us, it's great.

⬐ nik12795

You’re welcome! Thanks for checking it out

⬐ brunojppb

This is amazing work man! Creative and fun work while learning cool tech.

Through The Fire And The Flames Is still in my playlist to this day and reminds me about how cool was to play my PS2 back then.

⬐ nik12795

Thanks a lot!! Yeah it's an epic song and is so nostalgic.

⬐ ziml77

I love that you actually made it press the buttons instead of wiring directly into the guitar. Get all that authentic clacking :) Looks really cool!

⬐ nik12795

Thanks!! Appreciate it

⬐ fny

Neat project, but where is the AI exactly? I doesn't look liked you used anything "AI" related from OpenCV.

⬐ eliasbagley

Once upon a time blob detection was considered "AI"

⬐ drivers99

For some reason I emotionally want it to play perfectly. I would probably take the note data from multiple runs and use it as a basis to learn/memorize what the notes should be (like a human). I guess I’m looking at it like a TAS (tool assisted speed run) and try to perfect the inputs, even manually if necessary.

⬐ njacobs5074

Have so many great memories of playing this game with my family.

What an absolutely fantastic project you made for yourself!

⬐ nik12795

Thanks a lot!! Really glad you enjoyed

⬐ dom96

Nice! Big Mark Rober vibes in the video, was it a major inspiration for how you put this together? :)

⬐ nik12795

Thanks! Yes I love Mark and his videos. Huge inspiration for sure :)

⬐ jugg1es

What a crazy project. I love that you went all the way with it using the actual guitar. I would think you could have just reverse engineered the guitar signals but then it wouldn't be a robot.

⬐ nik12795

Yes exactly! Thank you!!

⬐ coverband

This was great, thanks for the clear explanation and no-fluff presentation. On a different topic, based on your other video —- what was the result from your Shark Tank audition?

⬐ nik12795

Haha thanks for checking out the videos! That company ended up failing. I was almost on Shark Tank again for my company DirtByMail.com (we were talking with the producers of ABC), but didn’t make it on.

⬐ swyx

freaking love your persistence. please keep it up! you earned my subscribe.

⬐ nik12795

Will do, thanks!!

⬐ quickthrower2

Hi there I have a question - why relays and not transistors? I would have though relays would add to the latency, and might be why some notes were missed.

⬐ nik12795

Honestly, I'm pretty new to robotics and haven't had a chance to mess around with transistors. I was able to find some videos on solenoids and relays and that made it easy for me to get something up and running quickly.

The latency wasn't an issue though. Even on expert mode on TTFATF, I still had to program a delay to time everything up. But that's great to know for future projects if I need a super low-latency solution.

⬐ quickthrower2

I don’t have massive experience but about 25 years ago I purchased a unit using transistors to power large stepper motors (probably a similar style of load to a solenoid - inductive with quick on/off) and it worked well, and didn’t get too hot. But don’t ask me what transistor tech it used, or the name of the company selling it!

But if you don’t need the response time or longivity then relays are probably much simpler to set up so a good choice.

⬐ throw7

"The thing that makes this all work is artificial intelligence". How so? That was an interesting statement that I was keen on understanding.

⬐ frankbreetz

He use computer vision to get information. Computer vision is a subcategory of ai

⬐ lars_thomas

Absolutely is not. There was nothing AI in this, simple classical image processing tools only.

⬐ throw7

Interesting. I never realized computer scientists categorize "computer vision" as "artificial intelligence". Seems to me OP could have gotten 100% correctness from the "vision" input without going the "computer vision" route then.

⬐ clircle

What is your suggested alternative solution?

⬐ exdeve

This is so funny! Jon-bot-jovi rocks!

⬐ nik12795

Haha thanks! Yeah it’s pretty hilarious

⬐ StavrosK

I love this, especially the sound the solenoids make when hitting notes. Well done!

⬐ nik12795

Super satisfying right?!

⬐ StavrosK

SO satisfying! I'm going to make one for my keyboard and drive my wife insane.

⬐ nik12795

Lol she’ll love that

⬐ vuciv1

Great job. Very well presented.

Keep it ul!

⬐ nik12795

Thank you!!

⬐ sodality2

This is incredible, hope this video gets more traction! Subscribed.

⬐ nik12795

Thank you and I appreciate your subscription :) Hope to continue to entertain you in future videos!

⬐ aledalgrande

Interesting, was expecting RL, but computer vision works too!

⬐ nik12795

Ah yes. And it worked beautifully!

⬐ Zedronar

Nobody talking about Rocksmith here!? Really!?

⬐ anonu

very cool stuff. feels Mark Rober-y too which is great.

Whats the tech behind the image recognition?

⬐ nik12795

Thanks! The image recognition is done by OpenCV which I highly recommend checking out.

⬐ njkumarr

this is so dope

⬐ nik12795

I was never great at Guitar Hero and couldn't fathom how people would play Through The Fire And The Flames, so I made a robot that could do it for me.

The video explains some of the technical details but I'll highlight a few other key points.

- My first attempt at note recognition was looking for specific pixels on the screen. The thought was that I would be able to detect if a pixel changed color and, depending on the color, it would know which note it was. The game outputs at 30 fps and on faster songs, it would miss nearly 30% of notes.

- The note recognition was eventually built using computer vision from OpenCV and specifically blob detection. It took a fair amount of effort to hone in the perfect filtering to make note recognition was highly accurate.

- Implementing the solenoids threw me for a loop. I ended up burning through a lot of them since I'm new to robotics and didn't understand the relationship between power, volts, and current. At one point I was giving these solenoids 4x the power it was expecting. Needless to say, they burnt out quickly. I eventually talked with a robotics guy and he gave me a 30 min crash course which straightened me out.

- The whole project, from idea to the YouTube video I linked in the post, took 2 months. It probably cost nearly $1,000 for everything (raspberry pi, pvc, solenoids, AC/DC power converter, relay modules, paint, stickers, etc). That might seem high, but like I said, I was learning about volts/current and part of that learning process was destroying many solenoids (oops).

⬐ fbelzile

Very nice work! I noticed that if there were notes missed, they all seemed like the long hold ones.

Was that a limitation of the AI software or was that by design to keep the strumming part easier to make?

⬐ nik12795

There wasn't any limitation from the mechanical perspective. The AI did a nearly perfect job of recognizing notes. I don't know exactly why it missed some notes, but my hypothesis is that it was the speed of notes in quick succession. If I do a follow up video with Jon Bot Jovi, I'll dig into it more!

⬐ bshep

it might have been a mechanical issue on the guitar itself ( i remember a similar issue when i played the game, it would sometimes detect a double strum or miss it )

Anyway it looks awesome!

⬐ fbelzile

Nice, thanks for the response! The video was well produced too, congrats! :)

⬐ nik12795

Thanks! Appreciate you saying that :)

⬐ bredren

Ya I was glad you cut it to specific pointed explanations.

Maybe I missed it, but it looks like there are 3d printed parts.

They aren’t what makes jbj’s brain work, but they might be worth going over the models (even if crude) as part of a follow up video.

If you posted that to /r/3dprinting it would be another audience to pull in.

⬐ StavrosK

Great work! Though, wouldn't it be easier to just open the controller up and use transistors to close the contacts? Transistors are cheaper!

⬐ nik12795

Yes! But I don’t think that’d be as cool. I really wanted to replicate the motion of having to press the buttons :)

⬐ StavrosK

Definitely agree that this way is much cooler.

⬐ utopcell

you bring shame to the world of robots! there are humans that can achieve 100% notes. mere, humans!

⬐ nik12795

Hahaha

⬐ Robotbeat

Honestly, I think people underestimate how good humans are mechanically. Like, a human arm is a 7 degree of freedom arm capable of incredibly fast motion.

Some of the songs on Beat Saber, on Expert Plus, are effectively impossible for any existing 7 axis serial robot arms to actuate fast enough to equal the best human players if you strapped VR controllers to where the end effectors are. Maybe some sort of parallel robot (like a SCARA) robot could do it.

Two robot arms with 7 degrees of freedom (the extra degree to allow them to avoid colliding with each other), plus the camera has to also move to avoid obstacles. It'd be really tough to beat the best human players, from a mechanical point of view. You'd need custom robots.

https://www.youtube.com/watch?v=LLb7cLTrSCE

⬐ kortex

Not only that, but each arm can produce dozens of newton-meters of torque, with possibly sub-millimeter precision, and like you said, fast. Just building an arm with comparable specs is quite the challenge. You end up needing hydraulics because even the finest, most expensive transmissions with belts to move the motors closer to the body, still end up being too heavy to achieve human agility+strength.

I looked into building a robot arm which could cook breakfast. It's almost ludicrous just speccing out the torque requirements to pick up a skillet.

You end up having to pick harmonic drives and less speed but more stiffness, or hydraulics, which are lighter, but messy, less stiff (can be both good and bad) and less reproducible.

⬐ Robotbeat

Yup. I think people also underestimate how much Boston Dynamics benefitted from using high performance hydraulics for some of their more impressive robot demos (vs others that relied on more conventional electric robotic actuators).

⬐ layer8

Nice! If you're looking for a follow-up project, you could pick up on this guy's work and build a Dance Dance Revolution bot: https://www.youtube.com/watch?v=M4DvlAb4eA4 (...and then challenge Boston Dynamics to a DDR duel ;))

⬐ nik12795

Lol challenging Boston Dynamics would be awesome

⬐ qyi

> couldn't fathom how people would play Through The Fire And The Flames

Either the game is slow enough to reason about in real time (certainly, a sweep like 4-3-2-1-2-3-4 as seen in the video wouldn't trip up any seasoned player) or the player resorts to memorizing finger positions (2+3 [2nd button held down at the same time as 3rd], 3+4, 1+4, etc) as well as sequences (1,1,1, 2+3, 1,1,1, 2+3). Hopefully one can pause at a certain segment and redo it as much as needed (like with real instruments), instead of starting from the beginning of the song each time.

⬐ jsjohnst

Did you consider potentially not using solenoids at all? Underneath each of the GH buttons is a momentary switch, so theoretically you can remove all mechanical forms of potential failure by eliminating the switches and solenoids.

⬐ LocalH

>The game outputs at 30 fps and on faster songs, it would miss nearly 30% of notes.

That 30fps is interlaced though, so there is 60Hz motion. All of the PS2 GH/RB games I know run at 60fps internally.

⬐ nik12795

I don't understand exactly what you mean. There were only 30 frames coming in per second that I could parse. Do you think there might be a way to get 60fps from the PS2 and parse that in the OpenCV code?

⬐ toast0

What's your video setup? A look around says Guitar Hero 2 will do 480p with component cables. I'd assume it puts out a new frame each time, but it could be 480p with doubled frames.

⬐ LocalH

Yeah that's probably the easiest and most latency-free way. It's absolutely 60fps of unique frames, especially with GH2. GH1 can also be hacked for 480p, even though the option isn't present in the normal game.

⬐ hinoki

Very nice!

For debugging missed notes, do you replay the video to the bot? You could capture clips by detecting when the score multiplier drops.

⬐ freedomben

Nice work man, and posting it to HN was a great idea to start gaining some subscribers ;-)

Giving away my PS2 and Guitar Hero is one of the only things I regret by the way. If someone created a Guitar Hero that ran on Linux I would gladly pay.

⬐ geococcyxc

There's Frets on Fire: http://fretsonfire.sourceforge.net/

⬐ Gamemaster1379

Phase Shift is by far the most comprehensive "knock off" Guitar Hero. Blows Frets on Fire and Clone Hero out of the water.

Netplay, multi format support. Etc. All previous game charts are available in formats compatible with Phase Shift.

⬐ ghusbands

It seems to be abandoned and unavailable. They switched over to Early Access on Steam in 2019 or so, but it's now not possible to buy it there, and there's been no sign of the devs for a long time, now.

⬐ LocalH

Clone Hero has netplay in the public test builds and wider format support than FoF (CH supports .chart format and afaik is the only significant 5-lane guitar/band game to do so).

Phase Shift also unfortunately plays like complete ass, especially on guitar.

The entire Guitar Hero community has pretty much shifted to Clone Hero, with a few diehard fans who still mod GH2/RB

⬐ grecy

I second this comment, and if the OP is looking to grow his channel, I suggest watching a few from "Stuff Made Here".

His channel has skyrockted from a few thousand subs less than a year ago to a few million now, and growing VERY fast.

⬐ nik12795

Yes I'm a fan of his! Coincidentally, Stuff Made Here actually commented on this YouTube video. But I have been enjoying and learning from his channel.

⬐ LocalH

https://clonehero.net/