Player Piano Project (V2)

Github repos:

[Base] ESP32 Code, PCB files, CAD models: https://github.com/RCPilot1604/Player-Piano

[Webserver] Angular+NestJS webserver for controlling player piano [WORK IN PROGRESS]: https://github.com/RCPilot1604/Player-Piano-Server

Key changes from V1:

• Redesigned 3D printed solenoid mounts
• Attempted development of web application to control player piano

V2 wraps up the hardware aspect of the player piano, as I do not foresee any changes to the frame and mounting structure.

Hardware Changes – Finalized Solenoid Mounts

I modified the solenoid holders to now slide into the 2040 aluminium extruded profiles, thus preventing any unwanted warping of the mounts even under the force of multiple actuated solenoids. A small downside is that the mounts are more difficult to install and change out, as you have to slide out the entire 2040 profile to access even a single holder. Howevever, since replacement of solenoid mounts is unlikely, I consider this to be a worthy tradeoff.

Another great aspect of this improved mounting system is that it allows for on-the-fly repositioning of the mounts without compromising structural rigidity, since the mounts are free to translate laterally along the entire length of the keyboard. This helps with positioning the solenoids precisely over the keys and to account for slight deviations in the length of your individual keyboard (in my case, my Yamaha upright was slightly different from my CAD model).

Software – Web MIDI Player Interface (WORK IN PROGRESS):

Of far greater significance in V2 was the software side of things, specifically a total redesign of the way users interact with the player piano. Initially for V1, I was using the tried and tested method of Synthesia+MIDI connector via BLE to the ESP32. However I disliked this approach for a variety of reasons:

1. Cumbersome and time-consuming to set up Synthesia+MIDI controller combo – average time was around 2-3 minutes to open all the required apps and connect to Bluetooth.
2. BLE finicky – sometimes, the BLE connection would hang and the song would abruptly stop only to randomly resume in a crash of notes a while later. I attribute this to the BLE-MIDI library implementation on the ESP32.
3. Synthesia UI is hard to use on mobile – synthesia works well on a larger device like a tablet or PC, but it is absolutely a pain to use on a small device like a typical mobile phone: the controls are small and hard to manipulate. In addition, the main allure of Synthesia over other software is the “falling keys” visual animation. However on a small device such as a mobile phone, this feature is hard to appreciate due to the small size of the screen.
4. Single-user constraint – since the interaction between player piano and user is confined to only the device which is running Synthesia, it is unwieldy for multiple users to interact with the player piano at the same time. Compare this to, say, Spotify on a Google Nest where multiple users can interact with the song (play, pause, change volume etc) without having to physically walk up to the Nest. I wanted to achieve a similar experience with the player piano.
5. Lack of Volume Control in Synthesia – some .midi files can be very loud and with no in-buiilt volume (or note velocity) control, some songs are just too loud to play.

Given that I have pretty much zero experience in webdev and I have an upcoming internship with a startup maintaining an Angular webdev project, I decided this would be a good time to cut my teeth on some Angular webdev. I chose to use NestJS for my backend simply because it was the backend framework which was recommended by a bunch of tutorials.

Realtime communication between the server and the frontend (music controls) as well as between server and the player piano occurs via websockets (Socket.io). I spent a lot more time than I would have liked debugging the SocketIO connections (first I forgot to disable Sever-Side Rendering and then I forget to enable CORS….).

.midi files are parsed using a midi-player js library on the backend which sends the NoteOn and NoteOff commands via websockets. Overall, the system works as expected but there is large latency between the parsig of the midi in real time on the backend and when the note is actually played and received by the frontend:

I am unsure if the problem is with the JS Midi player library where the note events are being sent to the ESP32 at incorrect timings or rather with the websocket protocol introducing inconsistent latency to the note arrival at the ESP32. More testing is required but as of now, I have reverted back to the BLE-Midi code for everyday use of the player piano.