While I think it would be so fun to design my own conducting baton controller for my game idea, I am continuously pushing myself away from the idea since I know so little about the hardware, and any roadblocks (that I would most definitely run into) will keep me from doing what I think I would enjoy a lot more consistently, roadblocks and all: composition and game design.
I found "Conductor Hero" online which is remarkably close to what I might want to do, but this video excerpt demonstrates a very serious version of what I'd like to accomplish. "Conductor Hero" is more educational, with a seemingly static player (besides arms) doing one simple task to demonstrate the complexities of conducting. While this is awesome, I would like to blend two ideas: composition and conducting, into one game that is very satisfying for the player, and in a sense, more gamified and 'game-like'.
To continue this idea of increasing gamification, I'm thinking of taking inspiration from "Audioshield", the second attached video. I can use simple particles in unity/unreal (don't know which yet) to convey very small musical ideas that the player can easily grasp. Assuming I correctly match certain colors and other particle attributes to the correct emotional implications, I should be able to communicate the correct ideas to the player without any confusion. This is also a good workaround to me not being a great artist. This gameplay would include the player using a conductor-like baton controller to grab these musical particles/ideas and arrange them for performance. I can then introduce a mechanic for replay and editing of their current piece using the controller. I have to do more research into batons that can process speed and midi data, or how I can build my own simple one.
To expand on some of the specifics of the game world parameters, the world would be 3d and involve player movement -- definitely no standing in one place to manage sound -- the player must SEEK out their musical ideas, trying new things. A stretch goal would be to solve puzzles to unlock extremely idiosyncratic ideas. I want this to be similar to the actual compositional process all composers have to deal with -- they put in a little extra work to find that unique musical idea that has been evading them -- without being dead serious about the pain of composition and musical organization/decision making. While these musical ideas I'm thinking of including may seem too pre-composed, I would like to counter this with well-designed musical ideas, and a multitude of them to an extent that the potential for a unique composition becomes infinite. Given that the musical capabilities of Unity and UE SUCK in terms of spatialization and good asset organization and playback, I can still use this opportunity to include and practice Wwise integration.