Latest Update:
Developed a trigger function, so instead of increasing volume of track based on distance between thumb and fingers, touching them and crossing a certain threshold distance uses a boolean function to trigger the midi note.
Using this python script, all the input channels crossing the threshold of "21" will trigger the midi component to play a frequency on the assigned Ableton Live track.
The program now calculates the distance between fingers by plotting the circle TOPs assigned to them on a graph. Using geometric functions and modulus, the program can now compute gestures of any orientation.
Future Scope:
Instead of Song Sampling, the program is now able to trigger midi notes from Ableton. Assigning different gestures to these notes can help in designing a musical instrument.Instead of Song Sampling, the program is now able to trigger midi notes from Ableton. Assigning different gestures to these notes can help in designing a musical instrument.
We facilitated numerous interactions using touch design and Ableton, allowing users to engage in their own unique ways. As they experimented with different finger movements and actions, they intuitively discovered which gestures produced specific sounds. Although there were no explicit instructions, it was clear that everyone understood the connection between their actions and the resulting auditory and visual feedback
Speed: Users tried rapid hand movements and intricate gestures, while expecting instantaneous auditory feedback from the program. The prototype failed to work in these cases, not producing any noise as the changes were to rapid for the sampling to keep up with them.
Usability: As the prototype needs a specific distance to work, i.e. where the camera of the device can view the user with enough contrast to distinguish the different fingers from each other.
Distance: We need to tweak the function that accommodates the z axis parameter with the hand tracking. As of now, the tracking points merge into each other when hands are far away, scrambling any further use of their data.
Calculation Issues: The prototype can measure the x-axis and y-axis distance between fingers, but I have currently found no way to find the length of hypotenuse between them. It is more of a software issue, as I have no idea how to apply the mathematical calculations in TouchDesigner.
Visualization: A better visualization demonstrating users inputs as well as system outputs should be looked into. Different visualizations between strings, keys, bass and percussion can be explored.
Guidance: An experiential quality that we need to add to the interaction. Some subtle mechanisms to guide the user would enhance their experience, and help them use the prototype more easily.
Connected TouchDesigner with Ableton using the TDAbelton node to produce audio outputs from hand gestures. Initial attempts were hindered by unfamiliarity with Ableton's interface, but eventually, gesture-based audio sliders were implemented for dynamic sound modulation.
Future Scope:
Referencing the Z-axis data to the radius of the TOPs can significantly reduce the calibration and prerequisites required for users to successfully interact with the program.
Currently, this feature is jittery and inconsistent, making it difficult to use. Further development would require considerable time and could detract from work on other essential features. However, leaving it as is may lead to additional inconveniences—foreshortening will reduce the perceived distance between the circles but not between the fingertips, making the prototype difficult to operate in less-than-ideal conditions.
Assigned some Texture Operators to each fingertip. Some resolution management and maths was required to scale the values of each fingertip down. This helped us calibrate the live data from the video to the screen’s dimensions. Live Referencing the x and y values to the Circles helped us create objects with more usable attributes.
Using the MediaPipe Library of TouchDesigner, we were able to use necessary Python scripts needed to to both recognize hands and identify and track specific points of the hand. The challenge after that was to filter the channels we required from the media pipeline node. This involved recognizing the hand tracking nodes, and selecting, renaming, and then bringing the channel data out from the node.
Future Scope:
Following up on this method will be helpful in creating 3d and reactive visualizations. With more dimensions of input like sound created and gestures made can be helped to morph different aspects of the surfaces. This can be helpful to create a front-end experience for the users, as it provides a much needed visual feedback.
The aim was to experiment and track the finger movement here. Tried to use surface operators here along with channels to experiment which inputs and outputs would facilitate interactivity.
We were able to track directional movements and direct the axes of the surfaces used with the help of primitive gestures. Translation of this data to others proved difficult to execute.
Future Scope:
Using this specific algorithm can be helpful to add more nuance to the scope of interactivity to the final program.
e.g. The changes in the channel operator from this algorithm can be directly referenced to some attributes that can add a “vibrato” effect.
The aim was to experiment with the body movement, and here I was restricted to left and right movement. With this restriction, I tried to play around, thinking of ways to associate this with sound.
However :
The channel inputs were very noisy.
Every little change in pixels was being taken as input.
Segregating movement of different body parts would not be possible
The input was of relative nature. The directional input was based on continuous changes in pixel, and the information recorded by the system would just be dealing with 4 parameters of movement: Up and Down, Left and Right. This was achieved by using optical flow node of touchdesigner, and analyzing the flux in the input video pixel information. As long as there was flux, there would be output.
Possible Solutions:
Instead of feeding the whole video as inputs, we can use the TouchDesigner shape operators assigned to each fingertip as input. Using more thresholds can help further refine the channel output by filtering larger, or smaller changes.