Creating an Entertaining and Informative Music Visualization

Editor's Notes

• #2: Hello Everyone, my name is Michael Pouris, my co-author is Dr. Deborah Fels. We are from the IMDC at Ryerson University in Toronto Canada. My presentation is about how to create an entertaining and informative music visualization. More specifically to create better access to music through visual aesthetics that are grounded in a psychological cognitive model for translating auditory constructs to visual constructs. I call my solution musicViz
• #3: First I will present some background information to the project. This includes the use of music in conveying modern culture and how the deaf, deafened and hard of hearing do not have access this shared knowledge space or have very little. I will then describe the purpose of music visualization and some past industry attempts and how they failed.I will describe the priorities that were extracted from these failures and the psychological model that I built to conform to these priorities, which is what my music visualization called MusicViz built on.
• #4: In modern western culture, music permeates lives. It is in, restaurants, spas, gyms, sporting events, clubs, concerts. People listen to it while working, cooking, lawn work, writing, etc. It is a major art form that is present in all cultures world-wide. It is, not only intrinsic to all cultures in modern day, it also intrinsic to all ancient cultures throughout history.Essentially, music transcends cultural, linguistic and temporal boundaries. Individuals can understand the emotional aspects and enjoy music of completely different cultures. This also holds true with modern reconstructions of ancient cultures. Essentially music is a portal to a shared cultural experience buy hard of hearing and deaf have little to no access to this culture.For people who are deaf or hard of hearing, they experience sound as physical tactile vibrations, therefore they feel sound through their bodies and because the lower frequencies are higher powered, they are a stronger signal and mask the higher frequencies. Therefore the only signal is low frequency vibration that are available for the tactile domain.Another way to communicate musical entertainment for the deaf or hard of hearing is through our music visualization system called MusicViz.
• #5: MusicViz’s purpose is to serve as a tool, which depicts music visually for hard of hearing and the deaf. The goal of visualization is to transform music into the visual medium to be entertaining.Nanayakkara Showed that deaf and hard of hearing audiences were able to experience emotion from visualization; however, It is important to note that they did not provide a sensory substitution model.The main research questions are:Can sensory substitution models be used to provide access to music using the visual channelWhat are user reactions to musicViz?
• #6: There have been previous industry attempts at music visualization such as AniMusic, Itunes, Windows Media player and Music Animation machine.These previous attempts have failed. They are entertaining, yet they do not give any informational or emotional queues of music. There have been other attempts at music visualization; however, they have focused only presenting music notation in different forms to make music notation easier to learn. Others have attempted to interpret the meanings of musical pieces by using colour to convey emotions, which is a large problem considering the meanings of colours differ not only between cultures but even differ within a culture.
• #7: From analysing the deficiencies of the previous failures, I have extracted three main priorities that MusicViz should address:Aesthetically pleasingEmotionally movingAdaptable, must work with all types of music
• #8: Interpretation is where the meaning of something is explained by an individual from a subjective viewpoint, essentially adding in bias, which should be avoided. There exists multiple feelings for a singular piece of music, that differ from individual to individual based on personal experience; hence everyone interprets music differently. The goal of the visualization is to present entertaining, yet objective information to the individual.This is done through TRANSLATION… 
• #9: The definition of translation is to change or convert to another form. Therefore Music Visualization should simply translate auditory queues into visual queues using a proven psychological translation model. The goal is that the translated information can be interpreted by each individual based on personal experience. However, the visualization must not simply present information, it must also be entertaining.
• #10: In the auditory system, pitch height is associated with two different visual constructs. Higher Pitches = smaller objects (a mouse makes high pitched sounds)Lower pitches = bigger objects (an elephant)Higher pitches = higher in the air (cartoons where people are kicked into the horizon and a high pitched ding is used)Volume/amplitude is associated with object size and looming:Bigger object makes a louder soundSmaller object makes a softer soundBigger object are associated with looming (a car coming towards you or in evolutionary terms, a sabre tooth tiger attacking)Finally tempo and beats are indicate a repeated patter through beats per minute.
• #11: However, there is a problem with the translation from auditory to the visual medium. Pitch Height and Volume conflict. When studied independently, higher pitch is associated with a smaller object and louder sounds are associated with a larger object, but there is a conflict of variables when Therefore a note that is simultaneously high pitched and loud cannot be small AND big at the sometime. Essentially the problem is that amplitude and frequency are integrated in the visual system.
• #12: The translation was implemented using Java 6 and JOGL 2.0, which is OpenGL for Java. The API maintains performance through making native C calls through GLUEGEN (glue code generation), hence it acts as normal C OpenGL in terms of performance and function calls.Java and OpenGL was used because it was cross platform. The musical input is MIDI. The program called MusicViz received input in real time from a MIDI sequencer and displays the translations.The X-Axis is used to arrange the instruments, where the instruments are displayed as pipes.The Y-Axis is used to represent pitch-heightThe Z-Axis represents time progression hence tempo
• #13: The pitch height determines the height of the pipe on the Y-Axis.In order to differentiate the different instruments, a unique colour is assigned to each pipe. The colours are in no way made to represent emotions that the instruments convey and are in no way used to identify an pipe as a specific instrument. Colours are simply used to convey the information that different colour pipes are different instruments.
• #14: Volume is depicted using the thickness of the pipes. As can be seen, the blue pipe is thicker than the green pipe. Therefore we know that the green pipe is louder than the blue pipe.Tempo is depicted through clusters of activity, where a faster tempo means there are more changes in a smaller space. A slower tempo spreads the changes over more space.
• #15: An intrinsic element of deaf music is a deep bass drum. It is identifiable in all deaf music because it can be felt. Therefore the base drum is essential in the visualization of music. However, to provide more information to the users, the other drums are displayed as well.Inspired by deaf culturePitch contour is the movement of sound over time and this is represented as a worming motion by the pipes.
• #17: To determine the enjoyability and effectiveness of MusicViz, 3 H (one male, two female), 3 (HOH) (two male, one female), 6D (three male, three female). The participants completed a pre-study questionnaire to gather demographic information, such as gender, hearing status, musical preferences, reasons for listening to music and education.A post-song questionnaire was filled out after the participant listened to each song and a post-study questionnaire was conducted after the participant watched all six songs. The post-study questionnaire was used to give feedback on the meaning of individual constructs in the visualization. Such as shapes, movement, brightness etc.
• #18: The post study questionnaires were used to give feedback on participants’ enjoyment and emotional experience with the visualizations.Enjoyability was measured using a 5-point likert scale ranging from not enjoyable to enjoyable.Emotions are measured in two ways. The first way is with the discrete model proposed by Ekman that takes into account happy, sad, anger and fear. The second method through the 2D model of valence (happiness) and arousal proposed by Russell.Lastly the level of focus on the visualization and enjoyability of the colours used are measured.This presentation only focuses on the finding from the post-song questionnaires
• #19: The first to be analysed was the difference in enjoyability between genres and no statistical differences were found.Even though no statistical differences were found, the descriptives show that all genres are rated between somewhat enjoyable and enjoyable, which means the visualization is overall pleasant to view.Rock was the most enjoyable genre, possibly because it one of the simpler visualizations due to less instruments displayed but more movement from each pipe.However it is surprising that rap/hiphop and RB are the least enjoyable. Originally the hypothesis was that it would be the most enjoyable because it is what they said they liked. In the tactile domain, it is how they experience music; however this is not translated into the visual domain.Rap is possible the least enjoyable because its boring to see. Two instruments that are flashing, there is less movement.Classical is less enjoyable because it has too many instruments and too much movement.
• #20: There was a difference in the levels of overbearingness between genres. More specifically:Pop is more dominating than RHRBPop is more dominating than countryThe reason for pop being more overbearing than RHRB and country may be because of the same reason as previously mentioned, which is the amount of instruments on the screen.Pop music contains more instruments including drums and a prominent bass line that repeats a lot, while guitars performing with little or no drums often characterize country music and the lack of instruments characterize RHRB. Therefore more vigorous movement in pop is more overbearing.A noticed trend in the data is that even though a song can be overbearing, it can still be happy. For example, pop visualization is the most overbearing, with the highest arousal ratings yet it has the highest happiness ratingsValence and arousal have no statistical differencesValence is positive and negative feelingsArousal is engergy, sleepy, tired.Sadness is negative engery and negative feelingsFer is high energy and negative feelings
• #21: Differences in level of discrete emotionAnother measure is the differences in the level of discrete emotions. The levels of happiness, sadness, anger and fear.Unfortunately there is no significant difference in the ratings of discrete emotions between genres. An issue with this scale to rate emotions is that because music can have multiple emotions, the individuals had a difficult time choosing from a limited set of words
• #22: There was no significant difference in the level of focus on the visualization between genresPaid most attention to rock possibly because it has just enough change and movement to maintain focus and interest, whereas pop is too much and overtaxes the perception and cognitive systems and rap is too boring. It is too repetitive. Use example about class.
• #23: In conclusion, a major limitation to the study was the lack of participants. Therefore, in order to determine whether the observed trends are statistically significant, more participants are needed.In addition, more participants are needed to explore the differences among hearing statuses.However, in terms of descriptive analysis, the participants showed preferences for rock, pop and classical genres. They overall enjoyed the visualization and based on comments from participants, many thought MusicViz helped them understand musical information.
• #25: Deaf, deaf, Deafened, and Hard of Hearing ConsumersThe distinction between the terms Deaf, deaf, deafened, and hard of hearing is based principally on preferred modes of communication.Deaf (upper case 'D') is a term that refers to members of a socio-linguistic and cultural group whose primary language is sign language. In Englishspeaking parts of Canada, the main sign language is American Sign Language (ASL).Deafened and deaf (lower case 'd') are terms that refer to individuals whohave lost all or most functional hearing at some point in their lives. Thesepeople use spoken language and rely on visual forms of communication suchas speechreading, text, and, in some cases, sign language.Hard of hearing is a term that refers to individuals who have a hearing lossranging from mild to profound and who use their voice and residual hearingand, in some cases, sign language for communication.just fyi, found it in a canadian captioning standards document