MoodModArt – Emota™ v4.0
A reactive artwork that detects facial cues from viewers moods in a gallery space and/or via mobile apps.
On the left is four oil painintg still life panels. On the right is a large screen LCD panel with a dynamic reactive animation film loop.
This artwork was on display in the West Mich Redux exhibition in South Haven Center for the Arts, South Haven MI in 2022.
This installation presents a dynamic time lapse still-life painting that shifts subtly caused by sensing personal characteristics of the participants via a mobile app from remote locations. The body of work explores an autonomously responsive media delivery modified by detected and analyzed biometric data. In the exhibition space, the art works presented are a series of time lapse animated or filmed scenes that change with atmosheric, stylistic or other variable elements.
The video above, demonstrates capture, normalization and detection of a face and the process of emotion recogntion, with the resuting a response based on emotions detected.
The system intelligently analyzes the participants faces, by a camera capture via a mobile app. Based on sensing personal characteristics of the viewer modifies the form and content of the media. This installation seeks to engage the viewer as well as remote participants in an aesthetic experience that is audience controlled.
Technically, the design of such software requires an expert system intelligence that is ambient and ubiquitous – allowing for interpretation of a wide variety of stimuli that can be easily acquired. The system's intelligence must offer a range of options that can be autonomously reactive, giving meaningful responses to the visual/sensor cues captured from environment as well as from mobile devices with an active engagement by remote participants. The system can be implemented using backend server applicaions for the analysis of the facial images transmitted from social media mobile apps or web-based experience in browsers. Facial images are analyzed for individual's emotion expressions,
moods and personal physical attributes. Our method uses a comibination of vector map and 3D mesh local feature analysis (LFA) of ordered matches and an image processing approach by identifying shapes within a image field using algorithms such as Viola and Jones Open CV Haar-like application[1], [2] and a library of facial image (LibSVM.[3])
For emotion detection, we make use of use 3D mesh vector templates using elastic bunch graph matching that tags facial features and performs a local feature analysis (LFA) of ordered matches that enable a relative accurate detection basic emotional states, anger, disgust, fear, happiness, sadness, and surprise. [7]
No images of the participants are permanently stored in the process. as soon as the system analyzes the participants trans-mitted image the system purged the image. From the analysis the system determines the best guess emotional states.
The facial recognition was initially developed as a tool for marketing and advertising in public spacesand presupposses a customized, autonomous information delivery for communication or media installations in public spaces or social environments. Extending the basic technology, the content or the images and media delivered in such a system can vary widely depending on the installation location, expected crowd and population demographics.
Further work perfects the algorithm, increased accuracy and to allow capture and detection of physical cues from audiences or participants. The results can be seeen in the work under the link, "Rgw". This software detects biometric data for a graduated scale of the seven basic emotion states, as defined by Eckman, et al, in real time.
Collaborators: Pensyl, W. R.; Song Shuli Lily; Dias, Walson; Huang, Walter Yucheng Walter; Fish, Joshua; Animation by Dave Maggio
2. Bradski, G. and Kaehler, A., (2008). Learning OpenCV. OReilly.
3. Burges, C. J.C., (1998) A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery 2, 121-167
4. http://www.nist.gov/huma nid/colorferet
5. Wiskott, L.; Fellous, J.-M.; Kuiger, N.; von der Malsburg, C. (1997) Face recognition by elastic bunch graph matching, Pattern Analysis and Machine Intelligence, IEEE Transactions on Machine Intelligence, Volume: 19 Issue:7 Pages 775 - 779
6. Bronstein, A. M.; Bronstein, M. M., and Kimmel, R. (2005). "Three-dimensional face recognition". International Journal of Computer Vision (IJCV) 64 (1): 5–30
7. Ekman, P., (1999), "Basic Emotions", in Dalgleish, T; Power, M, Handbook of Cognition and Emotion, Sussex, UK: John Wiley & Sons, http://www.paulekman.com/wp-content/uploads/2009/02/Basic-Emotions.pdf