Minimal Displays or Breaking the Pixel Spell

Editor's Notes

• #2: My goal is to reflect on the technological trends on interactive displays surfaces And present some examples of alternative ways to go Ask some fundamental questions about the nature of displays and input interfaces This will be a very idiosyncratic talk, I want to promote a lively discussion during the workshop Alvaro Cassinelli, Assistant Professor, University of Tokyo
• #3: Reality is always “augmented” through our senses and activities It is just not “DIGITAL” “frame” is the world
• #4: We need perhaps to go back to move forward Elements of “Media Archeology” Some many contradictions! (typical of a word to take on its exact inverse meaning – necker cubes of the mind…) http://wro01.wrocenter.pl/erkki/html/erkki_en.html Note however the light behind the screen.. And how fire is “UNREAL”
• #5: Although we may be moving BACKWARDS (we WANT foldable displays), I will concentrate today on: FRAME -> framed content, giving meaning to the data PORTABILITY -> position in space meaningful or not?
• #6: Portable: then FRAMED Content: writing, then “DIGITAL” / GRID (note: rarely colors… but shapes) Summary account of silver for the governor written in SumerianCuneiform on a clay tablet. From Shuruppak, Iraq, circa 2500 BCE.British Museum, London (Wikipedia)
• #7: Canvas, panel painting Portrait of a man, ca. 125-150 AD (from el-Fayyūm) Fits well with architectural spaces (that act themselves as “frames”, grids or folders) Lots of hypothesis here: Transportation and selling Cheap confection of material (no need to paint on walls all the time) Panel painting is very old; it was a very prestigious medium in Greece and Rome, but only very few examples of ancient panel paintings have survived. Until canvas became the more popular support medium in the 16th century, it was the normal form of support for a painting not on a wall (fresco) or vellum, which was used for miniatures in illuminated manuscripts and paintings for the framing.  Attributed to Raphael, Guidobaldo da Montefeltro, c.1506, Galleria degli Uffizi, in a ‘Medici’ frame, last third 17th century.
• #8: This is at the CORE of the WINDOWS GUI (in particular windows 8) Stuck in this paradigm, including when we do Augmented Virtuality (difficult to share information!) I have been very intrigued by this problem, as a media artist and researcher.
• #9: This obsession with the RECTANGULAR PIXELATED SCREEN has not AUGMENTED reality… but perverted it! Formatted, Flat and Pixilated? exactly NOT like cave painting… The pixel becomes the atom of picture The rectangle display becomes the ideal portable frame In other terms: there may be an “ultimate” pixelated display: a super-high resolution, bendable, strechable, low power “sheet” or “painting” that can be applied over all sort of surfaces… If we abstract trends for a moment… what is the future of “information surfaces”? but this has nothing to do with Sutterland “ultimate display” (I will go back to that in a minute). Surfaces remain surfaces… Of course it will be possible, but… what about the alternatives?
• #10: Of course, advertisement panels, etc. These attempts are PERVASIVE Illumiroom is a perfect example of the need to go beyond the screen, which is the “point of attention” BUT WE NEED TO USE THE WORLD as the FRAME, NO NEED TO HAVE A FRAME IN THE WORLD…
• #11: I’ve been always very interested in this problem of the recursive image: our environment looks like a “visual LISP” ;) (one image frames another and so on – screens contain screens…)
• #12: http://theframeblog.wordpress.com/tag/musee-du-louvre/ A Madonna is a representation of Mary, either alone or with her child Jesus. These images are central icons for both the Catholic and Orthodox churches.[1] The word is from Italianma donna, meaning "my lady". No image permeates Christian art as the image of the Madonna and child.[2]
• #13: If the screen/painting is a WINDOW to the imaginary world… … then the SCREEN is a WINDOW to the DIGITAL WORLD (That we could go “through the screen” or pass things through it) Extending the power of a metaphor Challenging first the metaphor of the glass-pane window Deformable screens Membrane separating worlds Making the screen the interface
• #14: Same than the Volume Slicing Display, but the screen is FIXED but DEFORMABLE. Created as a media art project, and content is TIME LAPSE PHOTOGRAPHY or VIDEO.
• #16: Key concept: shared “frame of reference” between real and virtual world the screen is a membrane between the real and the virtual world. a physical attribute makes interaction more precise, even if this is PASSIVE FORCE FEEDBACK
• #17: Now, the screen is rigid, but can move in space: full body proprioception help better understand the content. The FRAME is the world The screen is still the locus of control (no “mouse” there). Of course this has been now realized on tablets.
• #20: Latest concrete application of these ideas: not only superimpose on reality a “virtual object”, but a VIRTUAL SPATIALIZED DATABASE And also a space for “chatting” and exchanging information: a SPATIALIZED SOCIAL NETWORK.
• #24: Finalizing my discussion.. Libraries SCALE this pattern, the same way SCREENS scale the pixel pattern And I am not even talking about the fact that ANALOG computers were displaced by DIGITAL ones. The idea of the “digital” won. Shanon?
• #25: Understanding of the production of color, psychophysics, eye A “reductionistic” approach to color: the pixel = atom of the picture Mimmics increase of transistor density (technological reason, TFT, etc) Microdisplays can be enlarged optically But ultimately, they are subjected to other constraints (psychophysics, substrate rigidity or flexibility, power, etc)
• #27: BUT BEFORE TALKING ABOUT the details: one important question: WHAT IS A DISPLAY? Minimal, interactive displays vs. pixelated projection mapping? Ambient displays (a little different: I want minimal displays to be amorphous, interactive displaying proyective displays) Initial research on ambient devices began at Xerox Parc with a paper co-written by Mark Weiser and John Seely Brown entitled Calm Computing. Associated fields include Ubiquitous computing (also known as Ubicomp) and Calm technology. Can be light, but does not need to be PIXELATED LIGHT: minimal displays. Can be something else (cold, heat, pressure…) The technology that we have been developing (using lasers) is capable of creating a MINIMAL INTERACTIVE DISPLAY Examples: The Laser Aura (for augmented expression) the “robot made of light” or “light pet”. MINIMAL DISPLAY: The goal is not to render a 2d “pixelated” image Organic, fluid behavior Attention driving graphics (extension of laser pointer) Location aware / opportunistic / unobtrusive (appear only when person near the interaction space). Iconic representations? Simple alphanumeric data possible though… Real time sensing Zero-mismatch
• #28: An example will give a better idea: cartoon-like graphics
• #29: Laserinne with Jussi Angesleva et al.
• #30: Looks like an artificial AURORA BOREALIS? Technology that is inspired by Nature. Uses the same language.. This project consisted on projecting a 300m artificial sea-shore (but looking very fluid) onto a snow mountain. Interactive games on the slope, annotation, etc.
• #31: (but still, this is a “feature language” and color is a complement to shape. What about a purely color writing system? That would really be good for a minimal display, ambient display, wearable… (draw examples on glasses, and ambient, and laser) Benin and Edo people of southern Nigeria have developed a chromatographic system of writing. That is a writing based on different color combination's and graphs. Aba Abhuluime of Nigeria brought the chromatographic writing system to our attention. The system is also called quantography or pictography. This system of color writing and tradition needs to be further investigated. According to Mr. Abhuluime, there is a published work entitled "African Science: The Art of Colour Writing" by Aba Ota in 1999. We hope to provide additional information on color writing in the future. Source:http://afrikafriend.4bb.ru/viewtopic.php?id=1344&p=2
• #34: Non-PHOTOREALISTIC RENDERING
• #35: How to generate a sense of presence for virtual objects? Or, “how to TAKE THE CLOUD DOWN TO EARTH?” This can go beyond projection mapping: MULTIMODAL AR. The Ghost Cloud or an “information-haunted reality”
• #36: An example beyond projection mapping: tactile feedback from virtual objects. This is like “feeling ghosts”, force fields around co-located virtual objects. Details: tracking is done using ultrasound (time differences between pulses).
• #38: Note the conceptual similarity with the “Virtual Haptic Radar” Indeed, this could be used for augmenting video games, even games for the blind.
• #41: High speed feedback is important for generating the feeling of “reality”. If delay is perceived, the “magic” brakes and we know that this is “not real”. Interaction becomes awkward and slows down not just because of the actual delay, but because we fight it (consciously or unconsciously). This generates cognitive load.
• #42: ENABLING TECHNOLOGY: zero-delay, zero-mistmacth Sensing has to happen AT THE SAME TIME with display
• #43: Examples of things done with the Laser Sensing Technology: TRACKING: in fact, it’s HOW this started: laser technology for tracking for wearable/portable devices: no need of imaging sensors. This works well because WE DON’T NEED LONG RANGE INTERACTION: the beams acts like CILIA or ANTENNAS from insects. SENSING is “SMART”: we only look at the information we WANT It is then possible to do it EXTREMELY QUICKLY (high-speed feedback) “low level image processing” directly on the object: no misalignment (example: vein viewing, contour enhancement, contrast enhancement…) AR surveying!
• #47: What can you do when you have very high response? REAL TIME PERFORMANCE, for instance SOUND GENERATION !
• #48: More video demos.
• #49: Another example of something that CANNOT be done with present camera-projector technology: projection mapping on a: Deformable surface Mobile surface Perhaps one day this will be possible with normal projectors and cameras of course. But not now…
• #52: Yet another example: Input-output “console” projected on the real world, opportunistically For communication from remote locations (not yet published) For controlling appliances (control panels projected on objects).
• #53: It is interesting to note that since the laser is capable of measuring DISTANCE, for “each pixels” of the projected image, it is possible to compute in real time the POSE of the surface (or its shape in case of a deformable surface). Therefore, it is possible to correct perspective in REAL TIME.
• #54: Details of the first idea (communication)
• #55: Now I think I can put all together…
• #56: Have you noticed? Today’s smartphones look like soap bars.. I am sure this form factor will be a thing of the past very soon. The smartphone (or tablet) is just more of the same: a tiny computer with a rectangular pixilated window (this “window” is an opening to the digital world). Should this be like this? We are naturally capable of telling function from shape, or meaning from place – in the world referential. This is lost in today’s “window” paradigm. Body referential vs. world referential. Is it “wearing” them the solution? (google glass). Perhaps not, because we retain the paradigm of the “squarish, pixelated window” to the virtual world (the screen). What is needed is to “blend” the real and the digital, and manipulate both in a seamless way, using the BODY.
• #57: Form, Function & Computing Embedded display & a tiny computer per object Ambient intelligence & one function projector augment objects … and the “final” thing would be to create a projector that project “function” on passive objects. That PROJECT or instantiate AFFORDANCES (perhaps perceived but not real in the physical object). Perhaps not doable today, but we are underway. FUNCTION on objects, but also HISTORY of interaction, and of course, SUPPORT FOR DATA (objects as “folders”) “Hybrid objects” (augmeted and physical) will become just REAL objects with enhanced affordances.
• #58: Revisiting skeumorphs
• #61: In collaboration with Shinoda-lab, a new, more performant system was developed in the Digital Image Control group.
• #62: ANOTHER EXAMPLE : using the fact that objects that move fast cover a large area but cannot be seen in detail by the eye, we CAN USE THE WHOLE SURFACE FOR PROJECTION if we can track and project in real time.
• #63: This is a final, unifying concept. The “light pet” acts as a physical projection of the Cloud: a “condensation” of it in the real world into a “tangible and sentient display” Why? FAST RESPONSE and NO spatial mismatch makes the user feel that the “agent” is really there: it’s NOT a projection. We can also project SOUND, further enhancing the impression that the presence is physical (and not a projection). From that, plus some behavioral rules, it follows that the user feel “intentionality”. This points to a novel paradigm of HCI design: playfulness, interaction to discover more affordances, an inviting “robot of light” that enables function-appropriation and function-redesign in real time.
• #64: Slow display 2d or 3d Liquid, particles Printer using capilarity LIQUID OPTICS : reconfigurable architectures, windows