Vokasi UI / Media Production, Class C
Session 02
New Media Technology

Media That
Responds.

Session two. The technology behind interactive and motion based media, and today you build a motion reactive demo of your own.

Instructor Nicky PandelakiLoad 3 SKSSession 02
Recap / Where we are
From last session

From landscape to interaction

  • Last session: what new media is, and generative AI as this semester's star.
  • New media is not only digital, it can respond to you in real time.
  • Today: the technology that makes media interactive and motion driven.
  • Our anchor: the interactive billboard that reacts to you.
New Media TechnologyClass C
Watch first / Warm up
Ogilvy

A billboard that looks up. It detects real planes flying overhead and reacts live. Interactive media in the wild, before we build our own.

New Media TechnologyClass C
01 / Foundations
The core idea

What makes media interactive?

  • It responds to you. Your input changes the output, in real time.
  • The loop: input, sensor, computation, instant output.
  • Old media is one way. Interactive media is a two way conversation.
  • The audience becomes a participant, part of the piece.
New Media TechnologyClass C
01 / Landscape
The forms

Forms of interactive new media

  • 01Interactive installations, billboards and screens that react to motion.
  • 02Immersive media, AR, VR, 360, and virtual tours.
  • 03Web interactive and creative coding, art you click and move.
  • 04Generative AI, media that responds to your prompts.
  • 05Computer vision and body tracking, the camera as controller.
  • 06Real time and live, streams the audience steers.
New Media TechnologyClass C
02 / Spotlight
The anchor example

How a motion billboard works

  • A camera watches the space. No marker suit, no controller.
  • Computer vision finds a body and tracks its pose in real time.
  • Software maps your movement to a response on the screen.
  • You move, it reacts, instantly. That is the whole magic.
New Media TechnologyClass C
02 / Case studies
Real campaigns

Interactive media in the wild

  • 01British Airways, Magic of Flying. A billboard that spots planes overhead and points at them, live.
  • 02Nike, Reactland. Shoppers ran on a treadmill to drive their own avatar. 48 percent bought the shoes.
  • 03Chris Milk, The Treachery of Sanctuary. A Kinect installation that turns your body into a flock of birds.
New Media TechnologyClass C
Why it works / By the numbers
Engagement converts
Nike Reactland players who bought the shoes
48%

When media reacts to you, attention turns into action.

New Media TechnologyClass C
Watch / Installation
teamLab

Art that moves as you move. A whole museum that tracks your body and responds. The output can be anything, not only a billboard.

New Media TechnologyClass C
03 / The shift
Now it is in your hands

You do not need a Kinect rig

  • A few years ago this needed depth cameras and custom code.
  • Now your webcam plus AI in the browser can recognise what it sees. Free.
  • This is image classification, you teach the machine by example, not by code.
  • Same idea as a smart installation, with a cheap camera and a neural net. No standing required.
New Media TechnologyClass C
Watch first / The tool
Google

Teachable Machine in two minutes. Teach a model by example, right in the browser. This is exactly what you are about to build.

New Media TechnologyClass C
Hands on / Build it
Your turn

Now you build a mini reactive screen

  • Webcam plus browser. Zero install, nothing to download. Do it sitting down.
  • Teach it a few things it can see, then make the screen react to them.
  • That is your own interactive installation, running on your laptop.
New Media TechnologyClass C
Hands on / Step by step
Teachable Machine

Train an image model in your browser

  • 01Open teachablemachine.withgoogle.com, Get Started, then Image Project, Standard.
  • 02Allow the webcam. Make 3 classes, for example Thumbs Up, Open Palm, Neutral. All while sitting.
  • 03For each: show the gesture, face or object and Hold to Record a few seconds. Nothing is uploaded.
  • 04Click Train Model. It runs in your browser in about a minute.
  • 05In Preview, the webcam reacts to what you show, live. Input becomes output, instantly.
New Media TechnologyClass C
Hands on / Build it
p5.js

Wire it into p5.js

  • 01In TM, Export Model, Tensorflow.js, then Upload (shareable link) to host it. Sign in with Google if asked.
  • 02Copy the p5.js code it gives, it already loads your hosted model.
  • 03Open editor.p5js.org, the free browser editor, and paste it in.
  • 04Add a few lines so a class does something, swap an image, play a sound, move a shape.
New Media TechnologyClass C
Hands on / Submit this
Your demo

Share your demo

  • In editor.p5js.org, hit Share and copy your sketch link (set it public).
  • Open the link once to check it runs with the webcam.
  • That p5.js link, your working interactive demo, is what you submit.
New Media TechnologyClass C
Hands on / Deliverable
What to submit

Your brief

  • Build an image reactive demo with at least 3 classes, gestures, expressions or objects.
  • Submit your p5.js sketch link (editor.p5js.org, Share), your working interactive demo.
  • Write 3 sentences: what worked, what broke, one idea to use it for real.
  • Submit via the link on the last slide. We review a few live next week.
New Media TechnologyClass C
03 / Responsibility
Before you ship

The ethics of watching

  • A camera on the audience is powerful. Always get consent.
  • Body and face data is personal data. Do not store what you do not need.
  • Design for everyone, not only people who can make the exact gesture.
  • Be honest that AI is watching, and what it does with the input.
New Media TechnologyClass C
See you / Session 03
New Media Technology

See you in session three.

Submit

Submit your work at
morari.studio/kelas-c/session-2/submit

Vokasi UI / Media ProductionClass C
← → to move · F fullscreen
← Class01 / 16Session 02