View Static Version

Multimedia Storytelling Comm 1002 * St. John's University

Welcome to Multimedia Storytelling!

This course is about the broad usage of multimedia tools and communication mechanisms used to form narratives that expand across media platforms.

“Multimedia” became a popular term in the 1990s to describe computer software that integrated image, sound, video and featured interactivity.

In this course, we approach multimedia as a broad term for combining multiple media formats. Whenever text, audio, still images, animation, video and interactivity are combined together, the result is multimedia. Slides, for example, are multimedia as they combine text and images, and sometimes video and other types.

Contemporary multimedia is also referred to as digital storytelling.

We will begin with the origins of storytelling and look at key components of narrative such as world and character building. Then we will trace the history of multimedia narration- with a focus on the media of photography, film, sound, video, and the remix of archival footage.

Last we will consider current issues such as transmedia narrative, interactivity, livestreaming, gaming, mobile social media, augmented reality and VR.

Some of the core questions this course will explore are:

  1. What are the fundamental components to narrative and storytelling?
  2. What is “new” about interactive storytelling today, and how can we better understand contemporary cases from the study of older models of multimodal or interactive storytelling?
  3. How can photography, film, and sound shape a narrative? How can we analyze multimedia narrative according to technical elements such as composition, form, editing, tempo, tone etc.?
  4. How can we understand and approach transmedia, intermedia, and nonlinear narratives across various platforms and social media?

This class is focused on both critique & production.

First, we will study how narratives have functioned multimodally and multimedially throughout history as people have taught, learned, entertained and communicated with stories.

We will examine and critique the techniques and strategies that allow creators to use stories to transmit information and transfer experience in an engaging way and successful way.

During the second half of the semester we will try our own hands at multimedia projects including a film treatment, storyboard, podcast, and audiovisual essay.

This course provides basic knowledge of the range of capabilities of available audio and video design applications. Don’t worry if you don’t have much production experience! The creative assignments are designed to encourage you to explore the creative and technical elements of the class

Emphasis will be on the exploration, utilization, limitation, and creative assemblage of words, images, sound, perspective, and personal creativity to create, edit and produce stories.

For one assignment, students will need access to software for working with images and text (Powerpoint or Keynote will suffice, but you are welcome to explore other digital software such as InDesign, Photoshop, or Acrobat. For a second creative assignment, we will explore software for editing audio (Garageband, or open source Audacity is a good option), and software for editing video (Windows Movie Maker Live & iMovie are free options. Adobe Premiere and Final Cut Pro is welcome).

Ultimately the focus is not on software mechanics, and calls for a discovery-based approach for learning how to achieve what is needed for the software of your choice that is being used.

Course Topics

  • Oral Storytelling
  • Narrative Structures and archetypes
  • Non-linearity
  • Character and World Building
  • Visual Storytelling
  • Cinematography and Editing
  • Film Production and distribution
  • Audiovision: Sound and Music
  • Video: Liveness, Feedback, and Remix
  • In between fact and fiction: archival footage and documentary
  • Mobile (Social) Storytelling
  • Self Performance and small stories
  • Transmedia and interactivity
  • Interactive movies, gaming, and “procedural” narrative
  • Immersion, VR, and the Senses

A Return to Oral Storytelling

The earliest narratives ever recorded were cave paintings.

The culture of cave art was in fact incredibly interactive. Modern carbon dating reveals that the paintings in many caves were drawn hundreds or even thousands of years apart. What that means is that for hundreds of years, people would enter the caves and interact with the pre-existing paintings. In the Civilizations clip, Simon Schama, art historian, suggests that the caves formed the first religious temples in this way.

What story do these hands communicate? To Simon Schama, the simple but powerful message that "I was here."

The oral tradition of storytelling was dominant before the spread of writing and literacy, and it was markedly interactive and non-linear.

Non-linear narrativity is when events are portrayed, for example, out of chronological order or in other ways where the narrative does not follow the direct causality pattern of the events featured, such as parallel distinctive plot lines, dream immersions or narrating another story inside the main plot-line. It is often used to mimic the structure and recall of human memory, but has been applied for other reasons as well.

Non-linearity also occurs when interactivity is relied on to control progress as with a video game or self-paced computer-based training. There is an open-ended structure; there is no critical linear path, and the end user discovers key elements in the story as they reveal themselves, or as they are found.

The reenactment of myths (religious rituals) were one of the earliest form of stories. The reenactment of myths was a common element of all preliterate societies. These events were both non-linear and very interactive.

Carolyn Miller describes how Dionysian rituals often surrounded the themes of death, rebirth and the cycle of the seasons. We can locate modern manifestations of Dionysian rituals in our culture today.

In this lesson we will continue to investigate the meaning of interactivity and non-linearity in the oral tradition. We also examine the ”the campfire model” vs ancient ritual model offered by Carolyn Miller, the concept of procedural authority, and the concept of incunabula, or “cradle works” within new media traditions.

Narrative Structures and Archetypes

Psychoanalyst C.G. Jung (1875-1961) defined archetypes as universal, archaic symbols and images that derive from the collective unconscious.

Archetypal Images are forms or images of a collective nature which occur practically all over the earth as constituents of myths and at the same time as "autochthonous", individual products of unconscious origin.

Jung writes: “Dream is the personalized myth, myth the depersonalized dream; both myth and dream are symbolic in the same general way of the dynamics of the psyche. But in the dream the forms are quirked by the peculiar troubles of the dreamer, whereas in myth the problems and solutions shown are directly valid for all mankind”

Joseph Campbell (1904-1987) was an inspiring teacher, popular lecturer and author, and the editor and translator of many books on mythology, including The Mythic Image and The Hero with a Thousand Faces. Campbell applied C.G. Jung’s concept of the archetype to his study of mythology. Myth, according to Campbell, is the projection of a culture's dreams onto a large screen

Borrowing the term “monomyth," a word coined by James Joyce, Campbell puts forth the ancient idea—that the mysterious energy for inspirations, revelations, and actions in heroic stories worldwide is also universally found in human beings. People who find resonant heroic themes of challenges and questing in their own lives, in their goals, creative outpourings, in their day- and night-dreams—are being led to a single psychic fact. That is, that the creative and spiritual lives of individuals influence the outer world as much as the mythic world influences the individual.

One common monomyth focuses on the theme of death and rebirth. Another, according to Campbell, is “the hero’s journey” – in which the central character matures from a state of powerlessness to courage and self-assurance (in the form of a character arc).

In this lesson we will also examine other forms of archetypal literary criticism including Northrop Frye's Archetypal criticism:

  • Summer = Romance. The birth of the hero
  • Autumn = Tragedy. Movement towards the death or defeat of the hero
  • Winter = Irony or satire. The hero is absent
  • Spring = Comedy. The rebirth of the hero

Character and Worldbuilding

Worldbuilding is the process of constructing an imaginary world, sometimes associated with a whole fictional universe. Developing an imaginary setting with coherent qualities such as a history, geography, and ecology is a key task for many science fiction or fantasy writers. Worldbuilding often involves the creation of maps, a backstory, a flora, a fauna, different peoples and their technology, (And in many cases if one is writing speculative fiction, different races), including social customs and, in some cases, an invented language for the world

Lots of fantasy readers like referring to a physical map whenever the characters are going somewhere new. Maps not always necessary, but they’re useful for defining a sense of distance and space — and they can help you visualize your world as you’re building it.

8 Tips for Worldbuilding according to the Masterclass Staff

  • Decide where to start. Whether it’s the language spoken by the inhabitants or the apocalyptic landscape, pick the aspect of the world you’re most excited about exploring and start there.
  • List the rules and laws. The inhabitants who live in this world you’ve created will have their own independent existence. What is their governing system? Who is in charge?
  • Establish the type of world you want. Pick a genre. Is this a dystopian or fantasy novel (or both)? Does it take place in our Earth or is it an alternate earth?
  • Describe the environment. What’s the weather like? How does it affect the world or planet? Are there natural disasters? Are there extreme temperatures? What natural resources exist in this location? How do people use the land? Establishing the environment and how it impacts the life within it can be a useful detail in the creation of your world.
  • Define the culture. What do the inhabitants of this universe believe in? Is there religion? Is there a God? Do they have any sacred customs? What do they celebrate? Breathe life into the characters who populate this location by giving them a meaningful existence.
  • Define the language. How do the inhabitants communicate? Is there a common tongue? Are there any ‘bad words’ that are off-limits? Knowing what can and can not be said in your world can be an apt source for conflict.
  • Identify the history. What is the history of this place you’ve created? Have there been any world wars? Do the countries within your world have enemies? Are there rival races? Is there a sole antagonist? Providing the backstory for your world can give it an added dimension and make it feel more tangible.
  • Use existing works to inspire. Revisit the works of successful authors to get inspiration. Never steal ideas, but review the work of other fiction writers to see how they answer the same worldbuilding questions within their own novel writing.
For multimedia, the visual treatment gives the reader a rich understanding of the characters, the plot, and the visual style of the work that represents the world of the story, the tone of the story, and its aesthetic style.

Characterization

The Character and the Novel

The novel expanded on the inner consciousness and psychological interiority of the characters. This also resembles the spreading of literacy and “silent reading” that developed from the printing press, as well as the values of secularity and humanism that were celebrated during the Renaissance: people started to be seen more than just through religion or their class status, and their thoughts were worthy of being considered.

The modern novel distinguished between the authorial and narrative voice: what the characters thought and felt vs. what the novelist was trying to communicate overall. Sometimes, these two voices could be in complete opposition (an unreliable narrator) This led to a more complex form of reading.

Characterization can be achieved directly, through the statements of the narrator (or another character) about the character, or indirectly, as when the reader deduces from actions, speech, or context key traits of the character.

In film, comics, and video- game narration, the visual presentation of physical traits, dress, facial expressions, and postures contribute significantly to characterization. These modes of narration can achieve rich characterization without using words at all, even when the characters are nonhuman entities or animals

Some characters appear with a passage of detailed personal description, including gender, race, appearance, age, dress, social position, and past experiences (as in a reference to a character’s recent divorce). This strategy of block characterization can be avoided by employing instead indirect or scattered brief references to a character’s physical qualities and social identity.

“FLAT” AND “ROUND” CHARACTERS FROM E.M. FORSTER’S ASPECTS OF THE NOVEL

Both kinds of characters may start as types, but the flat character usually does not transcend the typical. Flat characters do not change; they possess a fixed set of traits, often a catchphrase, and they are comfortingly predictable in their functions. While it can be a damning criticism to judge a central character in a narrative ‘flat,’ in fact flatness is a desirable trait in a minor (foil) character
Round characters are capable of surprising the reader in a convincing way. This suggests the complexity and appearance of psychological depth of central or major characters. One way that a writer can achieve this surprising complexity is to begin with one type (the beautiful young love interest) and to fuse it with another type and its possibilities (the action- adventure hero).

Non-linearity, Interactivity, and Engagement

Interactivity is the communication process that takes place between humans and computer software. It refers to an involvement in the exchange of communication.

Interactivity opens the narrative system from a director/viewer relationship to one in which the viewer becomes a viewer/maker. In these works, the viewer both experiences and “makes” the work by participating in options that shift the course of the narrative.

Two poles of interactive states are immersion and engagement.

Immersion results largely from two essentials that work together to create a state in which readers are totally absorbed with the text: First, a lack of demands placed on the reader by the physical act of reading creates a “steady unbroken rhythm” that occupies the mind completely.

Immersion is the experience that results from what Jay Bolter and Richard Grusin describe as transparent immediacy in art and media: the tendency to overcome or dissolve the sense of mediation and make something as realistic as possible. In the story, this makes the movement between paths and through the story somehow seamless and without obvious transitional devices.

Hypermediacy is a tendency in which the technological process is emphasized rather imperceivable. For example, multiple, simultaneous windows into a story offer a less realistic but comparative and holistic perspective for the viewer.The story reveals and complements the extensive range of paths and links available to the reader. When an art work or story is self-reflexive and refers to other artistic tendencies outside of itself this is hypermediacy.

Hypermediacy produces an engaged state: When the audience/readers must go outside the framework of the schema, when we find ourselves grappling with the way the story is presented, when we must work to make sense of the schema, then we come out of the immersive state and we become engaged. When we are engaged our attention is brought to the material itself and we move in and out of the frame of the narrative.

Non-Linearity

Non-linearity is characterized by a comparatively more open-ended structure- there is no critical linear path, and the end user discovers key elements in the story as they reveal themselves, or as they are found

Nonlinear narrative, disjointed narrative or disrupted narrative is a narrative technique, sometimes used in literature, film, hypertext websites and other narratives, where events are portrayed, for example, out of chronological order or in other ways where the narrative does not follow the direct causality pattern of the events featured, such as parallel distinctive plot lines, dream immersions or narrating another story inside the main plot-line. It is often used to mimic the structure and recall of human memory, but has been applied for other reasons as well.

One example of non-linearity in the novel occurs in modernist novels, ie. Virginia Woolf's stream of consciousness device. However, films lilke Memento or really any flashback device can be considered a form of non-linearity.

Non-linearity is not new. In "The Ultimate Interactive Storyteller," J.F. Culhane describes four contemporary non-linear devices that can be comparable to similar devices used in the oral storytelling tradition:

  • A narrative trigger is a discrepancy or a point in which the listener, viewer, receptor, or user, reaches a temporal comparison of differences or similarities.
  • The Fractal storyteller - When the interaction takes place in an environment that slowly changes its conditions as in an “artificially living scene over time”. You lack a predictable day or night, or there may be unpredictable weather conditions that effect the narrative. Can depend on random algorithms.
  • Harmonic Paths of Multiple Camera- Simultaneous channels telling the same meta narrative. Stories can be told from different visual or character perspectives.
  • AI and “the Smart Storyteller”- A learning-capable narrator that reacts to your behavior and adapts the narrative accordingly. May be an unreliable narrator
The narrative trigger, or a discrepancy or a point in which the listener, viewer, receptor, or user, reaches a temporal comparison of differences or similarities, is demonstrated in the movie A Princess Bride. This technique can be located in the traditional oral storytelling form through questions and interruptions by the listeners.
Fractal storytelling is emphasized in stories that repeat single moments in time with added details each instance, such as Groundhog Day and Russian Doll. However, fractal storytelling, or stories within stories can also be located in many other works and especially videogames.
Harmonic paths can include Stories told from different perspectives (Lost, Haunting of Hill House, Pulp Fiction). They are featured in multi-channel video games or interactive movies. In this nine-screen multi-channel video installation titled "The Visitors" by Ragnar Kjartansson, a musical performance is staged at historic Rokeby Farm in upstate New York, where the artist has been a frequent visitor since 2007. Kjartansson assembled a group of friends to help him produce this work; all the musicians recorded the hypnotic musical arrangement in a different setting of the home. When all of these single takes are combined, the performance emerges and they make sense as a whole. (Guggenheim-Bilbao)
The smart storyteller reacts and reflects decisions or characteristics of the reader/audience, such as an interactive choose your own adventure. Perhaps nothing in contemporary times can better reflect the smart storyteller than the Facebook and social media algorithms that organize and present content based on the user's previous behavior.

Visual Storytelling

  • What is unique to the medium of photography?
  • How has photography been used to tell stories throughout the 19th and 20th centuries?
  • How does the photograph function as a language?
  • What are historical parallels with analog photography and today’s digital environment?

The Invention of Photography

The invention of photography grew out of an interest in the structure of light, and how to capture it onto light-sensitive material. This interest goes back to the history of civilization.

An interest in the physical properties of optics was also an influence. From ancient China to ancient Greece to the Renaissance, artists and scientists experimented with the phenomenon that through a pin-hole in a wall, an image is projected onto a wall or screen behind it (but in reverse and upside down). This phenomenon is discussed in ancient Chinese texts as well as in the writings of Aristotle.

Extremely important to the invention of photography was knowledge of how sensitive to light certain materials were. More than 2,000 years before the invention of the camera obscura, the ancient Phoenicians (the first civilized nation in the world) knew that a certain snail left a yellow slime that turned purple in sunlight.

"View from the Window at Le Gras, France” (1826) The birth of photography happened in 1826 when a French scientist, Joseph Nicephore Niepce, put a plate coated with bitumen (an asphalt used in ancient times as a cement or mortar) in a camera obscura. He put the camera obscura facing his house for eight hours and made a photograph. It is the earliest camera photograph that we still have today.
GUSTAV OEHME. Three Young Girls, c. 1840s. Daguerreotypes were introduced to the world in 1839 when the French government bought the patent and made it free to use. The technology was used widely in the 1840s and 1850s before being superseded in 1860 by other photographic technology. Printed on copper plates, and originally produced using mercury There were no negatives to reproduce the image; there was only one single daguerreotype (Although unique images, they could be copied by re-daguerreotyping the original, with a decimated quality)

Famous Daguerreotypes

Documenting Reality: 20th Century Photography

  • During the 20th century, documentary photography became a prominent force in society

Photojournalism and the Photographic Essay

  • How does photography function with text to tell a story?
  • What are different storytelling techniques of the photo essay?
  • How has photo-journalism influenced American society?

Photographic Theory

  • Effects of eproducibility
  • The semiotics of the photograph (what is its language, or message)
  • Ethics of photo taking
  • Publicity and the “hyperreal” image
  • Social capital
  • Exploitation vs. Critique
  • Taking pictures as non-intervention
  • Post-colonialist critique of the gaze
  • The psychology of the image
  • "Post-truth" and hyperrealism

Cinematography and Editing

ELECTRICITY & THE ART OF LIGHT

  • Cinema was a product of the second Industrial Revolution, a phase of rapid standardization and industrialization from the late 19th century into the early 20th century.
  • Electrification was called "the most important engineering achievement of the 20th century" by the National Academy of Engineering.
  • The technology of film emerged mostly from developments and achievements in the fields of projection, lenses, photography and optics.

Early techniques that involve motion and/or projection include:

  • Shadowgraphy (probably in practice since prehistoric times)
  • Camera obscura (a natural phenomenon that has possibly been used as an artistic aid since prehistoric times)
  • Shadow puppetry (possibly originated around 200 BCE in Central Asia, India, Indonesia or China)
  • Magic lantern (developed in the 1650s, preceded by some incidental and/or inferior projectors)
  • Stroboscopic "persistence of vision" animation devices (phénakisticope since 1832, zoetrope since 1866, flip book since 1868)
  • Stereoscope: Hand-held viewers that created three dimensional effects by using oblong cards with two photographs printed side by side
EARLY INFLUENCES: ZOETROPE (1833)

THE NOVELTY OF CINEMA

  • Cinema as a recent novelty in the field of entertainment was officially recognized in the 1900 universal exhibition held in Paris.
  • Since the mid 19th century, the World Fairs became places for people to display the invention of new scientific, cultural and industrial items
  • What was valorized in the Exhibition was 'innovation and technical development,' rather than the quality of films or company's commercial success. Nor was cinema seen as an instrument in favor of education and teaching and much less as a work of art.

MUYBRIDGE’S RUNNING HORSES (1878) — MAJOR PRE-CURSOR OF MOTION PICTURE

  • Muybridge was a photographer who competed in a challenge to answer the question whether horses lift all four feet up in the air at the same time
  • Set up a row of 12 cameras, each making an exposure in one-thousandth of a Second
  • He was able to take crisp, clear images of motion due to the development of faster shutter speeds on cameras
  • He used a lantern to project the images which were eventually drawings copied from the photographs onto a revolving disk
  • Demonstrates the cutting of movement into individual ”frames”
  • Crisp, clear images, when aligned already suggests movement

HISTORY OF EARLY FILM

  • Edison Labs
  • Lumiere Brothers
  • Single Shot vs. the Cut
  • George Melies: Science Fiction
  • D.W. Griffith: “Birth of a Nation”

Inventions: Edison Labs

Edison's nickelodeon, which became popular in the early 1900s
  • Out of his labs in New Jersey, Thomas Edison developed some of the first movie technology
  • 1894: Thomas Edison opens first kinetoscope parlor
  • 1895: Edison’s Kinetoscope evolved into Vitascope (life scope) with Thomas Armat
  • 1894: American Mutoscope Company

The Moving Image Industry

Hollywood: A Short History

The Hollywood Studio Age: 1932 - 1946

  • The period 1920-50 marked the “Golden years” of the Hollywood studio system. Films were produced rapidly and regularly. Often following a formula: Western, slapstick comedy, Film Noir, musical, cartoon, biopic… depending on the studio.
  • The increased costs of movie / film making had created the Studio system with its complex financing and control systems designed to ensure that the mix of expensive technology, cast of actors and associated technicians and financiers would return a profit.
  • There is a commercial differentiation between “movies” and “films”. Movies = popular entertainment with a mass circulation of copies of the movie. The audience being largely passive.
  • The product of an industry dominated by the producer (money) in which there is no individual film-maker but a team under the producer’s control. (The studio system.) The director is hired to create the movie from the script. The final version is, however, the responsibility of the producer and editor.
  • The director of a movie is known as: metteur en scene = an interpreter of a score / script

Hollywood Studio Decline & Antitrust Lawsuits: 1947–1959

  • In the early 1930’s, the Justice Department found that the major studios were vertically-integrated monopolies. One monopolistic practice was block booking: licensing one feature film or group of feature films on the condition that the licensee-exhibitor will also license another released by the same distributor. Block booking prevents customers from bidding for individual feature films on their own merits.
  • In 1946 the Department of Justice brought an antitrust lawsuit against the eight of the studios alleging violations of Sherman 1 and 2—restraint of trade and monopolization.
  • Antitrust laws are statutes or regulations designed to promote free and open markets. Antitrust laws were created to keep the big conglomerates, or trusts, that were forming across oil, railroad, steel, and other sectors in the late 1800s and early 1900s from growing too large and powerful.
  • 1946-51: Studios ordered to divest theatre chains - reluctance meant that they failed to capitalize on the possibilities of the TV as a medium for film.
  • The Antitrust lawsuits, along with the emergence of television, depleted the box office numbers. WWII delayed the TV age but post war the threat of TV re-emerged. (We will talk more about TV in Module 8 on Video) Movie audiences peaked in 1946 – 80 million tickets sold per week. By 1953, ticket sales drop to 46 million per week.

“New Hollywood”: Mid 1960s- 1980s

  • “New Hollywood” refers to a new generation of young filmmakers came to prominence in the United States. They influenced the types of film produced, their production and marketing, and the way major studios approached film-making. In New Hollywood films, the film director, rather than the studio, took on a key authorial role.
  • The films in the 1970s came in many different varieties, as the socially conscious directors that emerged in the late 1960s grew in very different ways, influenced by music, literature, crime and war. The decade is most known for excelling in the crime-drama genre.
  • In the years previous to 1970, Hollywood had begun to cater to the younger generation with films such as The Graduate. This proved a folly when anti-war films like R. P. M. and The Strawberry Statement became major box-office flops. Even solid films with bankable stars, like the Pearl Harbor epic Tora! Tora! Tora!, flopped, leaving studios in dire straits financially. Unable to repay financiers, studios began selling off land, furniture, clothing, and sets acquired over years of production. Nostalgic fans bid on merchandise and collectibles ranging from Judy Garland's sparkling red shoes to MGM's own back lots.
  • More of the successful films were those based in the harsh truths of war, rather than the excesses of the 1960s.
  • With young filmmakers taking greater risks and restrictions regarding language and sexuality lifting, Hollywood produced some of its most critically acclaimed and financially successful films since its "golden era.”
  • The 1970s saw a re-birth of Kung Fu, gritty crime dramas, horror films
  • In the middle 1970s, films began to also reflect the disenfranchisement brought by the excesses of the past twenty years. Ie. Taxi Driver

Todd Berliner on "New Hollywood":

  • Seventies films show a perverse tendency to integrate, in narrative incidental ways, story information and stylistic devices counterproductive to the films' overt and essential narrative purposes.
  • Hollywood filmmakers of the 1970s often situate their film-making practices in between those of classical Hollywood and those of European and Asian art cinema.
  • Seventies films prompt spectator responses more uncertain and discomforting than those of more typical Hollywood cinema.
  • Seventies narratives place an uncommon emphasis on irresolution, particularly at the moment of climax or in epilogues, when more conventional Hollywood movies busy themselves tying up loose ends.
  • Seventies cinema hinders narrative linearity and momentum and scuttles its potential to generate suspense and excitement.

Postmodern Cinema: 1990s

  • Postmodernist films attempt to subvert the mainstream conventions of narrative structure and characterization, and tests the audience's suspension of disbelief.
  • Typically, such films also break down the cultural divide between high and low art and often upend typical portrayals of gender, race, class, genre, and time with the goal of creating something that does not abide by traditional narrative expression
  • Can experiment with non-linearity
  • Can use different styles simultaneously (postmodern “pastiche”)
  • Meaning is highly subjective
  • Playful
  • Recognize the vast media culture and the media history that structures our reality and our cinematic imagination

From Script to Screen

  • Screenwriting is writing for the screen. (Recently, scriptwriting has come to refer to writing for video games)
  • A screenplay is the written text of the drama: the guide, or map of how the story on the page will be realized into the story on the screen.
  • A screenplay is a blueprint for production, a process that is collaborative, business-oriented and elaborate.
  • It is written in a specific way and to a specific style, carefully guiding the reader through the unfolding narrative.

Narrative Structure of the Screenplay

  • Screenplays are built around a narrative- a sequence of events made up of various constituents that come together to create the whole
  • Constituents include character, dialogue, and action
  • There are two levels to narrative structure: story and plot. The story is the emotional character arc, and the plot is the order of events.
  • “Structure and character are interlocked. The event structure of a story is created out of the choices that characters make under pressure and the actions they choose to take, while characters are the creatures who are revealed and change by how they choose to act under pressure. If you change one, you change the other (McKee, 1999)
  • The term “journey” is often used to describe the road the characters travel from start to finish (physical journey, emotional journey)

Three Act Structure

  • A base for all screenwriting models. Roots in Greek tragedy- beginning, middle, end.
  • Can be based on sequences or mapping the “hero’s journey”, etc.
  • Act 1: Setup (1/4)
  • Act 2: Confrontation (1/2)
  • Act 3: Resolution (1/4)

How to set action in motion

Film Criticism & Ways to Analyze Films

1970s-80s: The Birth of Film Studies

  • Video made it possible to study the history of film
  • Film Studies Coincided with philosophers like Derrida and Foucault who developed historiography and archive studies
  • Coincided with post-structuralist philosophy like Jacques Lacan
  • Christian Metz greatly influenced structuralist / formalist / Semiotic analysis of film as language
  • Film theory includes Feminist film theory, post-colonialist film theory, psychoanalytical film theory, Marxist film theory

Feminist Film Theory

  • Feminist Film Theory emerged out of the second wave of feminism and largely developed by the introduction to Women’s Studies in 1960 and 1970.
  • In the 1970s, the women’s movement aimed to increase the presence and agency of women, whilst film studies recognized the value of visibility and popular culture within what is essentially a political project.
  • More recently, scholars have expanded their work to include analysis of television and digital media.
  • Analysis inquires into: the representations of the female gender; the representation of female characters; the status of women in the film industry
  • Feminist Film Theory also concerns itself with identity – how we create identity- particularly around gender. An example of this juxtaposition lies in the kind of passive roles assigned to women that are meant to expose or solidify the opposite “masculinity” which the male character or heteronormative character portrays. In modern film structure, male’s dominance or “maleness” is defined in contrast to the passive feminine role.
  • Feminist Film Theory also looks at how domestic violence is portrayed on screen or how it is left out, but generally, how women’s issues such as rape and domestic violence are conveyed through film
  • Feminist Film Theory must involve itself in intersectionality. It examines how women of colour are portrayed next to their white female counterparts as well. Or begs the question- where are the shows that have a woman with a disability as the protagonist? Feminist Film Theory also might look at how women of a particular colour or body type are cast and in which kinds of roles?

Laura Mulvey “Visual Pleasure and Narrative Cinema” (1975)

  • Mulvey argued that historically, the camera follows the gaze of the male character while the female characters are often the object of the voyeurism. Studied examples of Alfred Hitchcock’s work
  • Mulvey discusses the ‘Male Gaze’ — mainstream cinema is constructed around a series of ‘gazes’ for male pleasure.
  • This includes the characterization, the camerawork, and the narratives
  • Females are objectified for the (sexual) pleasure of the male viewer. Females only experience narrative secondary by identifying as a male
  • Voyeurism is the pleasure of looking and plays a huge part in the sexual objectification of women in film. It’s the third person in the room- the audience who is capable and meant to identify with the male character, and in viewing the female is able to have her for their own viewing pleasure.

The Bechdel Test

  • The Bechdel test first appeared in 1985 in Alison Bechdel's comic strip “Dykes To Watch Out For’. In a strip titled "The Rule,” which features two women, who resemble the future, the women discuss seeing a film. One woman explains that she only goes to a movie if it satisfies the following requirements:
  • Are there two or more women who have names in the movie?
  • Do they talk to each other?
  • Do they talk to each other about something other than a man?
  • This argues that the majority of screen time of female characters consisted of them talking about male characters. The film strip was acknowledged by a wide array of theorists and writers

Bell Hooks

  • Hooks is an impressive feminist author has written significantly on feminism as it pertains to women of color, contributing greatly to Critical Race Theory and Intersectional Feminist theory
  • In her book Reel to Real, hooks discusses the effect that movies have on any given individual, with specific emphasis on the black female spectator. She argues that, although we know that movies are not real life, "no matter how sophisticated our strategies of critique and intervention, [we] are usually seduced, at least for a time, by the images we see on the screen. They have power over us, and we have no power over them.”
  • In her book Black Looks: Race and Representation, in the chapter "The Oppositional Gaze: Black Female Spectators", Hooks discusses what she calls an "oppositional gaze". She discusses it as a position and strategy for black people, especially black women, to develop a critical spectatorship in relation to mass media.
  • Describing how for her, the "gaze" had always been political, hooks explains how she began to grow curious of the results of black slaves being punished for looking at their white owners. She wondered how much had been absorbed and carried on through the generations to affect not only black parenting, but black spectatorship as well.
  • Drawing on Foucault’s thoughts about power always coexisting with the possibility of resistance, hooks discusses this looking as a form of resistance, as a way of finding agency, and declaring: "Not only will I stare. I want my look to change reality

Postcolonial and Race Theory

  • Postcolonialism = the relationship between nations/ areas that have been ruled / colonized.
  • Postcolonialism refers to the time after the period of colonialism, (mainly through flashbacks).
  • It is often rendered controversial and relates to identity politics.
  • Postcolonial theory addresses identity matters, race, racism, ethnicity and gender.
  • It deals with resistance, conflict, challenges to western schools of thought (including religion and law, creativity/ capitalism and spirituality).

Structuralist / Formalist Critique

  • The theory explores, or demystifies, how films create meaning through codes and conventions (or semantics and content
  • This is built from Roland Barthes’ analysis of the photographic image: Denotation / connotation ; Sign / signified
  • A shot of someone looking sad, then a shot to a glass of water and back to the person, the audience would understand that the person was thirsty.
  • The use of editing, a particular shot or lighting technique can all emphasize a meaning, emotion or reaction from the audience.
  • How the different elements create an effect and have a style. Film is art and therefore is not a reproduction of reality; it is a constructed illusion.
  • Auteurs create their own style - for example Quentin Tarantino and his use of dialogue and close ups and how they communicate ideas, themes and an emotive response to their audience / fans.

Psychoanalytic Theory

  • Developed in the 1970-80s associated with critical theory which analyses film from a psychoanalytical standpoint.
  • The viewer is identified as the subject of the ‘gaze’ what is constructed by the text.
  • What is on screen (mise en scene) is the object of the subject’s desire.
  • Identification is normally through the male protagonist for the subject / viewer.
  • The theory argues that the film seems to offer a completeness to the subject / viewer, although this is always an illusion (as films are merely constructions of reality).
  • Jacques Lacan (French-Freudian-Philosopher 1901-1981)
  • The main idea is that the theory deconstructs both the spectacle of cinema and the elements of film, which are both shaped by the unconscious.
  • The unconscious has been broken down into four areas:
  • The Filmmaker’s Unconscious ; The Character’s Unconscious ; The Audience’s Unconscious ; The Unconsciousness of Cinematic Discourse.
  • It is also worth looking at Freudian theory in relation to this.

The Future of Cinema

  • In an essay written in 1955, titled The Cinema of the Future, cinematographer Morton Heilig predicted that filmmaking would advance to the point where it can “reveal the new scientific world to man in the full sensual vividness and dynamic vitality of his consciousness.” Heilig outlined many of the properties of virtual reality – but didn’t use those words, given they hadn’t been coined yet.
  • With many filmmakers having swapped traditional cameras for 360 cameras (that capture views from all angles), the current moment is comparable to the intensely experimental early years of motion pictures during the late 1800s and early 1900s.
  • According to VR guru and artist Chris Milk, films of the future will offer tailored immersive experiences. They will, he tells BBC Culture, be capable of “crafting a story in real-time, that is just for you, that uniquely satisfies you and what your likes and dislikes are.” Milk prefers terms like “story living” over standard nomenclature such as “storytelling."
  • An array of rapidly developing technologies offer thrilling potential for the future of motion pictures, some suggestions including
  • The rise of AR (augmented reality)
  • AI (artificial intelligence)
  • The ever-increasing capacity for computers to power detailed digital worlds.
  • “Volumetric” (Beyond single flat screens)
  • Neural Diversity

Video: Liveness, Feedback, and Remix

A Brief History of Broadcasting: Radio and Live Broadcasting

Live radio is radio broadcast without delay. Live radio is sound transmitted by radio waves, as the sound happens. Before the days of television, audiences listened to live dramas, comedies, quiz shows and concerts on the radio much the same way that they now do on television. Most talk radio is live radio where people can speak (anonymously) about their opinions and lives.
The radio era, known as “the shortest golden age in history” lasted from 1930 to 1955.

The first radio transmission was made from a temporary station set up by Guglielmo Marconi in 1895 on the Isle of Wight. The radio broadcasting of music and talk intended to reach a dispersed audience started experimentally around 1905–1906, and commercially around 1920 to 1923. VHF (very high frequency) stations started 30 to 35 years later.

Radio broadcasters were not allowed access to congress until the 1930s. The newspaper industry attempted to cast radio as unreliable and unpredictable. It exaggerated the story of The War of the Worlds radio broadcast (1938) as having caused chaos and heart attacks

Early fears about radio included radiophobia (a fear of ionizing radiation); attention distraction; unwholesome content; incitement (radio was the preferred communication method of Hitler; and foreign interference.

NBC and CBS began their empires in the radio business, using the mass media and emphasizing the value of liveness. These companies and supporters of the "American system of broadcasting" defined radio as "commercial, national, live, and network on economic, technological, aesthetic and legislative levels." As of 1929, NBC announced its pride and superiority among radio program companies stating that live broadcast was superior to recorded programs.

Live broadcast brought about a sense of spontaneity and immediacy, and in cases of live music intimacy as you hear the artist as you would live in concert. NBC asserted that the imperfections of live radio added to its authenticity and the pleasure of listening

From Radio to Television

Until the late 1930s, radio was woven into the fabric of American life. It allowed millions to enjoy public events in the comfort of their own home. Adding picture was the next logical step. Television was a natural development out of radio; it was adding picture to the sound broadcast. The same radio broadcasting stations (NBC, CBS) were equipped to adapt to television. Many of their radio shows were reformatted for television including soap operas and talk shows.

While Philo T. Farnsworth designed the first prototype of television in the 1920s and David Sarnoff launched the NBC television station as an experiment in 1938, the licensing of new TV stations was suspended in 1948-1952 due to World War II. It was only until after the war that commercial television took off in America.

Television’s success was bolstered by the 1950s Antitrust actions by the federal government disintegrated the motion picture industry’s control of motion picture houses. At the same time, the Federal Communications Commission (FCC) began licensing television stations on VHF bands; within a decade the FCC had granted licenses to 10 additional commercial broadcasters.

The fact that many people watch television at home while they are multi-tasking, and that people would channel surf until something grabbed their attention, the role of sound is particularly important to television. Additionally, the earliest televisions did not have good picture; the sound was still the dominant modality.

Video Technology

The mediocre quality of kinescopes led the networks and other corporations to commit considerable resources to coming up with something better. That something turned out to be videotape.

In 1956, the Ampex VR 1000 “quad” video tape recorder (VTR) was developed and released for the broadcast television industry by an American company based in Redwood City, California. The two-inch quadruplex videotape (also called 2″ quad, or just quad, for short) becomes the first practical and commercially successful analog recording videotape format. This physical technology led to a change in organizational technology by allowing high-quality television production to happen away from the New York studios. It is essentially the first videotape recorder, but because of its $50,000 price, the Ampex VRX-1000 could be afforded only by the television networks and the largest individual stations. It wouldn’t be until the 1970s that video technology would be available and affordable to household consumers

By 1956, engineers had developed and perfected a method of capturing both audio and video on magnetic tape. Just as audio tape had been used to pre-record radio programs, TV shows could now be preserved on videotape. It was no longer necessary to quickly develop and print a kinescope film for airing a few hours later. A videotape could merely be rewound and played back immediately. The savings in both time and cost were substantial. And as The New York Times noted at the time, “Tape recordings and ‘live’ broadcasts will be so nearly the same in quality that a viewer would be hard put to tell the difference.”

Videotape technology allowed for television networks to capture and rebroadcast pre-recorded content much more easily, which encouraged the practice among the networks.

VHS & Home Recording

  • By the late 1970s: Videocassette Recorder (VCR) becomes household appliance.
  • Use of a VCR to record a television program to play back at a more convenient time is commonly referred to as timeshifting. VCRs can also play back prerecorded tapes. In the 1980s and 1990s, prerecorded videotapes were widely available for purchase and rental, and blank tapes were sold to make recordings.
  • Movie studios fight spread of VCRs, but 1984 Supreme Court decision says consumers can make recordings for own use.
  • 21st Century: DVRs, DVDs, on-demand replacing VCR technology.
  • Video can record and transmit at the same time, unlike any other medium

Camcorders and video recorders

  • Although video recorders were actually invented in 1918, it wasn’t until the 1970s that the technology began to shrink to a size more workable for consumers
  • In 1982 JVC and Sony officially announced the creation of the “CAMera/recorder”, or camcorder. Sony’s Betamovie Beta camcorder used the slogan “Inside This Camera Is a VCR” and came to mainstream market in May 1983
  • In 1985, Sony introduced the first chip-based camcorder “Video 8”, and JVC introduced the VHS-C, a more compact alternative to the typical VHS cassette.
  • In 1992, the first color LCD screen replaced the traditional viewfinder that necessitated squinting through a tiny hole to witness a scene.
  • First digital video camcorder introduced in 1995

The Birth of Video Art

Video art emerged during the late 1960s as new consumer video technology such as video tape recorders became available outside corporate broadcasting. Much video art in the medium's heyday experimented formally with the limitations of the video format.

One of the key differences between video art and theatrical cinema is that video art does not necessarily rely on many of the conventions that define theatrical cinema. Video art may not employ the use of actors, may contain no dialogue, may have no discernible narrative or plot, or adhere to any of the other conventions that generally define motion pictures as entertainment. This also distinguishes video art from cinema's subcategories such as avante-garde cinema, short films, or experimental film.

Video art is often said to have begun when artist Nam June Paik (neologist of the term “electronic super highway”) used his new Sony Portapak to shoot footage of Pope Paul VI's procession through New York City in the autumn of 1965. Later that same day, across town in a Greenwich Village cafe, Paik played the tapes and video art was born.
In general, video art appears in two ways: single channel and multi-channel.
An early multi-channel video art work (using several monitors or screens) was "Wipe Cycle" by Ira Schneider and Frank Gillette. Wipe Cycle was first exhibited at the Howard Wise Gallery in New York in 1969 as part of an exhibition titled "TV as a Creative Medium". An installation of nine television screens, Wipe Cycle combined live images of gallery visitors, found footage from commercial television, and shots from pre-recorded tapes. The material was alternated from one monitor to the next in an elaborate choreography.

Marshall McLuhan (1911-1980)

  • The first major communications theorist of how the new media have the power to transform human nature.
  • McLuhan theorized that human history could be divided into four eras: the acoustic age, the literary age, the print age and the electronic age. He outlined the concept in a 1962 book called The Gutenberg Galaxy, which was released just as the television was starting to become popular.
  • He predicted the world was entering the fourth, electronic age, which would be characterized by a community of people brought together by technology. He called it the "global village" and said it would be an age when everyone had access to the same information through technology. (The "global village" could be understood to be the internet)
  • He soon became a TV personality, making regular appearances to explain his theory of why "the medium is the message”; No matter how powerful or persuasive the message, he said, it’s the media that have changed our patterns of thought and behaviour. He also argued that all media is composed of earlier forms of media
  • He also developed the concepts of “Hot” and “Cool” Media, which is somewhat comparable to “immediate” media

Video and Narcissism

Narcissism is the pursuit of gratification from vanity or egotistic admiration of one's idealized self image and attributes. The term originated from Greek mythology, where the young Narcissus fell in love with his own image reflected in a pool of water.
  • In our digital culture, people can become more easily transfixed or stuck in how their own self image is reflected back at them (through screens, mirrors, people, etc.)
  • The Culture of Narcissism: American Life in an Age of Diminishing Expectations (1979) by the cultural historian Christopher Lasch became an immediate best-seller. In it, Lasch proposes that since World War II, post-war America has produced a personality type of "pathological narcissism."
  • It has cut itself off from history and people have retreated into themselves. He uses examples of radical political movements of the 1960s and spiritual cults and movements in the 1970s to illustrate this turn inwards to the self in the guise of “radical individuality."
Narcissism is so incredibly common today that it is no longer considered a “pathology” (social aberrance) because it so common. However, this trajectory to narcissism isn’t a recent phenomenon.
  • In Rosalind Krauss’ influential article “Video: The Aesthetics of Narcissism” (1976) she argues that the medium of video is essentially a narcissistic conduit between one person and their own ego
  • Argues that people get absorbed by their own image on video, in a narcissistic feedback loop of simultaneous video recording
  • Process is marked by retrospection; voyeurism
  • People get stuck in Lacan’s mirror stage: when there is a recognition of one’s self (via the mirror) as the social self in the world.
  • Krauss gives two examples of Video art, demonstrating this ontological property of video, as a feedback loop with the human psyche as a conduit.
Video records and transmits simultaneously. How do people act differently in the process of watching themselves on screen?
Vito Acconci, Centers (1971) —  In Centers, Acconci faces the camera, his head and arm in close-up as he points straight ahead at his own image on the video monitor, attempting to keep his finger focused on the exact center of the screen.
Richard Serra and Nancy Holt, Boomerang (1974) — Serra taped Nancy Holt as she talks and hears her words played back to her after they have been delayed electronically.

Hyperreality: “An American Family” — The first reality show (1971)

  • An American Family is an American television documentary filmed from May 30 through December 31, 1971, and first aired in the United States on the Public Broadcasting Service (PBS) from January 11, 1973, to March 29, 1973.
  • After being edited down from about 300 hours of raw footage, the program of 12 hour-long episodes aired Thursday nights at 9:00 p.m.
  • The groundbreaking documentary is considered the first "reality" series on American television.
  • It was originally intended as a chronicle of the daily life of the Louds, an upper middle-class family in Santa Barbara, California, but ended up documenting the break-up of the family via the separation and subsequent divorce of parents Bill and Pat Loud.
By the end of the shooting of "American Family," the Louds were at the brink of a divorce. This raises the question: did the production affect their marriage, such that what was being watched was a product of the cameras? Pierre Baudrillard refers to this phenomenon as "hyperreality."

“Liveness” Today: Hyperliveness? “Live” Breaking News since 9/11

  • "Live" television has increasingly evolved into a mostly pre-recorded format.
  • However, "live" news has expanded into 24 hour programming on cable news stations, which often feature repeated footage and commentary in order to fill the programming time.
  • After 9/11, Cable News multiplied its “breaking news” banners to the extent that everything became “breaking news”.
  • However, because Cable News runs 24/7, often it replays footage over and over with the same caption of “breaking news.”
This has led to a televisual experience in which viewers go from one crisis to the next, continually focused on the “present” breaking news without contextualization or historicization (Lauren Berlant)

YouTube and Web 2.0

  • In the early 2000s, as the internet and computer technology improved with broadband and faster CPU, people began to watch video on their computer interface, which disrupted the preeminence of watching video on the television
  • 2005 and the Web 2.0 era saw the launch of the website YouTube, which made the viewing of online video much faster and easier; Google Videos, Yahoo! Video, Facebook and Myspace's video functionality use similar technology.
  • YouTube was created by PayPal employees as a video-sharing website where users could upload, share and view content. The Internet domain name "www.youtube.com" was activated on Monday, February 14, 2005 and bought by Google on October 9, 2006
  • In 2006, Time Magazine featured a YouTube screen with a large mirror as its annual 'Person of the Year'. It cited user-created media such as that posted on YouTube and featured the site's originators along with several content creators.

Self-performance and small stories

What kinds of stories do people tell in social media formats? What similarities and differences occur in the narrators’ choice of subject matter and storytelling style?
How are stories embedded in the multilayered contexts of social media? How do different sites, interactive patterns, offline contexts, and participant groups influence the characteristics of social media stories?
What purposes are fulfilled when stories are told in social media? What personal, social, and discourse identities are constructed for narrators and their audiences?

— Questions from “Stories and Social Media in Context” By R. Page

What is Digital Narratology?

Digital narratology was the dominant research approach in the early years of digital interactivity (the 1990s, 2000s) Page broadly defines narratives as:

  • Generated by the recognition of reported events, ordered within a temporal framework.
  • Second, the inferred connections between temporally ordered events are attributed with distinctive degrees of narrativity where causal connections are understood as more narrative-like than pure temporality.
  • Finally, narrativity is associated with sequences that signal a teleological focus, an overarching framework of complication and resolution, or a clearly defined point of closure attributed with interpretive significance.

However, Page argues that it overly focused on interactive fiction and not on the broad range of ways that stories are told through social media today, including contemporary genres such as blogs, digital storytelling, and life narratives. These contemporary examples challenge the three characteristics of traditional narratives.

Streaming music and the continuous performance of taste

  • Streaming music was facilitated by the development of mobile smart phones in 2007 with 24/7 internet access
  • Instead of purchasing and downloading content, users rent access to libraries from which they can consume content. This curbed, to an extent, the problem of music piracy of MP3s.
  • Researchers find that “adoption of streaming leads to very large increases in the quantity and diversity of consumption” esp. in the first months after adoption, with the effects attenuating over time (Datta et al 2018)
  • Sharing of content and the projection of identity on social media is seen as crucial to entice networks of consumers and is linked to Spotify’s initial success (Dewan and Ramaprasad, 2014).
  • Streaming music has led to a culture of “performing” playlists for different contexts, and a new form of cultural labor to be enacted by the user

How do algorithms shape the stories we read (and the stories about us)?

“Humans are lodged within algorithms, and algorithms within humans” — Louise Amoor, 2020
  • Can Algorithms narrate? According to Benjamin N. Jacobsen (2020), yes! Algorithms are fundamentally narrating agents just like humans.
  • Algorithmic Emplotment: Platforms, digital devices, and algorithmic systems proliferating in society today increasingly seek to ‘emplot’ people’s data, ordering it and rendering it meaningful in various ways (adapted from Paul Ricoeur, 1983)
  • For example, users encounter their digital past through a medium that algorithmically mediates and directs which aspects of the past one encounters.
  • Jacobsen examines ”Apple Memories”, focusing on three particular implications of algorithmic emplotment, specifically for the ways in which the self is represented and framed within the feature: inevitable, multiple, and participatory.

Mobile Media, augmented reality, and screen stories

Mobile media are media that register data from or deliver data to an object or user who is in motion. Mobile social media can loosely be considered software, applications, or services accessed through mobile devices that allow users to connect with other people and to share information, news, and content

The mobile phone has a long history, with the first mobile phones being used in the early 1980s. These early mobile phones were bulky and expensive, and only available to consumers in limited numbers. As technology advanced, mobile phones became smaller and cheaper, leading to the widespread availability of modern smartphones in the 2000s. Today, mobile phones are an essential part of everyday life, allowing people to stay connected, access information, and engage with the world around them.

The Smart Phone

A smartphone is a mobile device that combines the features of a cell phone and a computer, including access to the internet and various applications, such as for productivity and entertainment. Smartphones usually feature large, touchscreens and physical buttons, and many models also include GPS, a camera, and certain sensors, such as a compass, accelerometer, and gyroscope. The advent of the smartphone revolutionized the way people communicate by providing instant access to people and information anytime, anywhere. Smartphones enabled people to stay connected more easily, reducing the need to use landline phones and making it easier to make and receive calls, send texts, and use apps such as WhatsApp, Facebook, and Skype.

How is the experience of the mobile interface different from a PC or other technology?

Mobile phones and continuous partial attention

  • People check their phones on average every 12 minutes
  • Through the miniaturization and proliferation of wireless technologies, we now have the ability to become “a node on the network” in order “not to miss anything.” With the world continually at our fingertips, the internet provides an opportunity for the perpetual instant gratification of information and, conversely, a perpetual lack of delaying gratification.
  • Neurologically kept in “an artificial sense of constant crisis.”
  • We don’t notice our own impairment. We are under the impression that our brain can do more than it is capable of. The penalty is not merely our quality of attention while executing tasks but a hampered ability to refocus for the next task.
  • Loss of discretionary time,” the adult’s version of a “free period” during our day.
  • Cell phones encourage hyper-consumerism
  • Reading books for pleasure no longer works for many people

Augmented Reality

Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real-world are "augmented" by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. The overlaid sensory information can be constructive (i.e. additive to the natural environment) or destructive (i.e. masking of the natural environment) and is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment.

Augmented Reality (AR) is used to enhance natural environments or situations and offer perceptually enriched experiences. The primary value of augmented reality is that it brings components of the digital world into a person's perception of the real world, and does so not as a simple display of data, but through the integration of immersive sensations that are perceived as natural parts of an environment. In this way, augmented reality alters one's ongoing perception of a real-world environment, whereas virtual reality completely replaces the user's real world environment with a simulated one. Augmented reality is related to two largely synonymous terms: mixed reality and computer-mediated reality.

One of the most popular ways AR has infiltrated everyday life is through mobile games. In 2016, the AR game "Pokémon Go" became a sensation worldwide, with over 100 million estimated users at its peak, according to CNET. It ended up making more than $2 billion and counting, according to Forbes. The game allowed users to see Pokémon characters bouncing around in their own town. The goal was to capture these pocket monsters, then use them to battle others, locally, in AR gyms.
In 2018, "Harry Potter: Hogwarts Mystery" became the mobile AR gaming sensation. The game lets users see the Hogwarts world around them while having the ability to cast spells, use potions and to learn from Hogwarts teachers. As of this writing, the game had around 10 million downloads in the Google Play store. Researchers are also developing holograms, which can take VR a step further, since holograms can be seen and heard by a crowd of people all at once.

Mobile Technology Representation

  • How do we visually convey the experience of mobile communication?
  • The true depiction of the internet, PCs and mobile technology adds to a sense of immediacy in a movie
  • Entire movies can be shot using iphones
  • Increasingly, the text bubble is augmented on the screen to convey conversations
  • The practice is becoming so common that there are tutorials about how to recreate text messages for video creation: How to create realistic On-Screen Text Messages
  • How does the use of mobile technology relate to product placement / advertising? Tech companies must agree to the depiction. For example, Apple doesn’t let “bad guys” use iPhones in movies “Villains can’t use iphones in movies because it upsets Apple”

Screen Stories

  • Computer screen film or desktop film is a film subgenre where the action takes place entirely on a screen of a computer or a smartphone. It became popular in the 2010s with the growing impact of the internet on everyday lives. The technique is mostly associated with horror and thriller films and is considered to be born from the found footage genre.
  • Examples include the movies Unfriended and Searching produced by Timur Nuruakhitovich Bekmambetov. He has referred to the genre as computer screen films, Screen Life and Screen Movies.
The blinking dots, which was created to hold users’ attention to their phone (as described in the recent documentary The Social Dilemma, 2020) compel a sense of suspense and attention. In the film Searching, the deleted / attempted messages on the part of the main character convey a lot of information about his psychology and relationship to the other characters.
Using the screen video option makes it easy to record searches, text message conversations, and more. How might you imagine writing and creating your own screen story?

Transmedia and Multi-platform storytelling

Transmedia is a concept born from convergence culture.

  • Transmedia storytelling was a concept that emerged in the 2000s, during the period of Web 2.0.
  • It also coincided with remix culture. Important theoretical figures of remix theory include Henry Jenkins, Lawrence Lessig, and Cory Doctorow. Henry Jenkins coined the term ‘Transmedia”.

To Jenkins, Transmedia storytelling "represents a process where integral elements of a fiction get dispersed systematically across multiple delivery channels for the purpose of creating a unified and coordinated entertainment experience. Ideally, each medium makes its own unique contribution to the unfolding of the story.”

“We now live at a moment where every story, image, brand, relationship plays itself out across the maximum number of media platforms, shaped top down by decisions made in corporate boardrooms and bottom up by decisions made in teenagers bedrooms. The concentrated ownership of media conglomerates increases the desirability of properties that can exploit synergies between different parts of the medium system and maximize touch-points with different niches of consumers. The result has been the push towards franchise-building in general and transmedia entertainment in particular.” — Henry Jenkins

What is transmedia storytelling?

  • Transmedia storytelling (also known as transmedia narrative or multiplatform storytelling) represents a process where integral elements of a fiction get dispersed systematically across multiple delivery channels for the purpose of creating a unified and coordinated entertainment experience.
  • Ideally, each medium makes it own unique contribution to the unfolding of the story.
  • The platforms that transmedia storytelling uses are varied and may include digital media, story components, audience interaction, TV, radio, social media, comics, games, books, collectible items, and in particular, alternate reality games (ARG) a type of game that blurs the line between in-game and out-of-game experiences.
  • The many channels allow for the development of layers, sharp changes, and plot turns. Story content is developed through different techniques to engage different people.
  • While the most well-known examples of transmedia storytelling involve the science fiction franchises Star Trek and Star Wars, with action figures, comics, and games, transmedia or multiplatform storytelling now has a place in education, in branding, and in journalistic, non-fiction applications. This type of storytelling can be harnessed through franchises, marketing campaigns, entertainment content, children’s entertainment, educational platforms, interactive media, experimental art, and more.
  • Cross media platforms may include film, online video, games, social media, email messages, voicemail messages, traditional books, comic books, murals, photographs, audio, action figures, t-shirts, and whatever other conceivable platforms exist.
  • Transmedia intellectual properties often take the form of an ARG, which uses the real world as a platform and players (single or multiple) that may change the narrative and story outcome. Multimedia tools allow players to interact with characters in the story that game designers created. ARGs are often free, as in the Lost Experience that accompanied the TV show.
  • Transmedia stories offer a rich, encyclopedic storyworld, which is a detailed construction of context or universe in which a narrative is set. The presence of a storyworld requires more learning and can be mined for further story development.
  • Backstory, side stories, and prequels of the main property may be explored through other platforms and offers opportunities for the development of secondary characters and events, as well as non-linear exposition.
  • Transmedia storytelling builds worlds for, but also with, fans. Fans may play games, watch movies, and buy action figures, but they may also contribute original stories based on the storyworld to fanzines and other outlets.

Characteristics of Transmedia Storytelling

  • Spreadability: The knowledge of and interest in the property is spread through social media and the web, particularly by ardent fans.
  • Drillability: The story also reaches through layers of audience interest to the core group of fans who will promote and spread the story through different platforms. Intrigue is created from complex plot twists and open-endedness as in Lost.
  • Continuity: Uniformity and coherence of language and character across all platforms.
  • Multiplicity: Incoherence across platforms, such as settings in parallel universes, retellings of stories, or original views of characters.
  • Immersion: Good transmedia properties allow the audience to lose themselves in a rich and intriguing storyworld.
  • Extractability: Having immersed themselves in the story, audiences can take some thing or thought into the real world. Examples for movies and games include t-shirts and toys. In journalism, this includes games or other transmedia properties, such as special editions, which allow the news story to live beyond the news cycle and continue to impact the lives of readers.
  • Worldbuilding: The depth of detail that makes a narrative world seem more three dimensional.
  • Seriality: The method for conveying a large amount of information in an interesting way. Seriality does not need to be linear.
  • Subjectivity: Transmedia properties often tell a story from several character perspectives.
  • Performance: Some fans of transmedia efforts become prosumers (producers and consumers), with costumes or crafting their own short-stories for fanzines to extend their vision of the narrative, as exemplified to the maximum by Trekkies. Transmedia storytelling is geared towards user generated content (UGC).

Transmedia & Media Franchises

  • “West Coast-style transmedia: more commonly called Hollywood or franchise transmedia, consists of multiple big pieces of media: feature films, video games, that kind of thing. It’s grounded in big-business commercial storytelling”
  • Transmedia storytelling reflects the economics of media consolidation or what industry observers call "synergy."
  • Modern media companies are horizontally integrated - that is, they hold interests across a range of what were once distinct media industries. A media conglomerate has an incentive to spread its brand or expand its franchises across as many different media platforms as possible.
  • Consider, for example, the comic books published in advance of the release of such films as Batman Begins and Superman Returns by DC (owned by Warner Brothers, the studio that released these films).
  • These comics provided back-story which enhanced the viewer's experience of the film even as they also help to publicize the forthcoming release (thus blurring the line between marketing and entertainment).
  • The current configuration of the entertainment industry makes transmedia expansion an economic imperative, yet the most gifted transmedia artists also surf these marketplace pressures to create a more expansive and immersive story than would have been possible otherwise.

Examples of Transmedia Storytelling

The Blair Witch Project (1999)

The Blair Witch Project was the profitable independent film for over 15 years. It was an early use of an extended experience, coordinated with Mike Monello, marketer and co-founder of Campfire Media. The mythology was created before the film was shot and was used as a shared history for the actors to improvise from about “The Blair Witch." The creators began an online experience before the film was even finished, calling attention to “three missing teenagers” and the mythology of the Blair Witch.

The Matrix Franchise (1999-2021)

The Matrix franchise, key bits of information are conveyed through three live action films, a series of animated shorts, two collections of comic book stories, and several video games. There is no one single source or ur-text where one can turn to gain all of the information needed to comprehend The Matrix universe.”

“The Office” (2005-2013)

Serial Television in the late 1990s and 2000s provided an excellent environment for transmedia narrative. Viewers had to wait weeks in between episodes. Therefore, platforms such as fictional blogs, Twitter accounts, and fan-fiction allowed the viewers to engage further in the story in between television episodes. This also proved to be incredibly beneficial to writers, who were able to study what viewers liked. The feedback loop between fan-contributed content and writing can be especially witnessed in the television series "Lost".

Interactive Stories and Procedural Narrative

What is procedural narrative?

  • Procedural storytelling is essentially the “set of rules” a story follows, not a story itself. Rules like “how” the story is told (from which perspective?) ... Procedural storytelling leads to the creation of unique places and characters that are wholly unique to the player.
  • A transmedia text does not simply disperse information: it provides a set of roles and goals which readers can assume as they enact aspects of the story through their everyday life. We might see this performative dimension at play with the release of action figures which encourage children to construct their own stories about the fictional characters or costumes and role playing games which invite us to immerse ourselves in the world of the fiction.
  • Ideally, each individual episode must be accessible on its own terms even as it makes a unique contribution to the narrative system as a whole. Game designer Neil Young coined the term, "additive comprehension," to refer to the ways that each new texts adds a new piece of information which forces us to revise our understanding of the fiction as a whole.
  • The encyclopedic ambitions of transmedia texts often results in what might be seen as gaps or excesses in the unfolding of the story: that is, they introduce potential plots which can not be fully told or extra details which hint at more than can be revealed.
Procedural narrative is a design technique where a game's story consists of many fragments that can be collected, experienced and interpreted in many different orders and ways. Emergent narrative is the result of that collection and interpretation process — Edwin McRae 

Immersion, VR, and the senses

Presence: “the subjective experience of being in one place or environment when physically situated in another”

Virtual Reality

  • Virtual reality has been presented in the popular press as a medium, like the telephone or television.
  • The term ‘virtual reality’ was coined by Jaron Lanier in 1987 during a period of intense research activity into this form of technology. But before then, he had set up VPL Research – a company which pioneered research into virtual reality and 3D graphics which also sold the first virtual reality gear such as virtual reality glasses, data gloves and later, the full data suit.
  • VPL Research patents were later acquired by Sun Microsystems in 1999.
  • Lanier partnered up with Tom Zimmerman – the inventor of the first data glove and together they produced a range of virtual reality products such as glasses, gloves and a relatively inexpensive head mounted display (HMD).
  • Virtual reality became very popular around this time – especially in the 1990’s, but this soon dropped off due to a yawning gap between public expectations and technological limitations.

Haptic feedback, also known as haptics, is a technology that allows users to interact with digital devices in a more tactile manner. Haptic feedback makes use of vibrations, sounds, and other sensory cues to provide a physical feedback to the user, giving a greater sense of immersion and control, and ultimately providing a heightened sense of realism.

The haptic Touch feedback provides information on texture, temperature. It does not resist user contact. Force feedback provides information on weight and inertia. It can actively resist contact motion.

Virtual art is a term used for the virtualization of art. In other words, it is contemporary art created with the technical media developed at the end of the 1980s — computers, visualization casks, data clothes, data gloves, generators of 3D sound, digital sculpture and painting, stereoscopic screens and spectacles, etc.

Can virtual reality cultivate empathy?
  • First, how do you measure empathy?
  • Stanford’s Virtual Human Interaction Lab has focused an examination of VR’s pro-social uses, exploring how the technology may be used to positively impact society. Its ability to create empathy is measured by participant behavior. For example, those who went through the VR homelessness experience indicated that they would be more willing to support tax increases for affordable housing than before their experience, and they were significantly more likely to support those efforts than those who did not go through the VR experience.
  • However, according to Erick Ramirez in Aeon Magazine, “we should be skeptical of these claims. While VR might help us to cultivate sympathy, it fails to generate true empathy. Although they are often confused with one another, these capacities are distinct. Ramirez distinguishes between them like this: empathy relates to the cognitive and emotional abilities that help us feel with another. Empathy is what we use when we engage in perspective-taking. Sympathy, meanwhile, involves the capacities that help us feel for another. It doesn’t include imagining what it’s like to be someone else.”
  • It’s also important to note that the same arguments about VR and empathy were made about the immersive powers of cinema...
NextPrevious