Computers create pictures and sound

 Last update: 2005/03/08

Most of what has been said so far concerned audio and visual information generated in the real world and reaching human senses either directly or through a more or less transparent communication system. An increasing proportion of what we perceive today through communication devices, however, is no longer generated in that way. Images and sound from computers and game consoles, originally almost exclusively synthetically generated, have an increasing share of naturally generated media, while what we perceive from TV sets and movie theatre screens, originally almost exclusively naturally generated, are increasingly complemented by synthetically generated audio and video.

Because of this trend, the scene would not be complete without a ride on these other bits, particularly because since 1995 MPEG has been operating in this space trying to build bridges between business communities through disciplines proper to them. Purpose of this page is then to clarify the background of synthetically-generated pictures and sound to help understand the role that MPEG has played in it with the MPEG-4 project.

From the beginning, computers were machines capable of connection and control of all sort of devices and thus ideally suited to replace the infinite number of ad-hoc solutions that for centuries humans had conceived to make a machine respond, in a predictable way, to an external stimulus or to generate events directly correlated to its internal state. Besides typical "data processing" peripherals like card readers and line printers, during the years computer mice, joysticks, game pads, track balls, plotters, scanners, were attached to computers. Of more interest for the purpose of this page are other types of devices such as microphones/loudspeakers and video cameras/monitors. 

As early as the 1950s, computers were already connected to oscilloscopes used as display devices and in the 1960s direct-view storage tubes were also connected. In 1963, a film entitled "Simulation of a two-giro gravity attitude control system" was generated by computer at the Bell Labs. By the mid-1960's, major corporations started taking an interest in this field and IBM was the first to make a graphics terminal (IBM 2250) commercially available. The insatiable hunger for connecting all sort of devices to computers is well demonstrated by the computer controlled Head-Mounted Display (HMD). Realised in 1966 at MIT it provided a synthetically-generated stereoscopic 3D view by displaying two separate images, one for each eye. In 1975 Evans & Sutherland developed a frame buffer that could hold a picture. 

In the very same years, after my second return from Japan in October 1973, CSELT had developed a video simulation system with the capacity of 1 Mbyte built with 16 Kbit chips that was capable of capturing and storing a few monochrome and composite (PAL) video frames in real time. The RAM was interfaced to a GP-16, a simple but effective minicomputer manufactured by Selenia (now Alenia, at that time a company of the STET group) and video samples could be transferred to a magnetic tape and read by a mainframe computer. Video coding algorithms were tested on the mainframe and the processed data were again loaded on a tape, transferred from tape to RAM and visualised on the simulation system. If one considers that one cycle of this process could take days, it should not be difficulty to understand why I have used the word "vestals" to describe the people running the mainframe computers in those days. 

The set of techniques used to program a computer so that realistic 3D images can be generated and projected on a 2D surface, using computing resources in an effective way, is called 3D Computer Graphics (CG).  This field has undergone an impressive evolution and is paradigmatic because its academic interest successfully evolved to an eventual commercial exploitation. The development of output devices matched to the needs of viewing the synthetic pictures was a necessary complement to CG's value. The field evolved in a matter of 15 years through a number of milestones.

 

Algorrithm	Description
Hidden-surface	Determines which surfaces are "behind" an object and thus should be "hidden" when the computer creates a 2D image representing what a viewer sees of a 3D scene.
Colour interpolation	Improves the realism of the synthetic image by interpolating across the polygons so reducing the aliasing caused by the sharp edges of the polygons.
Texture Mapping	Takes a 2D image of the surface of an object, and then applies it to a 3D object.
Z-buffer	Speeds up the process of hidden surface removal by using a buffer containing the depth data for every pixel in an image (Z-buffer because Z represents the depth, Y the vertical position and X the horizontal position).
Phong shading	Interpolates the colors over a polygonal surface with accurate reflective highlights and shading.
Fractal	Curves around in a plane in such a way as to cover the entire surface of the plane, to create realistic simulations of natural phenomena such as mountains, coastlines, wood grain, etc.
Ray tracing	Simulates highly reflective surfaces by tracing every ray of light, starting from the viewer's perspective back into the 3D scene. If an object is reflective, the computer follows that ray of light as it bounces off the object or until it hits other objects with an opaque non-reflective surface or leaves the scene.
Radiosity	Uses heat propagation formulae to determine how light reflects between surfaces to overcome the earlier limitations, when image synthesis methods were based on incidental light, i.e. where a light source was shining directly on a surface that world could not reproduce the experience of light diffused or reflected from surfaces as in the real world.

This progress of 3D CG technologies and the reduction in price of computers enabled the establishment of companies. A milestone was reached in 1988 with the Renderman format providing all the information required to render a 3D scene: objects, light sources, cameras, atmospheric effects, etc. 3D CG software developers just had to give the modeling system the capability of producing Renderman compatible scene descriptions to be able to output their content on machines supporting the format. In 1990 AutoDesk introduced Studio3D, a 3D Computer animation product that has risen to a lead position in 3D computer animation software. 

Already in the 1970s CG had entered the world of television and prompted the development of complex hardware and software systems to scan existing artwork, manipulate it, make it squash, stretch, spin, fly around the screen, etc. Morphing, a technique to transform an image of an object into the image of another object, was first demonstrated in 1983 with a video sequence showing a woman transforming herself into the shape of a lynx. In 1991 massive use of 3D CG in movies began: "Toy Story" was the first computer-animated full-length feature film, in "Terminator 2" the evil T-1000 robot was sometimes the real actor Robert Patrick and sometimes a 3D computer animated version and many scenes of "Beauty and the Beast" contained 3D animated objects, flat shaded with bright colors so that they would blend in with the hand-drawn characters. 

In 1994 a group of companies established a consortium called Virtual Reality Modeling Language (VRML) - now Web3D Consortium - with the goal of developing a single format to represent 3D worlds. The first specification, issued in 1997 as VRML 97, provided the coded representation of a 3D space that defined most of the commonly used semantics such as hierarchical transformations, light sources, viewpoints, geometry, animation, fog, material properties, and texture mapping. 

The need to cater for the growing community of researchers and users of 3D CG prompted the establishment of the Special Interest Group on Computer Graphics (SIGGRAPH) of the Association of Computing Machinery (ACM). The first SIGGRAPH conference held in 1973 was attended by 1,200 people, but today the conference has an attendance of tens of thousands of people. 

One could call what has been described so far as the high end of computer graphics, but there is another, originally low- to middle-end application domain, that has given rise to an industry that has used the same computing technologies with an identity of its own: computer games. While the 3D CG field had a more traditional evolution - first academia and then exploitation - the computer game field had a foot in exploitation from the very beginning. Today the progress in computing devices is already blurring the borders between the two fields. 

The first video game - Space War - is said to have been created by an MIT student, in 1961 on the Digital Equipment (DEC) PDP-1. It was very successful and was even used by DEC engineers as a diagnostic program. Five years later the first home video game was created. Its object was to catch spots of light with manually controlled dots. The game was licensed to Magnavox who sold the game for the consumer market under the name Odyssey. In 1971 Space War became the basis for the world's first arcade video game with the name Computer Space. It was too sophisticated for the market of that time and had only limited success but the next arcade video game called Pong - from ping-pong -  released the following year was hugely successful. 

Atari ("atari" is the name of a move in the Japanese game "go" ), the company that developed the first two arcade video games was sold to Warner Communication in 1976. The following year Atari introduced the very successful 2600 VCS, with 2Kbyte of ROM and 128 bytes of RAM. This was the first home game console with multiple games. The pattern of video games hopping from the consumer to the arcade market and vice-versa was set from the early years. In 1979 the 8-bit Atari 800 was introduced and in 1981 home computers such as Apple, Atari, and TRS-80 became popular because dozens of games had been produced for those platforms. 

In 1978 Taito released Space Invaders, the first blockbuster videogame, that made the public aware of video games because they were installed in restaurants and corner stores. It was translated to the Atari 2600 video home game system with equal success. This was followed by a number of third-party development houses: Activision, formed in 1979 by Atari developers, and in the 1980's Epyx, Broderbund, Sierra On-Line and SSI.

In 1980 Philips released Odyssey2, Mattel released Intellivision and Namco released Pac-Man of which more than 300,000 arcade units were sold since introduction, worth more than 1 billion USD. This was a huge hit around the world and is an unforgettable experience of many no-longer-so-young people who have been raised in an environment populated by a long list of computer game names, such as Zork, Donkey Kong, Galaxian, Centipede, Tempest, Ms. Pac-Man and Choplifter!

In 1982 gaming companies that produce the hits of today, such as Access Software, Electronic Arts, and Lucasfilm Games (now LucasArts) were founded. Even Microsoft tried its luck in this field with "Olympic Decathlon", a not particularly successful game, one reason why it took many years before Microsoft would publish another computer game. In 1981 the game industry was worth more than 6 billion USD in sales and Atari alone did 1 billion USD with Asteroids throughout its life span. 

In the first half of the 1980s home computers with game capabilities started being released: the Atari 400 and 800, the Commodore VIC-20 and the Commodore-64 (C-64) of which 20 million units were sold in 1982 alone, the year it was released. Common features of these home computers were: colour display capabilities, composite video output for television sets, and tape units, floppy disk drives and cartridges as storage devices.  In 1984 the cartridge-based systems used, e.g., by the Atari 2600, became suddenly unpopular. The video game industry started losing ground, while home computers by Commodore and others were gaining ground because of the possibility to do other things besides games. The success continued in 1985, with the Commodore 64 outselling Apple's and Atari's computers and in 1985 with the launch of the Amiga personal computer - another unforgettable experience - with many advanced graphic features. 

In 1985 a new player - Nintendo - came to the fore with its Nintendo Entertainment System (NES), characterised by strict control on software, lockout chip, and the restriction to companies to 5 games/year. In the following year another new player - Sega - came to the fore with its Sega Master System console, technically superior to Nintendo, but a market failure because of the lack of games as Sega did not consider third-party developers. 

In 1989 Sega released the 16-bit Genesis console with Electronic Arts sports titles, while Nintendo kept its 8-bits. Nintendo released Super Mario 3, the all-time best-seller, while Amiga and Atari ST died out. In 1991 Nintendo launched the 16-bit Super-NES. The following year Nintendo had 7 billion USD in sales and higher profits than all U.S. movie and TV studios combined. In 1993 Sega and Nintendo consoles held 80% of the game market.

In 1992 PC gaming exploded. In 1993 Panasonic shipped the 32-bit Real console from 3DO. In 1994 Atari shipped the 64-bit Jaguar. In 1995 Sega shipped the 32-bit Saturn and Sony the 32-bit Playstation. Microsoft released Window 95 that included the Game SDK - Direct-X thus bringing major game performance into the folds of Windows. In 1996 Nintendo shipped Ultra 64 and the Internet favoured the growth of multi-player gaming. 

In 1997 3D acceleration started to standardise on 3D-FX, and 3D acceleration became a common game feature. In the meantime Pentium II at 200 MHz started providing serious game experiences. In 1998 a lot of very good PC games appeared, while Playstation ruled in the console domain. An interesting comparison with movies reveals the commonality of the two businesses: 300 games released each year with only 30 making money and 5 billion USD in PC games, about the movie industry's size. 

The first PC colour video adaptor was the 4-color Colour Graphic Adaptor (CGA). The first board for professional applications was the TARGA video adapter for the PC, capable of displaying 32 colours, released by ATT in 1985. In 1988 the 16-colour Enhanced Graphic Adaptor (EGA) graphics was used by Sierra On-Line and the following year the first game that used 256-colour Video Graphic Adaptor (VGA) graphics was published. 

Providing an output for a video signal placed two heavy requirements on computers. The first was  fast CPU to process a high amount of data and the second a large memory because of the need to store at least one screenful of data that could be generated asynchronously by the CPU and read out synchronously and converted to analogue form to drive a display. For this reason it took a considerable amount of time before computers of reasonable cost could provide a video output. 

For audio, matters were much simpler because waveform generation could easily be done in real time or read from a file on a disk, and converted to analogue form to drive a loudspeaker. If the waveform corresponded to a musical score, it was rather easy to provide special hardware designed to produce different types of sound. As an example, the C-64 had a built-in analog synthesiser chip and many games had an obsessive tune accompanying the game that changed depending on the state of the game. In 1989 the first sound cards, the Adlib and Soundblaster, brought a more professional sound to the PC, replacing the original "beep" of the internal speaker. 

The Musical Instrument Digital Interface (MIDI), developed in 1983 by Sequential Circuits and Roland, is a protocol to control electronic musical devices. A MIDI message can tell a synthesiser when to start and stop playing a specific note, the volume of the note, the aftertouch, i.e. the amount of pressure applied on the keys of a given channel, the instrument desired to play on a channel, how to change sounds, master volume, modulation devices, and even how to receive information. In more advanced uses, MIDI information can indicate the starting and stopping points of a song or the metric position within a song. 

Copyright © 2003 chiariglione.org