Module 2 3
- 1. Module 2 & 3 MULTIMEDIA HARDWARE & SOFTWARE Classes of multimedia Applications 1. Streaming Stored Audio and Video 2. Streaming Live Audio and Video 3. Real-Time Interactive Audio and Video 4. Others Streaming Stored Audio and Video • The multimedia content has been prerecorded and stored on a server User may pause, rewind, forward, etc… • The time between the initial request and display start can be 1 to 10 seconds • Constraint: after display start, the playout must be continuous Streaming Live Audio and Video • Similar to traditional broadcast TV/radio, but delivery on the Internet Non-interactive just view/listen • Can not pause or rewind Often combined with multicast The time between the initial request and display start can be up to 10 seconds • Constraint: like stored streaming, after display start, the playout must be continuous Real-Time Interactive Audio and Video • Phone conversation/Video conferencing • Constraint: delay between initial request and display start must be small o Video: <150 ms acceptable o Audio: <150 ms not perceived, <400 ms acceptable • Constraint: after display start, the playout must be continuous Others • • Multimedia sharing applications o Download-and-then-play applications o E.g. Napster, Gnutella, Freenet Distance learning applications o Coordinate video, audio and data o Typically distributed on CDs Audio compression Several techniques:
- 2. • • GSM (13 kbps), G.729(8 kbps), G723.3(6.4 and 5.3kbps) MPEG 1 layer 3 (also known as MP3) o Typical compress rates 96kbps, 128kbps, 160kbps o Very little sound degradation o If file is broken up, each piece is still playable o Complex (psychoacoustic masking, redundancy reduction, and bit reservoir buffering) o 3-minute song (128kbps) : 2.8MB Image compression: JPEG • Divide digitized image in 8x8 pixel blocks • Pixel blocks are transformed into frequency blocks using DCT (Discrete Cosine Transform). This is similar to FFT (Fast Fourier Transform) • The quantization phase limits the precision of the frequency coefficient. • The encoding phase packs this information in a dense fashion Video compression Popular techniques: • MPEG 1 for CD-ROM quality video (1.5Mbps) • MPEG 2 for high quality DVD video (3-6 Mbps) • MPEG 4 for object-oriented video compression We can classify multimedia software as system software and authoring tools used to develop multimedia applications. The former sets up the communications between the system resources and the application programs. The most common example of this class is Multimedia Control Interface (MCI) by Microsoft. Block diagram of this software is shown in the figure below.
- 3. We shall explain the various components of this structure in more detail. Telephony The most important developments in the recent years is the computer telephony integration. This way the reliable telephone technology has collaborated with the innovative computer industry and this generated switches controlled by software and computers connected by the telephone network. MPC level3 standards has defined a Telephony Applications Programming Interface for this integration. Windows supports this interface. TAPI is part of Windows Open Architecture and used to implement applications with computer-telephony interaction such as modem access. Video for Windows This software has been developed by Microsoft to prepare video data to be used in multimedia applications. It is used to compress and decompress and play the video. Microsoft Video I, Intel Indeo, Cinepak or Microsoft run Length Encoding (RLE) are examples of the codecs used for compressing the video data. The videos generated by VfW are stores in AVI (Audio Video Interleaved) format. In this approach the audio data is stored together with the image for each frame. Videos can be captured, edited and played using the set of software included in the VfW. Editing involves cutting, inserting and merging video data to generate a new sequence of frames. VfW uses VidCap to capture video, and VidEdit to edit it. It is a free software which comes with all the source codes. Quicktime This is a system software for Apple Macintosh. It enables the integration of video, audio, animation and graphics in a standard way in multimedia applications. It supports JPEG, Apple Compact Video, Apple Animation Compressor, Real Time Video, MPEG and motion JPEG. A Windows version of Quicktime -Quicktime for Windows- has been developed and the transfer of files between these two platforms has been made possible. In the Windows environment files with MOV extensions are generated using Quicktime. Indeo and Digital Video Interactive
- 4. Indeo is Intel Video (Indeo) technology and defines a file format for converting NTSC and PAL analog signals to digital video which can be stored and played back in a PC. The compression defined in Indeo is Real Time Video which uses a vector quantization scheme similar but less complicated than MPEG. DVI on the other hand defines a CD format based on Intel 80x86 processors. It is similar to CD_I and was developed for interactivity in games and generating special video effects. New Microsoft Multimedia Technologies DirectX by Microsoft is an example of new multimedia architecture designed to improve the application performance of Windows 95. It is a low level API which includes: • • • DirectDraw - composition engine for 2d and 3D video and animation Direct3D - rendering engine for real time 3D graphics DirectSound - this provides device independent access to sound cards
- 5. • • DirectPlay - provides Internet connectivity for multiplayer games DirectInput - provides connectivity to input devices such as joysticks and virtual reality goggles. The block diagram of the above software has been shown in figure 2.7 to give an insight into the interaction among different components. The GDI defined in figure 2.7 is 2D graphics engine for graphical elements such as lines, curves, and other shapes. In addition to these Active Movie has been developed. This is a software developed instead of MCI but it is also compatible with MCI. The next set of software used in multimedia applications is the actual tools used to develop them. It is possible to divide these authoring tools into 4 categories. • • • • Programming languages Presentation software Simple Interactive software Complex interactive software Programming Languages These are the 3rd and 4th generation programming languages such as C/C++, Visual Basic and Delphi. If these are chosen to prepare the multimedia applications then one can achieve maximum control over the contents and developing interactive applications become easy. They require considerable programming experience, however, and they are not easy to learn. The other tools are developed using these languages. Presentation Tools Powerpoint is an example of such presentation tools. We can also include graphics based word processing software where video and audio can be inserted. It is ideal for applications which use text and simple graphics. Limited applications with audio and video can be developed. These tools are fairly easy to learn and are suitable for developing cheap multimedia applications in a short time. Simple Interactive software
- 6. These are suitable for developing multimedia applications where there is interactivity. Buttons or Hypertext can be used to navigate within the application. Most of this software is page based which means the contents and control elements are placed within pages and they are used to go to different parts of the application. Again limited use of digital video and audio is possible. These programs are more difficult to learn compared to the previous class but they do not require much programming experience either. Adobe Acrobat, Astound are examples of this category. Complex Interactive Software These are the most powerful multimedia development tools. It is possible to incorporate programming languages into the applications and improve the control over the navigation and interaction. There are three different types. Page Based In these tools the contents and controls are placed in pages. These pages can be accessed through linked items or controls. Since the educational software is based on page based tools these are very useful in developing educational applications. ToolBook is an example of page based multimedia development software. Time Based These authoring software include a time line where the pages or data components are inserted and display of different components can be controlled either by assigning a duration to each page or manually. This way very successful animations and effects can be produced. The most common examples of these tools are Director and Scala. Icon Based These products use icons which makes it possible to access to the contents or controls. They are useful for producing complex structural applications. Authorware is an example of such a software.
- 7. HARDWARE REQUIREMENTS OF THE MULTIMEDIA COMPUTERS Multimedia computers should support the capture, editing and viewing of video and audio data and provide facilities for the transfer of these data types between computers either through modems or through local and wide area networks. Various parts that should be present in the multimedia computers (PCs in particular) in order to support these functionalities and their specifications have been defined by the Multimedia PC (MPC) group and included in the MPC Level 3 standards. This standards body has first defined MPC standards and then MPC2 and finally parallel to the developments in the computer technology the MPC3 standards. The MPC 3 standards defined the characteristics of the various parts of the computers shown in Figure below. USB : Universal Serial Bus MIDI : Musical Instruments Digital Interface NIC : Network Interface Card We shall be dealing with these parts in more detail in the proceeding sections. Below we have summarized what we think a multimedia PC should have as minimum requirements bearing in mind the developments in the computer industry. You have to remember that even these definitions may change by the time you read them because of the rapid technological developments. • • • • • • Processor: Pentium II 350 MHz Memory: 64 Mbytes Disk : 4 Gbytes CD ROM : 24 speed Sound Card: 16 or 32 bits Graphics card: one with 4 Mbytes of memory
- 8. It is possible to expand this list and enhance the specifications, remember that these are the subjective minimum requirements. Processor: Multimedia applications require powerful processors because of their demanding nature. Applications such as compression or decompression of MPEG (Motion Picture Expert Group) video for example requires that 25 to 30 frames of video should be processed and displayed per second depending on the use of PAL or NTSC standards respectively. The processing involved in the compression and decompression process is fairly complicated and time consuming. In order to meet these requirements either the extended instructions have been used in processors or special multimedia processors have been devised. Intel adopted the former approach and included the MMX technology in its processors. In these series there are extra 57 new instructions, which have been developed with the multimedia applications in mind. One of the instructions for example can do a number of multiplications in parallel and sum their results. Digital signal processing requires such operations and MMX can perform operations on more than one data unit at the same time. This ability is referred to as Single Instruction Multiple Data (SIMD) operation. To be able to achieve this Intel uses a 64 bit register which can be used as units of 8 bits which means that it can hold 8 bits, 16 bits or 32 bits of data and perform operations on the parts independently. Using special microprocessors for multimedia applications is another approach adopted by some manufacturers. Mpact/6000 which is a multimedia accelerator chip developed by Chromatic's is an example of such an approach. Memory: Considering the size of the multimedia applications memory capacity is important to reduce the number of disk accesses. Normally programs are transferred to the main memory from the disk and executed there. Those which do not fit into the memory - and most of the multimedia applications fit into this category - are transferred as pages. The bigger the memory the more pages shall be transferred hence the number of disk accesses will be reduced and the programs will be executed faster. CD_ROM
- 9. CD_ROM is the abbreviation of the words Compact Disk Read Only Memory. The fastest and cheapest way to distribute multimedia applications is using CD-ROMs. We shall be summarizing the different type of CD ROM formats that support multimedia applications below. CD-DA : Supports digital audio CD-I : This a format developed for applications requiring interactivity. Supports graphics, audio and video. It is suitable for interactive games. CD-XA: XA stands for extended architecture. This format improves the CD ROM's audio and video capability. MSCDEX: A capability to allow the use of CD-ROMs in the DOS environment. PhotoCD: It is used to save photos in the compressed form. The compression allows for 100 photos to be recorded in one CD. Video CD: It is used to store video compressed in MPEG_I format. Videos up to 74 minutes in duration can be stored in one CD using this format. DVD: Digital Video Disk technology improves the storage capacity to 4.7 to 17 Gbytes. It is mainly used to store video compressed in MPEG II standards in a CD ROM. It is being supported especially by the film industry because it supports interactivity and provides protection against illegal copying of the disks. Although it has been defined as part of the MPC Level 3 specifications it will be used in other areas as well due to the capacity it provides. Sound Card: The audio system specified in MPC Level 3 includes • • • • • • • 8 or 16 bits per sample sampling capability 8, 11, 16, 22, or 44 KHz sampling rate A linear Pulse Code Modulation (PCM) Codec to convert the sampled analog values to digital data. A playback and recording of 16-bit stereo that uses no more than 13 % of CPU time for 44.1 K samples/sec. CD ROM drive with CD-DA outputs and volume control. Speakers One MIDI port.
- 10. The block diagram of a sound card compliant with the MPC Level 3 specifications is shown in the figure. Video Graphics Interface Card A graphics display card is used to display an digital data on an analog monitor. Each pixel is represented with 8 to 32 bits and these are converted to RGB signals through a palette. These values are then converted to analog signals for display on the monitor. MPC Level 3 standards require that a graphics adapter should be capable of displaying a 352x240 video at 30 frames per second without dropping any frames. An additional card is usually required to playback videos compressed using MPEG-I. MPEG corresponds to Motion Picture Experts Group. Using this card MPEG videos can be displayed on the monitor. We need an additional board in order to be able to capture analog video and store it in our computer. These cards accept video from VCRs or video cameras and their performance depends on the speed of the codecs used in them. It has to be remembered that one minute of a video recording will occupy 10 to 20 Mbytes of disk space depending on the compression used. Universal Serial Bus With the introduction of multimedia applications the number of devices connected to the computers have increased considerably and this has created problems in management and maintenance of the systems. To avoid the increased number of connectors being used the USB has been developed. Using the USB • The current peripheral devices
- 11. • • • Telephony equipment Joysticks and other tools for playing games Video equipment such as cameras can be connected to the computers from a single port. The total speed of this connection can be 1.5 Mbits/s or 12 Mbits/s and the configuration is shown in figure. The peripheral devices could be connected through hubs or one of the equipment such as the keyboard could serve as a hub. This way a total of 127 devices can be connected to the bus. The maximum cable length in any one connection cannot be longer than 5 m. As stated before the USB has a limited bandwidth. This prevents many devices to be connected simultaneously and even if they can be connected they cannot be serviced simultaneously. To solve this problem High Performance Serial Bus defined by IEEE 1394 standards has been developed. This bus can support bit rates of 100, 200, and 400 Mbits/sec. The Apple version of HPSB is called firewire. Network Interface card - NIC NICs are used to connect the computers using local area networks. These cards can support Ethernet, Token Ring or Token Bus structures. The bit rates depend on the type of network being used and can be 4, 10, 16 or 100 Mbits/sec. There also exist network cards which support cable TV connections.