Operating Systems Protection and Security

Operating Systems
Protection and Security
Prepared By
Dr.M.Sivakumar
AP,NWC, SRMIST

Protection and Security
• Goals of Protection - Principles of Protection - Domain of Protection
• Access Matrix - Implementation of the Access Matrix - Access Control
• Revocation of Access Rights
• Capability-Based Systems
• Language-Based Protection
• The Security Problem
• Program Threats
• System and Network Threats
• Cryptography as a Security Tool
• User Authentication - Implementing Security Defenses
• Firewalling to Protect Systems and Networks, Computer-Security Classifications.

Goals of Protection
Prevent Unauthorized Access
Enforce Policy Compliance
Enhance System Reliability
Distinguish Authorized from
Unauthorized Use
Flexible Enforcement of Diverse
Policies
Support Dynamic Policy
Changes
Resource Guarding by
Application Subsystems

Principles of Protection
• Principle of Least Privilege
– The principle of least privilege dictates that users, programs, and systems should be granted the
minimum level of access or privileges necessary to perform their tasks.
– By limiting access, the potential damage caused by the loss, misuse, or compromise of credentials
or privileges is reduced. This principle helps contain the impact of security breaches or failures.
– A security guard with a passkey that only opens public areas poses less risk than one who has
access to every area.
• Implementation in Operating Systems
– Fail-Safe Design: ensures that even if a component fails (e.g., a buffer overflow), the damage is
limited, and attackers cannot gain maximum privileges or full system control.
– Fine-Grained Access Controls: offers fine-grained access controls, allowing for precise privilege
allocation and control over which services and resources users or programs can access.
– Privilege Enabling and Disabling: Privileges should only be enabled when absolutely necessary
and disabled immediately when not required

Principles of Protection
• Audit Trails
– Monitoring and Tracing: The system should maintain audit trails for all privileged operations. These
logs help system administrators, programmers, or law enforcement trace any security breaches or
violations.
• User Privilege Management
– Role-Based Access Control (RBAC): Users are granted specific roles with the minimum privileges
necessary to perform their tasks. For instance, an operator responsible for backups only has access to
the necessary commands and files, not the entire system.
– Separate User Accounts: Every user is provided with an account that has just the privileges they need
to perform their role, reducing the chances of misuse.
• Application in Computing Facilities
– Service Restrictions: Systems following the principle of least privilege limit services to only the ones
necessary. They restrict which remote hosts can be accessed, what services can be used, and even
when these actions are allowed.
– Access Control Lists (ACLs): ACLs are often used to implement these fine-grained restrictions, enabling
administrators to enforce least privilege effectively.

Domain of Protection
• Objects in a Computer System
– Hardware Objects: These include physical resources such as the CPU, memory segments, printers,
disks, and tape drives.
– Software Objects: These include files, programs, semaphores, and other software resources.
– Unique Identification: Every object has a unique name and can only be accessed through
predefined, meaningful operations specific to that object.
• Operations on Objects
– Different objects allow different operations based on their type:
• CPU: Can only be executed.
• Memory Segments: Can be read and written.
• CD-ROM/DVD-ROM: Can only be read.
• Tape Drives: Can be read, written, and rewound.
• Files: Data files can be created, opened, read, written, closed, and deleted; program files can be
read, written, executed, and deleted

Domain of Protection
• Access Control and Authorization
– A process should be allowed access only to those resources it is authorized to use.
– At any time, a process should only have access to the resources necessary to complete its current
task. This limits the risk of damage in case of faulty processes.
• Need-to-Know Principle
– A process should only have access to the resources it currently requires for its task. This principle
limits the potential damage a malfunctioning process can cause and restricts access to unnecessary
resources.
– Example 1: When a process invokes a procedure, the procedure should access only its own variables
and the formal parameters, not all of the process's variables.
– Example 2: If a process invokes a compiler to compile a file, the compiler should only access the
relevant files (source file, listing file, etc.) and should not arbitrarily access other files.
– The compiler itself may have private files for its internal use that should not be accessible to the
invoking process.

Domain Structure
• Protection Domains
– A protection domain specifies the resources a process can access and the types of
operations it can perform on those resources.
– A domain consists of a set of access rights, where each right is an ordered pair
<object-name, rights-set>
– The rights set defines which operations (e.g., read, write, execute) are allowed on the
object.
– For example, if domain D has the access right <file F, {read, write}>, then a process in
domain D can read and write file F but cannot perform other operations on it.
• Shared Access Rights
– Domains can share access rights. For instance, two domains might share the right to print an
object, meaning processes in either domain can print that object. However, different
domains may have different access rights for the same object.

Domain Structure
• Three domains: D1, D2, and D3.
• The access right <O4, {print}> is shared by D2 and D3, implying that a process executing
in either of these two domains can print object O4.
• A process must be executing in domain D1 to read and write object O1
• Only processes in domain D3 may execute object O1.

Domain Structure
• Association between a process and a domain may be either static or dynamic
• Static vs. Dynamic Domains
– Static Domains: The resources a process can access are fixed for the lifetime of the process. If more access is required
at different stages, the domain may be defined to include more access than necessary, violating the need-to-know
principle.
– Dynamic Domains: The resources accessible by a process can change dynamically during its execution. This allows
more flexibility and adheres to the need-to-know principle by providing only the required access at any given time.
• Implementing Domains
– A domain can be realized in different ways:
– User as a Domain: Each user represents a domain. Domain switching occurs when one user logs out and another logs
in.
– Process as a Domain: Each process is a domain, with switching occurring when a process communicates with another
and waits for a response.
– Procedure as a Domain: Each procedure can be a domain, where switching happens when a procedure is called.
Access is restricted to the procedure’s local variables.

Domain Structure
• Domain Switching
– Mechanism for Dynamic Domains: Domain switching allows a process to switch between domains,
enabling changes to the resources it can access based on the current phase of execution.
– This mechanism supports more precise control over the process’s access rights, minimizing unnecessary
access.
• Dual-Mode Operation
– In the monitor–user mode model, processes can operate in either monitor mode or user mode:
• Monitor Mode: The process can execute privileged instructions and has full control over the system.
• User Mode: The process can only execute non-privileged instructions and access resources within its
allocated space.
• Multiprogrammed Systems
– In systems where multiple users run processes simultaneously, two protection domains (monitor and user)
are not sufficient, as users need to be protected from one another. More sophisticated schemes are
necessary to provide the required level of security.

Access Matrix
• The access matrix is an abstract model for specifying access rights in a system.
• Rows represent domains (the execution environment of processes), and columns represent
objects (resources like files, printers, etc.).
• Each entry in the matrix defines the access rights that a process in a particular domain has over
a specific object.
• The entry access(i, j) specifies the operations a
process executing in domain Di can perform on
object Oj.
• A process executing in domain D1 can read files F1
and F3.
• A process executing in domain D4 can read and
write files F1 and F3
• The laser printer can be accessed only by a process
executing in domain D .

Access Matrix
• Domain Switching
– Processes can switch from one domain to another
based on access rights. A switch right in the access
matrix (switch access(i, j)) defines whether a
∈
process in domain Di can switch to domain Dj.
– A process executing in domain D2 can switch to
domain D3 or to domain D4.
– A process in domain D4 can switch to D1, and one in
domain D1 can switch to D2.
• Modifying Access Matrix Entries
Three special operations allow dynamic changes to the access matrix:
• Copy: Allows a process to copy access rights from one domain to another (e.g., allowing domain D2 to copy read
access to another domain).
• Owner: A process with the owner right can add or remove any access right in the corresponding column (e.g., D1
owning file F1 can modify rights for F1).
• Control: A process with the control right can modify access rights in a row (e.g., D2 controlling D4 can change
access rights for processes in domain D4).

Access Matrix
• The ability to copy an access right from one domain
(or row) of the access matrix to another is denoted
by an asterisk (*) appended to the access right.
• The copy right allows the access right to be copied
only within the column (that is, for the object) for
which the right is defined.
• a process executing in domain D2 can copy the read
operation into any entry associated with file F2.
• The access matrix of Figure (a) can be modified to
the access matrix shown in Figure (b).

Implementation of the Access Matrix
• Global Table
• Access Lists for Objects
• Capability Lists for Domains
• A Lock–Key Mechanism
• Comparison

• Generally, a sparse matrix
• Option 1 – Global table
– Store ordered triples <domain, object, rights-set> in table
– A requested operation M on object Oj within domain Di -> search table for < Di, Oj, Rk >
• with M R
∈ k
– But table could be large -> won’t fit in main memory
– Difficult to group objects (consider an object that all domains can read)
• Option 2 – Access lists for objects
– Each column implemented as an access list for one object
– Resulting per-object list consists of ordered pairs <domain, rights-set> defining all
domains with non-empty set of access rights for the object
– Easily extended to contain default set -> If M default set, also allow access
∈

• Each column = Access-control list for one object
Defines who can perform what operation
Domain 1 = Read, Write
Domain 2 = Read
Domain 3 = Read
• Each Row = Capability List (like a key)
For each domain, what operations allowed on what objects
Object F1 – Read
Object F4 – Read, Write, Execute
Object F5 – Read, Write, Delete, Copy

• Option 3 – Capability list for domains
– Instead of object-based, list is domain based
– Capability list for domain is list of objects together with operations allows on them
– Object represented by its name or address, called a capability
– Execute operation M on object Oj, process requests operation and specifies capability as
parameter
• Possession of capability means access is allowed
– Capability list associated with domain but never directly accessible by domain
• Rather, protected object, maintained by OS and accessed indirectly
• Like a “secure pointer”
• Idea can be extended up to applications

• Option 4 – Lock-key
– Compromise between access lists and capability lists
– Each object has list of unique bit patterns, called locks
– Each domain as list of unique bit patterns called keys
– Process in a domain can only access object if domain has key that matches one
of the locks

Comparison of Implementations
• Many trade-offs to consider
– Global table is simple, but can be large
– Access lists correspond to needs of users
• Determining set of access rights for domain non-localized so difficult
• Every access to an object must be checked
– Many objects and access rights -> slow
– Capability lists useful for localizing information for a given process
• But revocation capabilities can be inefficient
– Lock-key effective and flexible, keys can be passed freely from domain to domain, easy
revocation

Comparison of Implementations
• Most systems use combination of access lists and capabilities
– First access to an object -> access list searched
• If allowed, capability created and attached to process
– Additional accesses need not be checked
• After last access, capability destroyed
• Consider file system with ACLs per file

Revocation (cancel or withdraw) of Access Rights
• Various options to remove the access right of a domain to an object
– Immediate vs. delayed
– Selective vs. general
– Partial vs. total
– Temporary vs. permanent
• Access List – Delete access rights from access list
– Simple – search access list and remove entry
– Immediate, general or selective, total or partial, permanent or temporary

Immediate vs. Delayed Revocation
• Immediate Revocation: The access rights are removed as soon as the revocation
command is issued. This approach ensures that the domain loses access to the
object immediately, preventing any further interaction with the resource.
– Use Case: Often used in security-sensitive scenarios, such as when a user is
terminated from a job or detected engaging in suspicious activities.
• Delayed Revocation: Access rights remain active until a certain condition is met,
like the user logging out or completing an ongoing operation. This approach
allows for a smoother transition, avoiding sudden interruptions.
– Use Case: Appropriate when immediate revocation may disrupt ongoing tasks,
such as a background process that needs to complete.

Selective vs. General Revocation
• Selective Revocation: Only specific rights (e.g., write or execute permissions)
are removed, while others (e.g., read) may remain. This allows for fine-
grained control over access.
– Use Case: Useful when a user needs reduced access but still requires some interaction
with the object, such as read-only access after revoking write permissions.
• General Revocation: All rights to the object are revoked for the domain,
effectively cutting off all forms of access.
– Use Case: Applied when the domain should no longer interact with the object in any
capacity, such as after a user completes a project and no longer needs any access.

Partial vs. Total Revocation
• Partial Revocation: Only some domains lose their access rights to the
object, while others retain their rights. This approach is common in multi-
user systems where different users or roles have varied access levels.
– Use Case: Useful in collaborative environments where only specific team members
need their rights adjusted, like when an individual on a team is reassigned.
• Total Revocation: All domains are stripped of access rights to the object.
This is a blanket approach that eliminates all permissions for everyone.
– Use Case: Suitable when an object is no longer needed, or is being removed from
the system entirely, ensuring no one can access it.

Temporary vs. Permanent Revocation
• Temporary Revocation: Access rights are removed for a limited time and will
be restored later, either automatically or through a new command. This option
is useful for temporary suspensions or restrictions.
– Use Case: Useful when a user is temporarily away or on leave but will need access
again upon return, or for maintenance periods.
• Permanent Revocation: Access rights are removed indefinitely, with no
intention of restoration. The domain will need to go through a full
authorization process to regain access, if ever.
– Use Case: Appropriate when a domain no longer needs access to an object, such as a
user leaving the organization.

Revocation of Access Rights
• Capability List – Scheme required to locate capability in the system before capability can be
revoked
– Reacquisition – periodic delete, with require and denial if revoked
– Back-pointers – set of pointers from each object to all capabilities of that object (Multics)
– Indirection – capability points to global table entry which points to object – delete entry
from global table, not selective (CAL)
– Keys – unique bits associated with capability, generated when capability created
• Master key associated with object, key matches master key for access
• Revocation – create new master key
• Policy decision of who can create and modify keys – object owner or others?

Access Control
• Manages which users or processes can access certain resources (such
as files, memory, or hardware) and what actions they can perform on
those resources.
• It ensures that only authorized entities are allowed to interact with
sensitive or restricted parts of the system.
• Access Control Models
– Discretionary Access Control (DAC)
– Mandatory Access Control (MAC)
– Role-Based Access Control (RBAC)
– Attribute-Based Access Control (ABAC)

Discretionary Access Control (DAC)
• The file owner and resources set the access right permission.
• The data owner can grant permission to other users to access or perform any action on
resources.
• For example, any operating system owner can set access permissions for other users, which
allows flexibility and user preference.

Role-based Access Control
• Protection can be applied to non-file resources
• Oracle Solaris 10 provides role-based access control (RBAC) to implement least privilege
– Privilege is right to execute system call or use an option within a system call
– Can be assigned to processes
– Users assigned roles granting access to privileges and programs
• Enable role via password to gain its privileges
– Similar to access matrix

Mandatory Access Control (MAC)
• Operating systems traditionally had discretionary access control (DAC) to limit access to files and other
objects (for example UNIX file permissions and Windows access control lists (ACLs))
– Discretionary is a weakness – users / admins need to do something to increase protection
• Stronger form is mandatory access control, which even root user can’t circumvent
– Makes resources inaccessible except to their intended owners
– Modern systems implement both MAC and DAC, with MAC usually a more secure, optional
configuration (Trusted Solaris, TrustedBSD (used in macOS), SELinux), Windows Vista MAC)
• At its heart, labels assigned to objects and subjects (including processes)
– When a subject requests access to an object, policy checked to determine whether or not a given label-
holding subject is allowed to perform the action on the object

Attribute-based Access control matrix
• Access permission is granted based on different attributes and policies of subject,
object, and environment.
• Here, attributes can be a role, resources, time, system type, and location.
• Modern environments commonly use this matrix where access is given or denied
based on different attributes like the time and location of the user.

Capability-Based Systems
• Hydra and CAP were first capability-based systems
• Now included in Linux, Android and others, based on POSIX.1e (that never became a standard)
– Essentially slices up root powers into distinct areas, each represented by a bitmap bit
– Fine grain control over privileged operations can be achieved by setting or masking the
bitmap
– Three sets of bitmaps – permitted, effective, and inheritable
• Can apply per process or per thread
• Once revoked, cannot be reacquired
• Process or thread starts with all privs, voluntarily decreases set during execution
• Essentially a direct implementation of the principle of least privilege
• An improvement over root having all privileges but inflexible (adding new privilege difficult,
etc.)

Other Protection Improvement Methods
• System integrity protection (SIP)
– Introduced by Apple in macOS 10.11
– Restricts access to system files and resources, even by root
– Uses extended file attribs to mark a binary to restrict changes, disable debugging and
scrutinizing
– Also, only code-signed kernel extensions allowed and configurably only code-signed apps
• System-call filtering
– Like a firewall, for system calls
– Can also be deeper –inspecting all system call arguments
– Linux implements via SECCOMP-BPF (Berkeley packet filtering)

• Sandboxing
– Running process in limited environment
– Impose set of irremovable restrictions early in startup of process (before main())
– Process then unable to access any resources beyond its allowed set
– Java and .net implement at a virtual machine level
– Other systems use MAC to implement
– Apple was an early adopter, from macOS 10.5’s “seatbelt” feature
• Dynamic profiles written in the Scheme language, managing system calls even at the
argument level
• Apple now does SIP, a system-wide platform profile

• Code signing allows a system to trust a program or script by using crypto hash to have the
developer sign the executable
– So code as it was compiled by the author
– If the code is changed, signature invalid and (some) systems disable execution
– Can also be used to disable old programs by the operating system vendor (such as Apple)
cosigning apps, and then invaliding those signatures so the code will no longer run

Language-Based Protection
• Specification of protection in a programming language allows the high-
level description of policies for the allocation and use of resources
• Language implementation can provide software for protection
enforcement when automatic hardware-supported checking is
unavailable
• Interpret protection specifications to generate calls on whatever
protection system is provided by the hardware and the operating
system

Protection in Java 2
• Protection is handled by the Java Virtual Machine (JVM)
• A class is assigned a protection domain when it is loaded by the JVM
• The protection domain indicates what operations the class can (and cannot)
perform
• If a library method is invoked that performs a privileged operation, the stack is
inspected to ensure the operation can be performed by the library
• Generally, Java’s load-time and run-time checks enforce type safety
• Classes effectively encapsulate and protect data and methods from other classes

The Security Problem
• System secure if resources used and accessed as intended under all
circumstances
– Unachievable
• Intruders (crackers) attempt to breach security
• Threat is potential security violation
• Attack is attempt to breach security
• Attack can be accidental or malicious
• Easier to protect against accidental than malicious misuse

Security Violation Categories
• Breach of confidentiality
– Unauthorized reading of data
• Breach of integrity
– Unauthorized modification of data
• Breach of availability
– Unauthorized destruction of data
• Theft of service
– Unauthorized use of resources
• Denial of service (DOS)
– Prevention of legitimate use

Security Violation Methods
• Masquerading (breach authentication)
– Pretending to be an authorized user to escalate privileges
• Replay attack
– As is or with message modification
• Man-in-the-middle attack
– Intruder sits in data flow, masquerading as sender to receiver and vice versa
• Session hijacking
– Intercept an already-established session to bypass authentication
• Privilege escalation
– Common attack type with access beyond what a user or resource is supposed to have

Security Measure Levels
• Impossible to have absolute security, but make cost to perpetrator sufficiently high to deter most intruders
• Security must occur at four levels to be effective:
– Physical
• Data centers, servers, connected terminals
– Application
• Benign or malicious apps can cause security problems
– Operating System
• Protection mechanisms, debugging
– Network
• Intercepted communications, interruption, DOS
• Security is as weak as the weakest link in the chain
• Humans a risk too via phishing and social-engineering attacks
• But can too much security be a problem?

Program Threats
• Trojan Horse
• Trap Door
• Logic Bomb
• Stack and Buffer Overflow
• Viruses

Program Threats
• Many variations, many names
• Trojan Horse
– Code segment that misuses its environment
– Exploits mechanisms for allowing programs written by users to be executed by other users
– Spyware, pop-up browser windows, covert channels
– Up to 80% of spam delivered by spyware-infected systems
• Trap Door
– Specific user identifier or password that circumvents normal security procedures
– Could be included in a compiler
– How to detect them?

Program Threats
• Malware - Software designed to exploit, disable, or damage computer
• Trojan Horse – Program that acts in a clandestine manner
– Spyware – Program frequently installed with legitimate software to display adds, capture
user data
– Ransomware – locks up data via encryption, demanding payment to unlock it
• Others include trap doors, logic boms
• All try to violate the Principle of Least Privilege
• Goal frequently is to leave behind Remote Access Tool (RAT) for repeated access

C Program with Buffer-overflow Condition
#include <stdio.h>
#define BUFFER SIZE 256
int main(int argc, char *argv[])
{
char buffer[BUFFER SIZE];
if (argc < 2)
return -1;
else {
strcpy(buffer,argv[1]);
return 0;
}
}
Code review can help – programmers review each other’s code, looking for logic flows, programming flaws

Code Injection
 Code-injection attack occurs when system code is not malicious but has bugs allowing
executable code to be added or modified
• Results from poor or insecure programming paradigms, commonly in low level languages
like C or C++ which allow for direct memory access through pointers
• Goal is a buffer overflow in which code is placed in a buffer and execution caused by the
attack
• Can be run by script kiddies – use tools written but exploit identifiers
• Outcomes from code injection include:
• Frequently use trampoline
to code execution to
exploit buffer overflow:

Great Programming Required?
• For the first step of determining the bug, and second step of writing exploit code, yes
• Script kiddies can run pre-written exploit code to attack a given system
• Attack code can get a shell with the processes’ owner’s permissions
– Or open a network port, delete files, download a program, etc.
• Depending on bug, attack can be executed across a network using allowed
connections, bypassing firewalls
• Buffer overflow can be disabled by disabling stack execution or adding bit to page
table to indicate “non-executable” state
– Available in SPARC and x86
– But still have security exploits

Program Threats (Cont.)
• Viruses
– Code fragment embedded in legitimate program
– Self-replicating, designed to infect other computers
– Very specific to CPU architecture, operating system, applications
– Usually borne via email or as a macro
– Visual Basic Macro to reformat hard drive
Sub AutoOpen()
Dim oFS
Set oFS = CreateObject(’’Scripting.FileSystemObject’’)
vs = Shell(’’c:command.com /k format c:’’,vbHide)
End Sub

Program Threats (Cont.)
• Virus dropper inserts virus onto the system
• Many categories of viruses, literally many thousands of viruses
– File / parasitic
– Boot / memory
– Macro
– Source code
– Polymorphic to avoid having a virus signature
– Encrypted
– Stealth
– Tunneling
– Multipartite
– Armored

The Threat Continues
• Attacks still common, still occurring
• Attacks moved over time from science experiments to tools of organized crime
– Targeting specific companies
– Creating botnets to use as tool for spam and DDOS delivery
– Keystroke logger to grab passwords, credit card numbers
• Why is Windows the target for most attacks?
– Most common
– Everyone is an administrator
• Licensing required?
– Monoculture considered harmful

Four-layered Model of Security

System and Network Threats
• Worms
• Port Scanning
• Denial of Service

• Some systems “open” rather than secure by default
– Reduce attack surface
– But harder to use, more knowledge needed to administer
• Network threats harder to detect, prevent
– Protection systems weaker
– More difficult to have a shared secret on which to base access
– No physical limits once system attached to internet
• Or on network with system attached to internet
– Even determining location of connecting system difficult
• IP address is only knowledge

• Worms – use spawn mechanism; standalone program
• Internet worm
– Exploited UNIX networking features (remote access) and bugs in finger and
sendmail programs
– Exploited trust-relationship mechanism used by rsh to access friendly systems
without use of password
– Grappling hook program uploaded main worm program
• 99 lines of C code
– Hooked system then uploaded main code, tried to attack connected systems
– Also tried to break into other users accounts on local system via password guessing
– If target system already infected, abort, except for every 7th
time

• Port scanning
– Automated attempt to connect to a range of ports on one or a range of IP
addresses
– Detection of answering service protocol
– Detection of OS and version running on system
– nmap scans all ports in a given IP range for a response
– nessus has a database of protocols and bugs (and exploits) to apply against a
system
– Frequently launched from zombie systems
• To decrease trace-ability

• Denial of Service
– Overload the targeted computer preventing it from doing any useful work
– Distributed Denial-of-Service (DDoS) come from multiple sites at once
– Consider the start of the IP-connection handshake (SYN)
• How many started-connections can the OS handle?
– Consider traffic to a web site
• How can you tell the difference between being a target and being really popular?
– Accidental – CS students writing bad fork() code
– Purposeful – extortion, punishment
• Port scanning
– Automated tool to look for network ports accepting connections
– Used for good and evil

Cryptography as a Security Tool
• Encryption
• Implementation of Cryptography
• An Example: SSL

Cryptography as a Security Tool
• Broadest security tool available
– Internal to a given computer, source and destination of messages can be
known and protected
• OS creates, manages, protects process IDs, communication ports
– Source and destination of messages on network cannot be trusted without
cryptography
• Local network – IP address?
–Consider unauthorized host added
• WAN / Internet – how to establish authenticity
–Not via IP address

Cryptography
• Means to constrain potential senders (sources) and / or receivers (destinations) of
messages
– Based on secrets (keys)
– Enables
• Confirmation of source
• Receipt only by certain destination
• Trust relationship between sender and receiver

Encryption
• Constrains the set of possible receivers of a message
• Encryption algorithm consists of
– Set K of keys
– Set M of Messages
– Set C of ciphertexts (encrypted messages)
– A function E : K → (M→C). That is, for each k  K, Ek is a function for generating ciphertexts
from messages
• Both E and Ek for any k should be efficiently computable functions
– A function D : K → (C → M). That is, for each k  K, Dk is a function for generating messages
from ciphertexts
• Both D and Dk for any k should be efficiently computable functions

Encryption
• An encryption algorithm must provide this essential property: Given a
ciphertext c  C, a computer can compute m such that Ek(m) = c only if
it possesses k
– Thus, a computer holding k can decrypt ciphertexts to the plaintexts used to
produce them, but a computer not holding k cannot decrypt ciphertexts
– Since ciphertexts are generally exposed (for example, sent on the network), it
is important that it be infeasible to derive k from the ciphertexts

Symmetric Encryption
• Same key used to encrypt and decrypt
– Therefore k must be kept secret
• DES was most commonly used symmetric block-encryption algorithm (created by US Govt)
– Encrypts a block of data at a time
– Keys too short so now considered insecure
• Triple-DES considered more secure
– Algorithm used 3 times using 2 or 3 keys
– For example
• 2001 NIST adopted new block cipher - Advanced Encryption Standard (AES)
– Keys of 128, 192, or 256 bits, works on 128 bit blocks
• RC4 is most common symmetric stream cipher, but known to have vulnerabilities
– Encrypts/decrypts a stream of bytes (i.e., wireless transmission)
– Key is a input to pseudo-random-bit generator
• Generates an infinite keystream

Secure Communication over Insecure Medium

Asymmetric Encryption
• Public-key encryption based on each user having two keys:
– public key – published key used to encrypt data
– private key – key known only to individual user used to decrypt data
• Must be an encryption scheme that can be made public without
making it easy to figure out the decryption scheme
– Most common is RSA block cipher
– Efficient algorithm for testing whether or not a number is prime
– No efficient algorithm is know for finding the prime factors of a number

Asymmetric Encryption
• Formally, it is computationally infeasible to derive kd,N from ke,N, and so
ke need not be kept secret and can be widely disseminated
– ke is the public key
– kd is the private key
– N is the product of two large, randomly chosen prime numbers p and q (for
example, p and q are 512 bits each)
– Encryption algorithm is Eke,N(m) = mke mod N, where ke satisfies kekd mod (p−1)(q
−1) = 1
– The decryption algorithm is then Dkd,N(c) = ckd mod N

Asymmetric Encryption Example
• For example. make p = 7and q = 13
• We then calculate N = 7 13 = 91 and (
∗ p−1)(q−1) = 72
• We next select ke relatively prime to 72 and< 72, yielding 5
• Finally, we calculate kd such that kekd mod 72 = 1, yielding 29
• We how have our keys
– Public key, ke,N = 5, 91
– Private key, kd,N = 29, 91
• Encrypting the message 69 with the public key results in the cyphertext 62
• Cyphertext can be decoded with the private key
– Public key can be distributed in cleartext to anyone who wants to communicate with holder of public key

Encryption using
RSA Asymmetric Cryptography

Cryptography (Cont.)
• Note symmetric cryptography based on transformations, asymmetric
based on mathematical functions
– Asymmetric much more compute intensive
– Typically not used for bulk data encryption

Authentication
• Constraining set of potential senders of a message
– Complementary to encryption
– Also can prove message unmodified
• Algorithm components
– A set K of keys
– A set M of messages
– A set A of authenticators
– A function S : K → (M→ A)
• That is, for each k  K, Sk is a function for generating authenticators from messages
• Both S and Sk for any k should be efficiently computable functions
– A function V : K → (M × A→ {true, false}). That is, for each k  K, Vk is a function for
verifying authenticators on messages
• Both V and Vk for any k should be efficiently computable functions

Authentication (Cont.)
• For a message m, a computer can generate an authenticator a  A such that Vk(m, a) = true
only if it possesses k
• Thus, computer holding k can generate authenticators on messages so that any other computer
possessing k can verify them
• Computer not holding k cannot generate authenticators on messages that can be verified using
Vk
• Since authenticators are generally exposed (for example, they are sent on the network with the
messages themselves), it must not be feasible to derive k from the authenticators
• Practically, if Vk(m,a) = true then we know m has not been modified and that send of
message has k
– If we share k with only one entity, know where the message originated

Authentication – Hash Functions
• Basis of authentication
• Creates small, fixed-size block of data message digest (hash value) from m
• Hash Function H must be collision resistant on m
– Must be infeasible to find an m’ ≠ m such that H(m) = H(m’)
• If H(m) = H(m’), then m = m’
– The message has not been modified
• Common message-digest functions include MD5, which produces a 128-bit hash, and SHA-1, which
outputs a 160-bit hash
• Not useful as authenticators
– For example H(m) can be sent with a message
• But if H is known someone could modify m to m’ and recompute H(m’) and modification not
detected
• So must authenticate H(m)

Authentication - MAC
• Symmetric encryption used in message-authentication code (MAC)
authentication algorithm
• Cryptographic checksum generated from message using secret key
– Can securely authenticate short values
• If used to authenticate H(m) for an H that is collision resistant, then
obtain a way to securely authenticate long message by hashing them first
• Note that k is needed to compute both Sk and Vk, so anyone able to
compute one can compute the other

Authentication – Digital Signature
• Based on asymmetric keys and digital signature algorithm
• Authenticators produced are digital signatures
• Very useful – anyone can verify authenticity of a message
• In a digital-signature algorithm, computationally infeasible to derive ks from kv
– V is a one-way function
– Thus, kv is the public key and ks is the private key
• Consider the RSA digital-signature algorithm
– Similar to the RSA encryption algorithm, but the key use is reversed
– Digital signature of message Sks (m) = H(m)ks mod N
– The key ks again is a pair (d, N), where N is the product of two large, randomly chosen prime
numbers p and q
– Verification algorithm is Vkv(m, a) (akv mod N = H(m))
• Where kv satisfies kvks mod (p − 1)(q − 1) = 1

• Why authentication if a subset of encryption?
– Fewer computations (except for RSA digital signatures)
– Authenticator usually shorter than message
– Sometimes want authentication but not confidentiality
• Signed patches et al
– Can be basis for non-repudiation

Key Distribution
• Delivery of symmetric key is huge challenge
– Sometimes done out-of-band
• Asymmetric keys can proliferate – stored on key ring
– Even asymmetric key distribution needs care – man-in-the-middle attack

Digital Certificates
• Proof of who or what owns a public key
• Public key digitally signed a trusted party
• Trusted party receives proof of identification from entity and certifies
that public key belongs to entity
• Certificate authority are trusted party – their public keys included with
web browser distributions
– They vouch for other authorities via digitally signing their keys, and so on

Man-in-the-middle Attack on
Asymmetric Cryptography

Implementation of Cryptography
• Can be done at various layers of ISO Reference Model
– SSL at the Transport layer
– Network layer is typically IPSec
• IKE for key exchange
• Basis of Virtual Private Networks (VPNs)
• Why not just at lowest level?
– Sometimes need more knowledge than available at
low levels
• i.e., User authentication
• i.e., e-mail delivery

Encryption Example - TLS
• Insertion of cryptography at one layer of the ISO network model (the transport layer)
• SSL – Secure Socket Layer (also called TLS)
• Cryptographic protocol that limits two computers to only exchange messages with each other
– Very complicated, with many variations
• Used between web servers and browsers for secure communication (credit card numbers)
• The server is verified with a certificate assuring client is talking to correct server
• Asymmetric cryptography used to establish a secure session key (symmetric encryption) for
bulk of communication during session
• Communication between each computer then uses symmetric key cryptography
• More details in textbook

User Authentication
• Crucial to identify user correctly, as protection systems depend on user ID
• User identity most often established through passwords, can be considered a special case of either
keys or capabilities
• Passwords must be kept secret
– Frequent change of passwords
– History to avoid repeats
– Use of “non-guessable” passwords
– Log all invalid access attempts (but not the passwords themselves)
– Unauthorized transfer
• Passwords may also either be encrypted or allowed to be used only once
– Does encrypting passwords solve the exposure problem?
• Might solve sniffing
• Consider shoulder surfing
• Consider Trojan horse keystroke logger
• How are passwords stored at authenticating site?

User Authentication
• Passwords
• Password Vulnerabilities
• Securing Passwords
• One-Time Passwords
• Biometrics

Passwords
• Encrypt to avoid having to keep secret
– But keep secret anyway (i.e. Unix uses superuser-only readably file /etc/shadow)
– Use algorithm easy to compute but difficult to invert
– Only encrypted password stored, never decrypted
– Add “salt” to avoid the same password being encrypted to the same value
• One-time passwords
– Use a function based on a seed to compute a password, both user and computer
– Hardware device / calculator / key fob to generate the password
• Changes very frequently
• Biometrics
– Some physical attribute (fingerprint, hand scan)
• Multi-factor authentication
– Need two or more factors for authentication
• i.e., USB “dongle”, biometric measure, and password

Implementing Security Defenses
• Security Policy
• Vulnerability Assessment
• Intrusion Detection
• Virus Protection
• Auditing, Accounting, and Logging

Implementing Security Defenses
• Defense in depth is most common security theory – multiple layers of security
• Security policy describes what is being secured
• Vulnerability assessment compares real state of system / network compared to security policy
• Intrusion detection endeavors to detect attempted or successful intrusions
– Signature-based detection spots known bad patterns
– Anomaly detection spots differences from normal behavior
• Can detect zero-day attacks
– False-positives and false-negatives a problem
• Virus protection
– Searching all programs or programs at execution for known virus patterns
– Or run in sandbox so can’t damage system
• Auditing, accounting, and logging of all or specific system or network activities
• Practice safe computing – avoid sources of infection, download from only “good” sites, etc

Firewalling to Protect Systems and Networks
• A network firewall is placed between trusted and untrusted hosts
– The firewall limits network access between these two security domains
• Can be tunneled or spoofed
– Tunneling allows disallowed protocol to travel within allowed protocol (i.e.,
telnet inside of HTTP)
– Firewall rules typically based on host name or IP address which can be spoofed
• Personal firewall is software layer on given host
– Can monitor / limit traffic to and from the host
• Application proxy firewall understands application protocol and can control them
(i.e., SMTP)
• System-call firewall monitors all important system calls and apply rules to them
(i.e., this program can execute that system call)

Firewalling to Protect Systems and Networks

Computer-Security Classifications
• U.S. Department of Defense outlines four divisions of computer security: A, B, C,
and D
• D – Minimal security
• C – Provides discretionary protection through auditing
– Divided into C1 and C2
• C1 identifies cooperating users with the same level of protection
• C2 allows user-level access control
• B – All the properties of C, however each object may have unique sensitivity
labels
– Divided into B1, B2, and B3
• A – Uses formal design and verification techniques to ensure security

Security Defenses Summarized
• By applying appropriate layers of defense, we can keep systems safe from all but the most
persistent attackers. In summary, these layers may include the following:
– Educate users about safe computing—don’t attach devices of unknown origin to the
computer, don’t share passwords, use strong passwords, avoid falling for social engineering
appeals, realize that an e-mail is not necessarily a private communication, and so on
– Educate users about how to prevent phishing attacks—don’t click on email attachments or
links from unknown (or even known) senders; authenticate (for example, via a phone call)
that a request is legitimate
– Use secure communication when possible
– Physically protect computer hardware
– Configure the operating system to minimize the attack surface; disable all unused services
– Configure system daemons, privileges applications, and services to be as secure as possible

Security Defenses Summarized
• Use modern hardware and software, as they are likely to have up-to-date security features
• Keep systems and applications up to date and patched
• Only run applications from trusted sources (such as those that are code signed)
• Enable logging and auditing; review the logs periodically, or automate alerts
• Install and use antivirus software on systems susceptible to viruses, and keep the software up to
date
• Use strong passwords and passphrases, and don’t record them where they could be found
• Use intrusion detection, firewalling, and other network-based protection systems as appropriate
• For important facilities, use periodic vulnerability assessments and other testing methods to test
security and response to incidents
• Encrypt mass-storage devices, and consider encrypting important individual files as well
• Have a security policy for important systems and facilities, and keep it up to date

Example: Windows 10
• Security is based on user accounts
– Each user has unique security ID
– Login to ID creates security access token
• Includes security ID for user, for user’s groups, and special privileges
• Every process gets copy of token
• System checks token to determine if access allowed or denied
• Uses a subject model to ensure access security
– A subject tracks and manages permissions for each program that a user runs
• Each object in Windows has a security attribute defined by a security descriptor
– For example, a file has a security descriptor that indicates the access permissions for all
users

Example: Windows 7 (Cont.)
• Win added mandatory integrity controls – assigns integrity label to each securable object and
subject
– Subject must have access requested in discretionary access-control list to gain access to
object
• Security attributes described by security descriptor
– Owner ID, group security ID, discretionary access-control list, system access-control list
• Objects are either container objects (containing other objects, for example a file system
directory) or noncontainer objects
– By default an object created in a container inherits permissions from the parent object
• Some Win 10 security challenges result from security settings being weak by default, the
number of services included in a Win 10 system, and the number of applications typically
installed on a Win 10 system

Operating Systems Protection and Security

More Related Content

Similar to Operating Systems Protection and Security (20)

More from Sivakumar M (12)

Recently uploaded (20)

Operating Systems Protection and Security