What is message passing and how does it differ from shared memory parallel programming paradigms?

2 Shared memory

Shared memory is a paradigm where all the processors or machines share a common memory space and can access it directly. The shared memory can be physical, such as a single machine with multiple cores, or virtual, such as a cluster of machines with a distributed memory system. Shared memory allows the processors or machines to communicate implicitly by reading and writing to the shared memory. Shared memory requires mechanisms to ensure consistency and coherence of the memory, which can be done using various models and primitives, such as locks, semaphores, or atomic operations.

3 Advantages and disadvantages

When considering which approach to use, message passing or shared memory, it is important to consider factors such as scalability, simplicity, portability, and flexibility. Message passing can scale better than shared memory and be more portable, but it can also incur more overhead and latency due to network communication. Shared memory can be simpler than message passing, but it can also introduce complexity and errors due to concurrency and consistency issues. Message passing can be more flexible than shared memory and support different data structures, algorithms, and communication patterns; however, it can also be more difficult to optimize and debug.

4 Hybrid approaches

In some cases, it is possible to combine message passing and shared memory to leverage the benefits of both paradigms. For example, one can use message passing to communicate among different machines or clusters, and use shared memory to communicate within each machine or cluster. This can reduce the network communication cost and increase the parallelism. However, this also requires more coordination and integration of the two paradigms, which can add more complexity and challenges.

5 Examples and applications

Message passing and shared memory are widely used for parallel processing in domains such as scientific computing, machine learning, web services, and gaming. Examples of message passing applications include MPI-based applications, such as LAMMPS or GROMACS, which use message passing to perform molecular dynamics simulations on distributed systems; MapReduce-based applications, such as Hadoop or Spark, which use message passing to distribute and process large-scale data on clusters of machines; and actor-based applications, such as Erlang or Akka, which use message passing to create concurrent and distributed systems based on the actor model. Examples of shared memory applications include OpenMP-based applications, such as NAMD or BLAST, which use shared memory to parallelize computations on multicore machines; CUDA-based applications, such as TensorFlow or PyTorch, which use shared memory to accelerate computations on GPUs (Graphics Processing Units); and lock-free or wait-free applications, such as RCU or STM, which use shared memory to implement concurrent data structures without blocking or waiting.

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?