CS 411/511: Parallel Programming
Catalog Description: Analysis, mapping, and the application of parallel programming software to high-performance systems; the principles of spatial- and temporal-locality of data memory hierarchies in performance tuning; architectural considerations in the design and implementation of a parallel program; the tradeoff between threaded (shared memory) and message-passing (distributed memory) programming styles and performance. Additional projects/assignments required for graduate credit.
Type: CS 411 is a technical elective for CS majors.
Total Credits: 3
Course Coordinator: Robert Hiromoto
Syllabus: CS 411/511 Syllabus
Prerequisites: Analysis of Algorithms, Operating Systems, Concurrent Systems and Computer Architectures, or instructor permission.
Recommended preparation: Proficiency in programming using a modern language such as C or C++.
Textbook: An Introduction to Parallel Programming, Peter Pacheco, Morgan Kauffmann, ISBN-10: 0123742609
This course teaches the principles of parallel programming to upper division and graduate level computer science students. Topics to be covered include programming for symmetric multi-core processors using Pthreads and OpenMP, and distributed memory workstation clusters using the message-passing MPI communication library. Programming tools such as gdb and gprof, parallel programming semantics, and parallel program performance issues will also be covered. Several programming assignments will be used to reinforce the concepts learned in the class room, and a final parallel programming project will be required.
Major Topics Covered
- Spatial and temporal locality
- Parallel programming concepts
- Task and Data decomposition
- Critical-section (atomic updates) – Process starvation (live lock)
- Performance evaluation
- Primitives to control shared resources
- Mutual exclusion (mutexes)
- Thread-safe system libraries
- Shared memory programming using threads (PThreads and OpenMP)
- Shared memory architectural model
- Asynchronous computations
- Parallel programming advantages/disadvantages
- Message passing using MPI
- Block algorithms
- Single program, multiple data (SPMD)
- Performance issues
- Bottlenecks and speedups
- Impact of communication
- Tuning programs
- Understand the concepts of spatial and temporal locality, and their implications for program efficiency
- Be able to decompose an algorithm into parallelizable partitions based on task or data
- Understand factors that limit performance of parallel programs
- Understand mechanisms that can cause program failure: race conditions, deadlock
- Be able to implement primitives to synchronize access to shared resources: Busy-wait, mutex, semaphores
- Understand shared-memory architectures and their use
- Be able to write simple message passing programs using MPI
- Describe how to tune a program to mitigate performance issues such as communication bottlenecks