GL325 Linux Performance Tuning and Analysis

Prerequisites:

An intermediate knowledge of the Linux operating system is required. The equivalent knowledge can be found in the GL120 "Linux Fundamentals" and GL250 "Enterprise Linux Systems Administration" courses.

Supported Distributions: Red Hat Enterprise Linux 7

1. Tuning Principles & The USE Method
   1. What is Performance Tuning?
   2. Performance Tuning Methodology
   3. Model System
   4. Establish a Baseline
   5. Establish Performance Targets
   6. Identify Bottleneck
   7. Tune
   8. Measure, Monitor, Evaluate, Repeat?
   9. Final Thoughts
   10. The USE Method
   11. Gathering USE Metrics
   Lab Tasks
   1. Determining the System's Configuration
   2. USE Method: CPU
   3. USE Method: Memory
   4. USE Method: Network
   5. USE Method: Storage I/O & Capacity
2. Linux Internals Review
   1. The Linux Operating System Components
   2. Open Standards
   3. Memory Layout
   4. Important Kernel Threads
   5. Device Files in Linux
   6. Managing Linux Device Files
   7. udev
   8. Random Numbers and /dev/random
   9. Block Devices and the Device Mapper
   10. List Block Devices
   11. SCSI Devices
   12. File Systems with Linux
   13. Hardware Discovery Tools
   14. Cache and Write Barriers
   15. Advanced LVM: Components & Object Tags
   16. Advanced LVM: Automated Storage Tiering
   17. Advanced LVM: Thin Provisioning
   18. Advanced LVM: Striping & Mirroring
   19. Advanced LVM: RAID Volumes
   20. Kernel Modules
   Lab Tasks
   1. Adjusting Kernel Options
   2. Linux Kernel Driver Compilation
3. Linux Userspace Architecture
   1. systemd System and Service Manager
   2. Using systemd
   3. Security Model Overview
   Lab Tasks
   1. Managing Services With Systemd's systemctl
   2. Setting Limits with the pam_limits Modules
   3. Operation of the setuid() and capset() System Calls
4. Log File Administration
   1. System Logging
   2. systemd Journal
   3. systemd Journal's journalctl
   4. Secure Logging with Journal's Log Sealing
   5. gnome-system-log
   6. Rsyslog
   7. /etc/rsyslog.conf
   8. Log Management
   9. Log Anomaly Detector
   10. Sending logs from the shell
   Lab Tasks
   1. Using the systemd Journal
   2. Setting up a Full Debug Logfile
5. Monitoring & Troubleshooting
   1. Viewing Processes
   2. System Status – Memory
   3. System Status – I/O
   4. System Status – CPU
   5. Performance Trending with sar
   6. Determining Service to Process Mapping
   7. Analyzing Process Memory Usage with pmap
   8. Real-time Monitoring of Resources — Cgroups
   9. strace and ltrace
   10. lsof and fuser
   11. ss/netstat and rpcinfo
   12. tcpdump and wireshark
   Lab Tasks
   1. Process Related Tools
   2. System Activity Reporter
   3. Cgroup for Processes
   4. Analyzing Memory Usage
6. Benchmark Tools
   1. Benchmark Tools
   2. Disk/Filesystem Benchmarks
   3. Network Benchmarks
   4. Phoronix Test Suite
   Lab Tasks
   1. Benchmarking Disk and Filesystem
   2. Bonnie++ Visualization
7. Systemtap
   1. SystemTap Overview
   2. SystemTap Features
   3. SystemTap Requirements
   4. Invoking SystemTap
   5. The 5 passes of stap
   6. SystemTap Probe Point Syntax
   7. SystemTap Probe Modes
   8. SystemTap Probe Points
   9. SystemTap process Probes
   10. Listing and Using Probe Points
   11. Built-in SystemTap Functionality
   12. SystemTap Example Repositories
   13. SystemTap Advanced Example
   14. SystemTap Documentation
   Lab Tasks
   1. System Tap Intro
   2. System Tap
   3. Creating SystemTap Scripts
   4. SystemTap One-liners & Histograms
8. Performance Co-Pilot
   1. Performance Co-Pilot Introduction
   2. PCP Architecture
   3. PCP Performance Metrics
   4. PCP Installation
   5. PCP Performance Monitoring
   6. PCP Centralized Logging
   7. PCP Inference Engine
   8. PCP Deployment Strategies
   9. PCP Advanced Topics
   Lab Tasks
   1. Performance Co-Pilot Introduction
   2. Live Monitoring with pmchart
   3. Install Web Monitoring Tools
9. Remote Metric Gathering With SNMP
   1. SNMP Overview
   2. SNMP Operation - Query
   3. MIBs and OID Namespace
   4. SNMP Security
   5. SNMP for Linux
   Lab Tasks
   1. SNMP
10. Linux perf
    1. perf Overview
    2. perf Commands Typical Workflow
    3. perf Events
    4. perf stat
    5. perf record
    6. perf report
    Lab Tasks
    1. Perf
11. Linux Tunables
    1. Operating System Tuning Caveats
    2. Consider OS (or Kernel) Upgrade
    3. Tuning CPU - NUMA
    4. Tuning CPU - Scheduling
    5. Huge Memory Pages
    6. Tuning Virtual Memory Manager
    7. Tuning Virtual Memory Manager (cont.)
    8. Dropping Caches
    9. I/O Scheduler
    10. Tuning Filesystems
    11. Tuning Filesystems (cont.)
    12. Network Tuning
    13. tuned