Tuning the Kernel

Time Sync

  • Cassandra nodes identify valid data using timestamps

    • All nodes within a Cassandra cluster need to have synchronized clocks

  • Time Stamp Counter (TSC) is a simple register within the CPU that counts the number of clock cycles

    • Over time, TSC will drift because the clock cycles may vary between CPUs

  • Network Time Protocol (NTP) is a way to synchronize CPU clocks

    • Nodes communicate with a hierarchy of time servers to adjust their clock
    • Adjustments occur every 1-20 minutes

time sync\ntp

Kernel Configuration

User Resource Limits

  • Operating systems limit resources to prevent processes from hogging resources
  • To view current limits
ulimit -a

User Resource Limits

  • Since Cassandra nodes don’t need to share resources, these limits are not helpful
  • Turn them off globally by editing limits.conf
  • Limits take effect when you login
  • For Ubuntu, use root instead of *
* - nofile     1048576
* - memlock    unlimited
* - fsize      unlimited
* - data       unlimited
* - rss        unlimited
* - stack      unlimited
* - cpu        unlimited
* - nproc      unlimited
* - as         unlimited
* - locks      unlimited
* - sigpending unlimited
* - msgqueue   unlimited

swap

  • For Cassandra, swapping is a very bad event
  • You are better having a node go down than limp along swapping
  • To thoroughly disable swap:

    • Turn off swap for the current kernel process
    • Remove swap entries from fstab
    • Change the swappiness setting

Turning off Swap for the Current Kernel Image

  • You can check the current list of swap devices swapon -s
  • You can turn off swap without rebooting
  • This command will not persist (i.e., will not survive a reboot):
swapoff -a

Disabling Swap in fstab

  • This will permanently disable the swap devices after a reboot
  • In /etc/fstab, comment out any rows that are swap devices
  • Here’s a sed script to do this:
sed -i 's/^\(.*swap\)/#\1/' /etc/fstab

Changing the swappiness Setting

  • To look at the current swappiness setting cat /proc/sys/vm/swappiness
  • This value has a range of 0-200, (0 is low-swap and 200 is high-swap)
  • To make sure your kernel disables swapping after a reboot, edit /etc/sysctl.conf
  • Change or add a line to set vm.swappiness
  • Use sysctl -p to get the kernel to reload the changes made to /etc/sysctl.conf
vm.swappiness = 0

Changing Network Kernel Settings

  • Here are some recommended settings to tune network performance
  • Add or change these settings in /etc/sysctl.conf
  • Remember, sysctl -p will cause the kernel to load the settings from /etc/sysctl.conf
net.ipv4.ip_local_port_range = 10000 65535
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 2500
net.core.somaxconn = 65000
Useful explanation of some of the settings: https://www.frozentux.net/ipsysctl-tutorial/ipsysctl-tutorial.html
- ipv4 - network stack tuning
- core - all the networking stack tuning