Cluster Performance Estimation and Verification

Cluster Sizing

This is an Estimation Process

  • Estimates are only a rough-order of magnitude due to metadata
  • Things to consider when estimating cluster size:

    • Throughput - How much data per second?
    • Growth Rate - How fast does capacity increase?
    • Latency - How quickly can the cluster respond?

Cluster Throughput

  • Measure throughput in data movement per time period (e.g. GB/S)
  • Consider reading and writing separately
  • A function of:

    • Operation generators (e.g. users)
    • Rate of operation generation (e.g. 3 clicks per minute)
    • Size of the operations (number of rows X row width)
    • Operation mix (read/write ratio)

Example: Write Cluster Throughput by Posting Comments to KillrVideo

  • 2 million users
  • Each user comments 5 times/day
  • Yields ~100 comments/second = (2M * 5) / (24 * 60 * 60)
  • Each comment inserts 1 row
  • Each row is ~1,000 bytes
  • Result is writing 100KBS = 100 * 1,000

Example: Read Cluster Throughput by Displaying Comments for KillrVideo

  • 2 million users
  • Viewing 10 video summaries/day
  • Yields ~250 queries/second = (2M * 10) / (24 * 60 * 60)
  • 4 comments per video = 4 rows/query
  • ~1,000 bytes per comment
  • Results in reading ~1MBS = 250 * 4 * 1,000

Growth Rate

  • How big must the cluster be just to hold the data?
  • Given the write throughput, we can calculate growth

    • What is the new/update ratio?
    • What is the replication factor?
    • Additional headroom for operations

Growth Rate Example

  • Throughput assumptions:

    • 100KBS write throughput
    • 20% write updates
    • Yields effective growth of 80KBS = 100KBS * (1 - .2)

  • Replication assumptions:

    • Replication factor of 4 (2 in each of 2 DCs)
    • Yields effective growth rate of 320KBS = 80KBS * 4

  • Headroom requirements for anti-entropy operations

    • No more than 50% loading
    • Yields an effective growth rate of 640KBS = 320KBS / .5

  • Node capacity assumption: 3TB max using SSD (1TB with HDD)

    • Results in filling 1 node every other month = (640K * 60 * 60 * 24 * 30) / 3T

Latency

  • Calculating cluster capacity is not enough
  • Understand your SLAs

    • In terms of latency
    • In terms of throughput

  • Relevant factors

    • IO Rate
    • Workload shape
    • Access patterns
    • Table width
    • Node profile (i.e., cores, memory, storage, network)

  • Improve estimates with benchmarking
  • Workload shape has to do with the distribution of operations.
  • Access patterns has to do with what an operation consists of.

Future capacity

  • Validate your assumptions often
  • Monitor for changes over time
  • Plan for increasing cluster size before you need it
  • Be ready to draw down if needed

cassandra-stress

A stress-testing utility for benchmarking and load testing a Cassandra cluster

  • Simulates a user-defined workload
  • Use the cassandra-stress to:

    • Determine schema performance
    • Understand how your database scales
    • Optimize your data model and settings
    • Determine production capacity

Try out your database before you go into production

  • The data model is the number one factor affecting application performance

    • The performance impact of a data model may not be obvious
    • You will want to explore how your data model will perform

  • Load testing with several trials exposes issues with your data model
  • cassandra-stress helps you:

    • determine schema performance
    • determine how your database will scale
    • tune and test data model settings
    • verify production growth and capacity

Use YAML to configure cassandra-stress

  • The YAML file lets you:

    • Define your schema
    • Specify any compaction strategy
    • Create a characteristic workload
    • Without writing a custom tool

Use YAML to configure cassandra-stress

  • The YAML file is split into a few sections:

    • Schema Description — defines the keyspace and tables the test will create and use
    • Column Descriptions — outlines how to create the simulated data for the columns
    • Batch Descriptions — defines the data insertion pattern
    • Query Descriptions — defines the possible queries the test performs

Schema Description

  • Defines the keyspace and table information
  • If the schema is not yet defined the test will create it
  • If the schema already exists, then only defines the keyspace and table names

Schema Description Example

# Keyspace Name
keyspace: stresscql

# The CQL for creating a keyspace (optional if it already exists)
keyspace_definition: |
  CREATE KEYSPACE stresscql WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
# Table name
table: blogposts

# The CQL for creating a table you wish to stress (optional if it already exists)
table_definition: |
  CREATE TABLE blogposts (
        domain text,
        published_date timeuuid,
        url text,
        author text,
        title text,
        body text,
        PRIMARY KEY(domain, published_date)
  ) WITH CLUSTERING ORDER BY (published_date DESC)
    AND compaction = { 'class':'LeveledCompactionStrategy' }
    AND comment='A table to hold blog posts'

Column Descriptions

  • Describes how to generate the data for each column
  • The data values are meaningless, but simulate the patterns in terms of size and frequency
  • Generated values follow a specified distribution such as Uniform, Exponential, Gaussian…​
  • Parameters include:

    • Data size — how many characters are in the data value
    • Value population — how often values re-occur
    • Cluster distribution — the number of values for the column appearing in a single partition (cluster columns only)

Column Descriptions

  • possible distributions are:

    • EXP(min..max) — An exponential distribution over the range [min..max]
    • EXTREME(min..max,shape) — An extreme value (Weibull) distribution over the range [min..max]
    • GAUSSIAN(min..max,stdvrng) — A gaussian/normal distribution, where mean=(min+max)/2, and stdev is (mean-min)/stdvrng
    • GAUSSIAN(min..max,mean,stdev) — A gaussian/normal distribution, with explicitly defined mean and stdev
    • UNIFORM(min..max) — A uniform distribution over the range [min, max]
    • FIXED(val) — A fixed distribution, always returning the same value

Column Descriptions Example

columnspec:
  - name: domain
    size: gaussian(5..100)       #domain names are relatively short
    population: uniform(1..10M)  #10M possible domains to pick from

  - name: published_date
    cluster: fixed(1000)         #under each domain we will have max 1000 posts

  - name: url
    size: uniform(30..300)

  - name: title                  #titles shouldn't go beyond 200 chars
    size: gaussian(10..200)

  - name: author
    size: uniform(5..20)         #author names should be short

  - name: body
    size: gaussian(100..5000)    #the body of the blog post can be long

Batch Descriptions

  • Specifies how the test inserts data
  • For each insert operation, specifies the following distributions/ratios:

    • Partition distribution — The number of partitions to update per batch (default FIXED(1))
    • Select distribution ratio — portion of rows from a partition included in particular batch. (default FIXED(1)/1)
    • Batch type — The type of CQL batch to use. Either LOGGED/UNLOGGED (default LOGGED)

Batch Descriptions Example

insert:
  partitions: fixed(1)        # Partition key is the domain - insert one per batch

  select:     fixed(1)/1000   # 1000 posts/domain so 1/1000 allows 1 post per batch

  batchtype:  UNLOGGED        # Unlogged batches

Query Descriptions

  • You can specify any CQL query on the table by naming them under the queries section
  • fields specifies if the bind variables should be from the same row or across all rows in the partition
queries:
   singlepost:
      cql: select * from blogposts where domain = ? LIMIT 1
      fields: samerow
   timeline:
      cql: select url, title, published_date from blogposts where domain = ? LIMIT 10
      fields: samerow

Test Inserts Example

cassandra-stress user profile=blogpost.yaml ops\(insert=1\)
  • Without any other options stress will run our inserts starting with 4 threads and increasing them until it reaches a limit
  • All inserts are done with the native transport (CQL as opposed to Thrift) and prepared statements

Test Queries Examples

cassandra-stress user profile=blogpost.yaml ops\(singlepost=1\)
cassandra-stress user profile=blogpost.yaml ops\(timeline=1\)

Mixed Inserts and Queries Example

cassandra-stress user profile=blogpost.yaml ops\(singlepost=2,timeline=1,insert=1\)
  • We can also run many types of queries and inserts at once
  • This syntax sends three queries for every one insert

Exercise 2—​Configure and Run Cassandra Stress