Performance Testing of Large Scale Distributed System

Traditionally performance testing approach usually follows Identification of Key Scenarios, Setting up the Load Environment, Designing the Scripts, Generating load, Monitoring and at last Analysis and Reporting.  It works for most of the system but it is completely a different ball game when conducting performance testing of large scale distributed system.  In this article, some of the key characteristics and challenges will be discussed for performance testing of large scale distributed system.

Key Characteristics

High Volume

»        Terabyte of transactional records in database

»        Network throughput in gigabits per second

High Transactions

»        Millions of transactions per second from end users

»        Millions of transactions in database due to few triggers (e.g. large report generation due to batch processing)

High Concurrency

»        Huge user base accessing simultaneously (e.g. Facebook)

Geographically Distributed

»        Traffic from all over the world

High availability

»        Huge revenue loss and complaining customers due to unavailability

Huge Data Analytics

»        Big data, data warehousing

 

Key Challenges

Proposed Solution

High cost for test environment setup

»        Production or staging environment

»        Scaled down environment

High cost for load generation environment

»        Cloud based load generation tool

High license cost for tools and utilities

»        Open source load generation / monitoring tools

»        Use pay per service if number of runs are less

Production like environment configuration

»        Use CI tool like Jenkins for automatic build and deployment

Configuration consistency for large number of nodes

»        Automatic validation of configurations before and after the execution

»        Take restore point and roll it back after the performance run

Population of high volume of test data

»        Copy production data and mask it

»        Alteration of DB volumes

»        Use tool like database benerator, dbmonster

»        Use historic data

Simulation of realistic load

»        Identify key scenarios and usage patterns from log files, market research, BA etc.

»        Generate load from different geographies

»        Baseline response with CDN and without CDN

Penetrating the system complexity, touching all system nodes and database tables

»        Understand system architecture

»        Manually walkthrough the scenarios and watch traffic on different nodes and database tables

»        Detail analysis of application logs

»        Understand load balancer strategy

Identification and testing of failover scenario

»        Test the failover scenario separately during load condition

Third party interactions

»        Simulate using stubs

Monitoring of large number of disparate systems

»        Use diagnostic tools like AppDynamics, DynaTrace, HP Diagnostic, Glassbox

Result collation and analysis

»        Automatic result collection and collation

»        Collection of built-in anti patterns for quick analysis

»        Knowledge base on historical failures or bottlenecks

 

Comments

Post a Comment

Popular posts from this blog

Test Case Template for Project Using Agile Methodology

User Story One template that can be used while writing user stories could be – “As a (role) I want (functionality) so that (benefit).” For example – As a ‘Bank Customer’, wants to ‘Transfer Funds’ to one of their friends / acquaintances online, which is fast and auditable. Or, the user story could be described as a single line item (2-5 words) as well. For example – Fund Transfer Test Cases Few of the test cases (summary) for above user story could be as follows.  These test cases should be developed with the help of Business analyst. Transfer fund when fund is in range of acceptable limit and existing balance is sufficient. Or, Fund transfer amount less than existing balance Transfer fund when fund is in range of acceptable limit but existing balance is not sufficient. Or, Fund transfer amount more than existing balance Transfer fund when fund amount is beyond acceptable limit. Or, Fund transfer amount more t...
Read more

Measurement & Metrics

"You cannot improve what you cannot measure" A lot had been talked and spoken already about ‘Measurement & Metrics” in Software Testing.  In this article I am not saying that something new will be written or discussed, but my idea for this article is to summarize on this topic from various different sources. Measuring, Monitoring and Acting on it are key roles of any manager in any industry and Software Testing is no different.  A lot of times it has been seen that KPIs are being measured and monitored but not being acted.  In that case, this whole exercise becomes worthless.  Sometimes, the measurements themselves are not collected and reported correctly, which is more dangerous than not doing this activity at all. Other times, it had been observed that there are numerous KPIs are being monitored without having any goals in mind.  As per me, the number of KPIs that should be measured and monitored should be minimal and any addition to new KPIs sho...
Read more

Performance Test Run Report Template

You have gathered requirements from clients, identified key critical business scenarios, analyzed the workload distributions, found out performance goals and requirements, identified  tool for generating load, setup the test environment, designed the scripts, executed the performance run, collected the raw data, analyzed it, what next?  Send information to stakeholders? No!!  Wait!! It needs to formatted, customized for different stakeholder(s) and reported accordingly. This article suggests one of the ways to report performance run results to client especially if the performance testing activity has been outsourced to yours company.  It discusses the performance report template and importantly the reasons for including the items in the report. The performance report template contains information by data collected from client side only and doesn’t discuss / talk about data from server side like processor utilization etc. Broadly the performance report has been divi...
Read more