Screening of Performance Testing Profiles

For a recruiter sourcing performance testing profiles, filtering relevant profiles is one of the most challenging task. They need to have better job description and set of questions to take informed decision. This article provides pointers around this. There are many steps involved for sourcing the profiles but here in this article, two important steps will be discussed in detail. (1) Initial Screening of Resumes (2) Initial Phone Screening.

The Initial Screening of Applications and Resumes

In this step, applications are screened to see if they meet minimum qualifications for the job. The goal is to weed out fake profiles, below average profiles and profiles not matching the existing job openings. Below criteria can be used for initial screening of performance testing profiles.
P = Performance Engineers
L = Performance Leads
A = Performance Architects

Criteria	Description
Relevant Experience	It is one of the most important criteria for filtering during the initial screening. The candidate might be having many skills set except the one mentioned in JD. Relevancy is dependent on matching skill(s) and number of years of experience with it. Both need to be considered while screening the resumes. The Following skills relevancy requirements should be available in JD for better screening – » Load testing tool (e.g. Load runner, JMeter, Silk performer etc.) » Protocols (e.g. web, AJAX, Oracle NCA etc.) » Monitoring tools (e.g. AppDynamics, Diagnostics etc.) » Operating system (e.g. Windows 2012, Linux etc.) » Platforms (e.g. IIS, JBoss, Websphere, Tomcat etc.) » Databases (e.g. SQL Server, Oracle, DB2, MySQL etc.) » Utilities (e.g. netmon, wireshark, httpwatch etc.) Note: » It has been observed that performance engineers have used Load runner with web protocol, whereas requirement could be for Oracle NCA protocol instead. Sometimes, understanding on protocol is more important than load testing tool. It should be clearly specified in JD. » Roles and responsibility section of resume can be searched for skills relevancy and duration of project can decide the number of years’ experience with it. » Experience in any skill with less than 3 months of experience can be considered as “Nice to have” from JD and must not be considered as relevant experience.
Recommended Experience	Apart from relevant experience, below is general guidelines that can be used for filtering the resumes for performance testing roles. The skills requirement is must for the roles mentioned in right side. » Experience in at least one monitoring tool (e.g. AppDynamics) – PLA » Experience in bottleneck analysis in at least one platform (e.g. windows, Java) – LA » Experience in at least one development language (e.g. C#, Java, Perl) – LA » Experience in doing at least one end-2-end project (requirement phase to reporting) – LA » Experience in completing at least one long duration (7-8 months) project - LA » Experience in setting up at least one platform (e.g. SQL Server, JBoss, IIS) - A
Preferred Profiles	In case number of profiles are large in numbers, below added criteria can be used for giving the preference – » Experience at onsite location (client location) » Experience in both open source and commercial tools » Experience in waterfall analysis using tool like HTTP Watch, Webpagetest.org etc. » Candidates with Certifications » Candidates with any kind of development experience in past » Candidates, having authored articles, whitepapers, presented in a seminar etc.

The Initial Phone Screening

In this step, filtered resumes in earlier step is called on phone to have another check before it can be shared with delivery. The goal is to weed out fake profiles from real profiles. In this step, majorly success depends on judgment and experience of recruiter. Below are some questions on performance testing that can be asked over the phone and feel the pulse and confidence of candidates. There are no ‘What Is’ questions rather questions related to their experience. It is observed that the candidates with false experience usually flattened out after few remarks. Each question is suffixed with role, which should be considered while screening.
P = Performance Engineers
L = Performance Leads
A = Performance Architects

Question Type	Sample Questions
Practical experience in tools and platforms	» Tell me at least five challenges faced while working with load testing tool (PLA) » Tell me at least complex features of load testing tool that you have used and why (PLA) » What are the top three protocols you are comfortable with and worked for more than 6 months or as mentioned in JD (PLA) » How any other protocols like Oracle NCA, SAP etc. are different than web protocols (LA) » Tell me in detail how web protocol works(LA) » How did you monitoring tool and what challenges did you face while setting up the monitoring tool? (LA) » Did you setup any tool in UNIX / Linux environment? What were the activities that you performed while working with Linux environment? (PLA) » Tell me at least three utilities that you have used, which comes with OS and platform? (PLA) » Have you developed SQL queries and executed them against database? If yes, what was the reason for developing the SQL queries? (LA)
Planning	» What would be your top 10 questions that you will ask to stakeholders for gathering the requirements? (A) » What activities are performed during performance testing of any application? (LA) » Explain the basic requirements of performance test plan? (LA) » What are top 10 criteria that you consider while doing tool feasibility analysis? (LA) » What are different licensing model you are aware of? (A)
Scripting	» What are the top five items you focus while designing scripts? (PLA) » Have you developed test data for scripting? What were different ways you parameterized the test data? (PLA) » Have you reviewed the scripts designed by others? What were your checklist? (LA) » What are different scripting issues you faced so far? Tell me at least five. (PLA) » Did you handle errors in your scripts? How? (PLA)
Execution	» How many virtual users can be generated from one desktop class machine? (PLA) » Had you generated load from cloud? How is it different from generating load from local network? (LA) » What are some of the things you consider before conducting final performance run? (LA)
Bottleneck Analysis	» What are the top three issues you identified and reported when working with monitoring tool? (LA) » What would be your approach in identifying the bottlenecks when you find response time is high? (A) » Tell me how network bottleneck could affect the performance of the application? (PLA) » How to identify a memory leak? (LA) » Apart from CPU, Memory, Disk & Network, what else do you monitor and why? (LA) » How did you find web server / database server related issues? (LA) » What were the performance bottlenecks that you found in projects you were working? What are the recommendations made to overcome those issues? (LA)
Reporting	» List those three information that you won’t share with executives whereas it is required to be submitted to technical team (LA)
Technical	» How caching works? What are some different caching mechanisms used in multi-tier application? (A) » What is extrapolation and when it is used? (LA) » What do you understand by vertical and horizontal scaling? (LA) » What are the changes that you can make during run-time settings and why? (PLA) » Explain the configuration and architecture of your system (PLA) » What is the difference between baseline and benchmark testing? (LA) » What is correlation and parameterization? Why it is important? (PLA) » Have you heard about Little’s Law? What is it? How did you apply in your project? (PLA) » Can you define following performance testing terms in one sentence? Server time, HTTP status code, receive time, round trip time, time to first byte? (LA) » In the era of cloud computing, what attribute of software applications that are tested using load tests do you consider as the most important? (A)
Initiatives	» Had you initiated anything in parallel to your regular activities? What were they? (PLA) » What is one problem you see in your team, which you would like to solve? (PLA)
Flexibility	» Will you be interested in travelling to onsite for 3 months? Do you have any concerns? (LA) » What was your most time consuming projects? What is your average working hours? (PLA) » Have you worked during weekends? Why? What was the need in the project? How many times it happened? (PLA)