Automation Tool Feasibility Analysis

During this phase focus should be primarily on identification of test automation tool(s) from the technical perspective. The finalization of tool can be worked upon later after due diligence of all factors collected, during this phase.

Prerequisites
  • Test environment should be ready and available for feasibility analysis 24*7
  • This environment must be dedicated for tool feasibility activity to avoid any schedule slip & test data corruption

Identification of test scenarios & test data
  1. Number of scenarios should be minimal (to avoid high commercial cost)
  2. Should cover all screens & all possible unique UI components of the application (e.g.)
    • Editing of Grid control
    • Picking dates from Calendar
    • Filtering and selecting items in ListBox
    • Attachments of Files
    • Popups
    • Trees & Submenus
    • Reading value from Graphs (Bar, Chart etc.)
    • Dragging and dropping of items
  3. All scenarios must be accompanied with 3-5 different data sets (positive flow, negative flow, boundary values)
  4. Every step in a scenario should be accompanied with 2-3 different assertions. Usually assertion is done with text label only (easy way) but it leads to passing of automation, which shouldn’t be the case. If possible, resulting data value should be read and compared against expected.
  5. If possible, test a scenario where a component is upgraded and check maintenance cost involved in fixing the script

Test suite execution
  • Should be able to select and execute any scenario(s)
  • A test scenario should be able to execute multiple times in a loop w/o manual intervention
  • Should be able to execute multiple scenarios in parallel
  • Test automation should support below platforms and browsers
    • Windows – IE 6.0 and above
    • Windows – Firefox
    • Windows – Chrome
    • Linux – Firefox
    • Linux – Chrome
    • Mac – Safari
    • Mac – Chrome
    • Android - Chrome
    • iOS - Safari

Expected output from tool feasibility
  • Demo on multiple tools (e.g. Selenium, Ranorex, Test Complete etc.)
  • Tool feasibility & comparison report
  • Script design productivity information for each tool
  • Script maintenance cost for each tool
  • License pricing of tool (1st year and onwards)
  • Cost of execution in parallel (assume 10 machines)

Factors to consider for tool feasibility
Below is a list of comprehensive factors to determine and finalize an automation tool. Of course weightage of each factor depends on specific needs of teams.

FactorCategory
Identification of all UI components of the applicationScripting
Existing Framework (in-built framework)Scripting
Support on mobile platforms like iOS, Android (in future)Scripting
Support of scripting languageScripting
External support, in case of any issues found during scripting or executionScripting
Support BDD frameworkScripting
Support script less automationScripting
Data driven testingScripting
Execution support on all browsers (IE, Chrome, Firefox, Safari, Edge)Execution & Reporting
Ease of use in various environments (AUT)Execution & Reporting
Ease of reporting and understandingExecution & Reporting
Integration with Issue TrackerExtensible
Support for Unit TestingExtensible
Execution Flexibility (should be able to execute from anywhere)Flexibility
Script design flexibility (should be able to design script from any machine)Flexibility
Script designing license costCost
Runtime license costCost
Resource Experience levelCost
Availability of resources in the marketCost
Training (to Jr. Associate)Cost

Comments

Post a Comment

Popular posts from this blog

Test Case Template for Project Using Agile Methodology

User Story One template that can be used while writing user stories could be – “As a (role) I want (functionality) so that (benefit).” For example – As a ‘Bank Customer’, wants to ‘Transfer Funds’ to one of their friends / acquaintances online, which is fast and auditable. Or, the user story could be described as a single line item (2-5 words) as well. For example – Fund Transfer Test Cases Few of the test cases (summary) for above user story could be as follows.  These test cases should be developed with the help of Business analyst. Transfer fund when fund is in range of acceptable limit and existing balance is sufficient. Or, Fund transfer amount less than existing balance Transfer fund when fund is in range of acceptable limit but existing balance is not sufficient. Or, Fund transfer amount more than existing balance Transfer fund when fund amount is beyond acceptable limit. Or, Fund transfer amount more t...
Read more

Measurement & Metrics

"You cannot improve what you cannot measure" A lot had been talked and spoken already about ‘Measurement & Metrics” in Software Testing.  In this article I am not saying that something new will be written or discussed, but my idea for this article is to summarize on this topic from various different sources. Measuring, Monitoring and Acting on it are key roles of any manager in any industry and Software Testing is no different.  A lot of times it has been seen that KPIs are being measured and monitored but not being acted.  In that case, this whole exercise becomes worthless.  Sometimes, the measurements themselves are not collected and reported correctly, which is more dangerous than not doing this activity at all. Other times, it had been observed that there are numerous KPIs are being monitored without having any goals in mind.  As per me, the number of KPIs that should be measured and monitored should be minimal and any addition to new KPIs sho...
Read more

Performance Test Run Report Template

You have gathered requirements from clients, identified key critical business scenarios, analyzed the workload distributions, found out performance goals and requirements, identified  tool for generating load, setup the test environment, designed the scripts, executed the performance run, collected the raw data, analyzed it, what next?  Send information to stakeholders? No!!  Wait!! It needs to formatted, customized for different stakeholder(s) and reported accordingly. This article suggests one of the ways to report performance run results to client especially if the performance testing activity has been outsourced to yours company.  It discusses the performance report template and importantly the reasons for including the items in the report. The performance report template contains information by data collected from client side only and doesn’t discuss / talk about data from server side like processor utilization etc. Broadly the performance report has been divi...
Read more