Best Practices for Unit Test Case Automation

Make each test orthogonal (i.e., independent) to all the others

• Any given behavior should be specified in one and only one test. Otherwise if you later change that behavior, you’ll have to change multiple tests

Don’t make unnecessary assertions

• Which specific behavior are you testing? It’s counterproductive to Assert() anything that’s also asserted by another test
• It just increases the frequency of pointless failures without improving unit test coverage at all
• Have only one logical assertion per test
• Unit tests are a design specification of how a certain behavior should work, not a list of observations of everything the code does

Test only one code unit at a time

• Architecture must support testing units (i.e., classes or very small groups of classes) independently, not all chained together
• If you can’t do this, then your architecture is limiting your work’s quality – consider using Inversion of Control

Mock out all external services and state

• You’ve definitely taken a wrong turn if you have to run your tests in a specific order, or if they only work when your database or network connection is active.
• Behaviour in those external services overlaps multiple tests, and state data means that different unit tests can influence each other’s outcome
• If you can’t, at least make sure each test resets the relevant statics to a known state before it runs.

Avoid unnecessary preconditions

• Avoid having common setup code that runs at the beginning of lots of unrelated tests

Don’t unit-test configuration settings

• By definition, your configuration settings aren’t part of any unit of code

Name your unit tests clearly and consistently

• Avoid non-descriptive unit tests names such as Purchase() or OutOfStock()
• One naming convention could be - Action_ConditionsToTest_ExpectedResult. Example: ProductPurchaseAction_IfStockIsZero_RendersOutOfStockView()

Follow consistent test structure

• Setup -> Action -> Assert -> TearDown

Memory Utilization & Garbage Collection Analysis

• The goal should be to optimize garbage collection (GC) in such way that its impact on application response time or CPU usage is minimized.
• Monitor the utilization of different memory pools (young, survivor and old). Memory shortage is the number-one reason for the increased GC activity
• The young generation should be big enough to ensure that all temporary objects die there
• If overall memory utilization is growing continuously despite garbage collection, it is an indication of memory leak and requires heap analysis
• Monitor the churn rate (number of object allocation per interval) at the younger generation. The high number of young collection can cause high response time. It can cause unnecessarily copying of objects from young generation to old generation
• Free tools that can be used for GC analysis are:
  • JConsole, jStat, Java VisualVM, JRockit Mission Control, verbose:gc flag of the JVM
• A high GC activity generally has a negative effect on CPU usage and not necessarily the response time of application, whereas only suspensions affects directly the response time of application. JVM suspensions should be monitored to find out whether the root causes lie in:
  • The younger generation, Object churn, Old generation, wrong sizing or memory leak
• Case where old generation utilization rises continuously but come back to normal after GC, indicates the high churn rate. It might not be enough to increase the young generation's size, which will be accommodating more live objects, which in turn will lead to longer GC cycles. The best optimization is always to reduce the number of allocations and the overall memory requirement
• It is important to differentiate between young and old generation GC and equally important is noticing frequency and duration of those GC. For the young generation, duration might be low but frequency might be higher due to frequent allocation of objects. The high GC of young generation can be result of either too small memory allocation for young generation or high churn (allocation) rate. The root cause for higher allocation need to be analyzed
• The GC time should never take up more than 10% of the total CPU time used by the application
• Optimize algorithm by unnecessary allocations or allocating the same objects multiple time e.g. Creating same temporary objects in a loop.
• JVVisualVM can be used to find out the root cause of high allocation (churn rate)
• High memory utilization is a cause for excessive garbage collection. Increasing the heap size of JVM might solve the problem. But increasing the heap size not always solves the problem but delays the problem and in that case GC takes more time.
• Memory usage can be analyzed using Heap dump. Multiple tools that can be used for analyzing heap dumps are - jmap, jhat, VisualVM, JRockit Mission Control
• Analysis of memory results are - a) Identification of memory leak b) Identification of memory eaters
• Multiple heap dumps are needed and compared against trends to find objects that are leaking. For analysis, It is advisable to stick to classes that are created by your application
• The purpose of heap dump analysis is to find out memory leak or whether cache (e.g. HashMap) is using too much memory

Checklist for Designing Better Performance Test Scripts

Make sure the script doesn’t contain incorrect or extraneous URLs. The urls specified should be in correct sequence.

• It might be possible while recording, script writer would have gone to his / her popular website.
• It can be validated by using test tool’s “playback” feature to confirm what the script actually does.

Identify all dynamic data (as the response from server) present in a script and correlate it.

• Usually it can be found by recording scripts two times and making comparisons between them.

Parameterize scripts to support dynamic data set.

• In presence of dynamic data, every simulated user exercises the exact same path but avoids responses from the cache and exercises database interactions properly.

Use proper checkpoint(s) / assertion(s) for all steps involved in the transaction.

• Absence of checkpoint might result in better response time when a page is not getting downloaded completely / correctly.
• Text used for assertion should be either static or should be consistent accross all runs / environments. If this is not done properly, scripts maintainence becomes an overhead. For example, if name of top selling car used as assertion, the script might start failing after few days once top selling car name changes.

Confirm if your performance test tool handles cookies automatically.

• If the website under test set cookies, these cookies might appear in the recorded scripts and it needs to be handled explicitly by script designer by using variable. The variable allows the script to receive a different cookie value during the test, rather than using the recorded value.
• Cookie substitution is must if the site uses HTTP session cookies from an application server.

Check the think time and pacing time in the script.

• It is not recommended to use the constant think time value for every step or for each user.
• Check if your tool supports distributing think time values in a range instead.
• The think time value & pacing time value should be planned and finalized during performance requirement gathering phase.

Users rarely log out of web site, so don’t assume the same and design scripts accordingly.

• Logging out every time, might clear http session information from the cache sooner than actual.

Verify the scripts during design time

• Validate it with one iteration and one user
• Validate it with multiple iterations and one user
• Validate it with multiple iterations with multiple concurrent users

Scripts should be written in a way so that it can be executed against multiple environments without any significant changes

• Different environments can be test, stress, pre-production etc.

Consider building scripts for the primitive paths first

• It helps in easy troubleshooting and optimization

Final scripts should be representative of actual user activities.

• Should not be too simple and too focused, until is it really required.

Look out for re-usability while designing the scripts.

• Develop simple scripts to build more complex scripts and scenarios.
• All simple scripts should be atomic in nature.

Follow standard naming conventions and folder structures.

• Resist temptation of using default values (e.g dir path, log file location) provided by tool(s). Understand the consequence of each setting and then apply it
• Helps in readability and reviewing of scripts