Test Automation for SaaS Products

Software as a service is a popular delivery model for various business application domains including content management, service desk applications, accounting, invoicing, human resource management, CRM and ERP.

Versus long-term software licensing of client packages, cloud delivery of SaaS enterprise apps offers greater convenience, lower risk and the economic advantages of reduced capital and operational expenditures with pay-as-you-go subscriptions.

Challenges of SaaS Testing

SaaS testing poses unique challenges:

  • Frequency of Releases – The instant delivery model of SaaS means shorter release cycles, which implies an agile or DevOps development and testing practices.
  • Functionality and Performance – SaaS implies no product versioning, thus a new release affects all users at once. So, functionality and performance testing must ensure the absence of negative impacts.
  • Cross-Browser Compatibility – A SaaS deployment needs a single OS but must present itself consistently across browser brands.
  • Scalability and Accessibility – Understanding how a SaaS app responds to increased load, zero to peak load rates, demand from multiple sources, such as mobile users, and meeting SLAs are all critical aspects of scalability testing.
  • Data Integration – If a SaaS application is part of a suite data sharing must be thoroughly tested, which is especially difficult if suite members follow different release schedules.
  • Usage Models – Configurability of SaaS apps is critical for adoption by enterprises who each have their unique ways of doing business. Testing configurability requires understanding your users’ business models to optimize which workflows to prioritize.
  • Security – The convenience of anywhere access of SaaS means testing the strength of a service’s identity management and data security measures is critical.
  • License Functionality – Without sound licensing policies that are correctly enforced, both users and providers suffer, so each release must re-verify that licensing is being correctly enforced.

SaaS Testing Advantages

Many traditional application testing aspects are not applicable to SaaS testing. SaaS does not require multiple version support nor will the software run on multiple OS platforms. Support for client or in-house servers is also unneeded as is backward compatibility testing.

Applying SaaS Test Automation

Test automation improves test coverage in the areas mentioned previously as well as complementing agile or DevOps environments required for SaaS products’ compressed release cycles. It does not completely preclude manual testing, however, which can suggest new test cases using exploratory testing immediately after each code release.

Test Automation as a Development Project

When approaching the unique aspects of SaaS testing automation, we find it extremely helpful to constantly identify areas where automation may be employed from unit testing to production.

Additionally, the most successful automation efforts, in our experience, create test automation capability via the same development methods used for the code itself:

  • Develop a detailed test plan designed to keep pace with agile or DevOps practices
  • Plan for continual test documentation maintenance
  • Start with a framework-driven approach that includes defect tracking functionality
  • Always strive for reusability in your tests, which promotes less brittle tests while saving time
  • Develop a solid test reporting protocol that rapidly returns results to the developers
  • Hire testing experts with coding skills as no test automation tool completely substitutes for such skills

Measure Success

We have seen many test automation efforts fail because no metrics were put in place from the beginning to measure success or failure and guide corrective steps as the project progresses.

Start by setting goals for the percentage of test cases that are automated for each area of testing, a percentage improvement in test coverage and the percentage improvement of time to release. Simply applying these metrics will suggest other useful metrics to apply as your test automation capability matures. Strive for metrics that are relevant to assessing the overall ROI of your test automation efforts.

Test Automation Framework Capabilities

When building or buying a test automation framework assess its maintainability to avoid it falling into disuse in a short time. For instance, if new tests are difficult to add to existing collections, testers will resist augmenting test coverage.

Your automation framework should accommodate testers with coding skills but reduce complexity for those testers without such skills. It must separate scripts and data, handle multiple data formats, such as CSV files, plain text, XML and so on.

If a candidate test framework does not include a test manager, defect management tool or continuous integration and continuous development capability, assess the effort required to integrate those critical functions.

Strongly consider using a cloud-based automated test framework such as Saucelabs, BrowerStack, TestingWhiz, ITTester, etc. to lower costs and increase scalability. Cloud deployment services such as AWS, Azure or Google Cloud Compute also offer the ability to virtualize test automation instances using your preferred hypervisor to obtain greater efficiencies.

Consult with Test Automation Experts

It cannot be overemphasised that SaaS test automation is a complex, long-term endeavour with unique requirements that may easily overwhelm an enterprise’s QA capability. Thus, it often pays to enlist experts who have been down this road many times before. Such a move is especially critical given the short release cycles inherent in SaaS products and the value of thorough testing across multiple domains in the highly competitive SaaS market.


If you have questions about getting started with Test Automation or want to push the toolset further, give us a call. We’re always happy to answer any questions.


Testing in an Interconnected World: Event Recap

How to test applications as they become increasingly universal

In case you missed it, the latest instalment of the Optimus Breakfast Series went off Tuesday morning, June 13th, 2017. The event brought together three of the top software testing executives from Vancouver’s leading companies to talk software testing and the complexities they face in our new interconnected world.

This was a sold-out event that drew over 60 attendees from 35 different organizations!


Moderated by Pankaj Agarwal – Founder and Managing Partner at Optimus Information

Steve Whitfield – Technical Director, Mobile Apps at lululemon athletica

Stu Ashman – QA Director at Mio Global

Larry Ng – Director of Quality at Visier

We would like to extend a big thank you to those who joined us! If you missed it, join our mailing list in the right pane of this page to get notified about our upcoming events.

Stay tuned for the complete video recording to be posted on the Optimus blog.

Fun fact: Larry Ng will also be speaking at VanQ, our QA and Testing Meetup, on Thursday, June 29th. He’ll be speaking on Endpoint Testing with Gatling. Check out the details and register here.

Highlights from the Morning

IMG_20170613_090834-1030x773 Testing in an Interconnected World: Event Recap IMG_20170613_080726-1030x773 Testing in an Interconnected World: Event Recap IMG_20170613_075813-1-1030x773 Testing in an Interconnected World: Event RecapIMG_20170613_093215-1030x773 Testing in an Interconnected World: Event Recap
IMG_20170613_093234-1-1-1030x773 Testing in an Interconnected World: Event RecapIMG_20170613_093255-1030x773 Testing in an Interconnected World: Event Recap

Automated Testing for SSRS Reports

Motivations for SSRS Report Testing

Both data warehouse developers and end users of data analytics reports have a keen interest in the accuracy and appearance of data content. As SSRS reports are being developed, they are typically tested piecemeal during construction and as a whole when completed for both these aspects.

However, it is always possible to overlook certain report items or their representation after report deployment. Furthermore, issues with changes in data sources or the ETL flow may introduce anomalies or errors that affect data values or their presentation at a later date.

SSRS Report Testing Automation Issues

To increase report testing efficiency and accuracy, automated testing is key during both development and maintenance of SSRS reports, especially for organizations utilizing dozens or hundreds of reports on a regular basis. Automation can decrease development time and play an important role in correcting discrepancies in report data post-deployment, which could otherwise negatively impact confidence in the data warehouse.

The complexity of interactions within SSRS itself and the many other components of SQL Server, however, make the creation of fully automated testing a tricky business. Furthermore, the high-degree of customization possible for SSRS reports implies that customized testing approaches are probably required in addition to standardization of report content and layout wherever feasible.

SSRS Testing Areas for Automation

Unit Testing

There are a large number of characteristics that can be tested during development via unit tests. Unit tests can migrate to test suites for troubleshooting post-deployment bugs quickly. These are a few example coverage areas for such automated tests:

  • If data is not reaching a report, validate the data source, check that the correct mappings to the dataset are being used and that each column’s Visibility property is set correctly.
  • If data is not present and the report uses stored procedures, validate the SP parameters, their data types, conversions and formatting requirements.
  • The failure of data to group as expected can be tested by examining the grouping expressions and that the Aggregate function is applied to Numeric columns.
  • Create a query test framework that can take as input individual queries or query scripts. Such a framework would contain validated comparison datasets including metadata to run against actual queries/datasets with each report’s specific data source, parameters or configuration.
  • Test report rendering into formats used by the organization. Initial tests require manual inspection of results, but subsequent tests could be automated using input reports under version control. The most useful of these tests is to output the report in XML format, which is likely to be complete, free of data conversions and most amenable to automated comparison tests.

A possible starting point for building a permanent RS unit testing framework can be found here: http://rsunit.codeplex.com/

Layout Testing

Automating report layout tests typically presents the greatest difficulties. If your organization uses the SSRS web portal, however, you can take advantage of a number of web UI automation tools that facilitate such testing.

Selenium Web-Driver is a free tool for testing web page layout and functionality, which works with Firefox, IE, Chrome and Safari. Automation test scripts are written in either Java, C#, Ruby, Python or JavaScript. Scripts utilize the Web-Driver API which invokes a live browser or runs headless.

Other UI-based testing tools are also available such as the open-source Sikuli, freeware AutoIt or TestComplete, which is a proprietary tool by SmartBear.


Test automation has its limits, of course. This is especially so with regard to SSRS reports testing. For instance, usability testing is clearly out of scope for automation. Automation is further complicated in SSRS, since data is often manipulated beneath the covers beyond control of the report itself. For example, data may be manipulated outside of queries in embedded VB.Net code or in external class libraries.

Even so, automating report testing wherever feasible always pays off. Choose the low-hanging fruit first before expanding into other areas, such as troubleshooting suites and GUI-based layout testing. As you progress, you are likely to find that developing a mindset aimed at automation frequently instills a virtuous cycle of discovering new opportunities for further test automation.

In SSRS testing areas that present the most difficult obstacles, employ third-party tools or employ an experienced automation consultant who can demonstrate automation methods most appropriate for your SSRS development and usage scenarios.

Dos and Don’ts of Software Outsourcing

Outsourcing software development can be a great way to save your company or organization time and money. However, if done haphazardly, it can also become a source of great headaches. It’s easy to make common mistakes, such as not clearly defining goals or expecting the process to magically produce results. It’s much harder to manage the process and see that it achieves the results you desire. Here are some dos and don’ts for you to consider as you get involved with software development outsourcing.

Do Define Your Goals and Metrics

One of the hardest things for any outsourcing services provider to deal with is a moving target. It’s important that you have clear planning documents in place that outline the goals for each project. Likewise, you also need to use clearly understood and widely trusted metrics to measure the success of a project. If you and your outsourcing provider agree on the measures of success, there’s a much better chance you’ll hit your target.

Don’t Fire and Forget

Handing a pile of specs to an outsourcing firm does not count as passing the project along. The process is inherently collaborative. It’s important that everyone on your side, the client side, stays engaged. When mockups come in, check them. When requirements are reportedly fulfilled, verify them. If errors occur, correct them. Do not plan to hand off the specs and just come back months later expecting to see a finished product.

Do Put Processes in Place

Every aspect of your project should have a clear process in place to ensure that both you and your outsourcing company understand what it means for a step to be considered complete. This means verifying that mockups are passed along and that both parties sign off on each step. Requirements for each stage should also be outlined clearly, and all parties involved should agree to them. Nothing dooms a project as fast as an ambiguous understanding of what it means for a step to be truly completed.

Do Communicate Well

This can be trickier than it sounds. Even if you’re running a North American firm that’s outsourcing to another firm on the continent, time differences matter. If your vendor doesn’t have flexible hours, then you’re likely losing collaboration time. For example, if a company in New York City is outsourcing to a firm in Vancouver, it’s important to know that the Vancouver folks are prepared to have someone show up early on some days to touch base with the New York crowd before the work day gets rolling.

Don’t Judge on Price Alone

One of the greatest temptations of outsourcing software development is to simply take the lowest price offered. This is a terrible idea. There are many tradeoffs required to get to the lowest price. Will you be dealing with people who speak your language natively? If the vendor is on the other side of the planet, will they even be able, at that price point, to have someone communicate with you live at a convenient time?

Do Expect a Performance Curve

If you’re beginning your first project with a specific vendor, it’s unrealistic to assume they will be able to just hit the ground running. Every outsourcing firm requires time to make sense of your company’s culture and figure out the best way to achieve the results you seek. Getting frustrated by this process can be very costly. There’s little benefit that can be expected by bouncing from vendor to vendor. Give vendors time to acclimate.

Don’t Outsource Core Functions

Some parts of your company just have to work. The best way to see that your operation works the right way is to ensure that core functions stay in-house. For example, if you ran a vinyl sign company, it would be insane to outsource your graphic design work. On the other hand, it might be perfectly reasonably to outsource backend coding for your website.

Do Prepare for the Future

No matter how strong your relationships may be with your current set of vendors, you need to know the market and be prepared for the future. If the day comes that your preferred vendor can no longer handle the scale of your work, you want to have a list of vetted alternatives in place. You can lose months of project time trying to find a new vendor.

Don’t Use Outsourcing as a Stopgap

Adding an outsourced software development team to your organization means incorporating it on a long-term basis. If you drop a project on an outsourcing company and then turn around expecting in-house people to maintain or even expand it, the results will be rubbish. It will may foster discontent among the in-house team. Look at long term partnerships where certain projects or tasks stay with your outsourcer and others stay with your in-house team. Your guys handling the outsourcing on your software will then become comfortable dealing with your in-house people handling core functions, and vice versa.

Don’t Rely on Technology Alone

There’s a lot to be said for getting in at least one face-to-face meeting, ideally more as required. Your vendor and you can both demonstrate commitment to the project. You also can interface more quickly. If at all possible, try to make in-person meetings part of the process.


Outsourcing is an amazing tool for any company to have access to. It is important, however, to appreciate that it’s not magical. It’s a process that your organization has to fully accept and integrate. With the right checks and balances in place, you can ensure that your outsourced software development efforts achieve the results you want.

Ultimately, successful IT outsourcing comes down to finding the right partner that fits with your organization and IT needs. Optimus Information has become a trust partner to companies of all sizes and in all verticals; we know what works and doesn’t work when it comes your your IT challenges and working with your organization. 

Contact us today for your next IT project. We’re always happy to help.

Start outsourcing effectively. Download our How to Overcome IT Outsourcing Challenges whitepaper.

Tactical Outsourcing vs Strategic Outsourcing

The term outsourcing scares people for various reasons, but there is no reason that it should. Outsourcing is very effective for certain situations, especially when developing software solutions. For a company to grow its internal IT department, it must search for a qualified candidate and go through the long process of interviewing, hiring, and training the new employee. This can take a long time and cost the company a lot of money. The alternative to this is outsourcing.

There are two main types of outsourcing that you can find for your IT needs: tactical and strategic.

Tactical Outsourcing

Tactical outsourcing refers to hiring a firm to perform specific development functions as part of your existing software development process. You retain oversight of the project, giving you more control over the process. This is very helpful if you need a project done on short notice because you can avoid the long process of searching for the right candidate.

Tactical outsourcing is also helpful when you have a short-term need for a highly skilled developer in a specific technology that you do not normally use. A developer with the right skill set may be very difficult to locate, so going through a skilled tactical partner can mean fast turnaround and lower overhead than hiring internal resources.

With tactical outsourcing, you maintain the planning process, including gathering requirements from stakeholders and designing the system that the developers will create. This increases management overhead, but it also allows you to have more control over the process and more knowledge about what is happening. They are familiar with your processes and answer to you throughout the process.

Strategic Outsourcing

Strategic outsourcing involves partnering with an IT company that provides top-down services. They handle the entire planning and development processes, leaving you free to focus more on other aspects of the business. This is nice for some companies because they are able to pay prices per project instead of per hour of work.

With either type of outsourcing, you should address a few key points with your prospective outsourcing partner. The first is documentation because you never know what could happen in the future. If a support issue comes up or you need new functionality added to an existing product, you will need proper documentation to know how the product is supposed to work. Ensuring that your partner provides proper documentation, both in the code and out, can be vital to handling these situations.

You should always provide full technical information about your requirements to your outsourcing partner. Leaving out any details could cause severe problems as you go through the process. Always remember to include all of your key decision makers and stakeholders in discussions to ensure that everybody’s goals align properly. Repeated changes over time can greatly increase the cost of any project, so planning everything up front is vital to a successful partnership.

You should always examine your prospective partner’s portfolio, looking for consistent, high-quality performance over time. Not all outsourcing companies are equal, and even great companies may not be the right choice if their goals do not align with yours. It is also important to note for what industries the company has worked. If the company has worked for other companies in your industry, it could mean that they are already familiar with many of your processes and terminology, making communications much easier.


Both outsourcing solutions could work for you, depending on your situation. Many companies prefer tactical outsourcing because it allows them to maintain more control over the process and usually means less of a commitment. However, the trade off is significantly more involvement from the client. 

At Optimus, we find there is a lot to gain from ongoing strategic outsourcing partnerships, and clients typically reap more benefits: less hands-on management, higher quality work, faster delivery and greater flexibility.

For an in-depth look at strategic outsourcing, check out our guide.

Download our Guide to Strategic Outsourcing


What to Look for in an Outsourcing Partner

Bad experiences with outsourcing providers are often traceable to badly designed selection processes or the use of deficient selection criteria. Whereas, you can find ample guidance online on how to build a robust selection framework, we will share the most important vendor attributes for comparing and contrasting outsourcing companies to ensure you select one right for you.

Sizing a Vendor to Your Project

When optimizing the capabilities of an IT provider to your project, size matters. If your organization can fund nine-figure deals, then the number of companies to field such a deal is not large, whereas for smaller projects you have a lot more choices.

The key is to find a provider of a size that will consider your deal to be a big deal. This significantly increases the odds that your project receives the attention it deserves by having their most talented staff assigned to it. Additionally, right-sizing usually provides meaningful accommodation in contract terms and professional treatment from the executive staff.

The risk of choosing an outsourcing company that is too small, however, is that they may not have a sufficient level of technical capabilities, skilled staff, certifications or experience to deliver what you hope to accomplish.

Local Presence with Global Delivery

If your company is based in North America, then choose an outsourcing vendor whose headquarters there. They will better understand your industry, business model, goals and processes since you are working within a similar cultural context.

Your company also benefits from local contractual protections should your project hit a serious speed bump. It also provides the distinct possibility that they can provide onsite staff at your site, which improves communication and timely escalation of critical issues.

However, vendors that also provide delivery from offshore will save you money. Furthermore, vendors with a global presence could directly interact with your own global sites and offer the possibility to add shifts in other time zones that work collaboratively with your local staff, which provides you with 24-hours a day of development.

Consistency in Quality and Delivery

Until recently, the majority of IT outsourcing firms sold themselves mainly on cost and based contracts on hourly rates. These days, more companies compete on their ability to produce results. Those results should include both timely delivery and measurably high-quality products or services.

During your due diligence, evaluate the vendor’s past work and pursue references to gauge how well the vendor has delivered on their promises. Have a detailed discussion with their senior staff about how their corporate culture reinforces the importance of on-time delivery and high quality throughout the ranks.

When you are convinced they will deliver what they say they will, it is still prudent to start the relationship with one or more smaller projects of a few months in duration to validate their work and timeliness for yourself.

Communication Capabilities

Well-planned, thorough and frequent communication is critical when using an IT outsourcing vendor. This goes double if the company you select has offshore resources, since both time and language may present communication barriers.

How much, when and how you each of you communicates with the other should be driven by the client. Both sides must identify primary contacts for specific areas. These people must have seconds in cases where the primary is unavailable. Daily meetings with program and development managers are not unreasonable as are weekly meetings with BDMs or department managers. To gauge frequency, ask yourself how much time you can afford to lose should a process go astray.

Vendors uncomfortable with your communication plan should raise a red flag with you, since this is such an essential element in your business relationship.

Their Range of Skill Sets

Except for the largest IT organizations, most companies do not have all the personnel with all the right skill sets for every project. When evaluating vendors’ technical and process capabilities, strike a balance between broad and deep skills that align with your business and project needs.

If you hope for the vendor to work on more than one type of project or you wish to establish a long-term relationship, then one with a broader range of skills may work out better in the long run. A possible drawback is that a project comes along that is a mismatch for the vendor’s skills and quality suffers.

Many enterprises today recognize that one size does not fit all, especially when working with small to mid-size IT outsourcing companies, so they choose to multi-source these services. This can complicate internal management of vendors, but often the point solutions that smaller vendors provide are of higher quality, with faster delivery and at the same or lesser cost.


Proper selection of an IT outsourcing vendor will significantly augment your company’s strategy and operations. Lack of due diligence, however, often leads to negative consequences plus lost time and money.

Use the selection criteria above along with a robust process comparing business requirements against each company’s pros and cons. This will lead to asking the right questions and building a seamless working relationship with a talented development provider.

The Optimus Information model is designed to allocate the right mix of local and offshore resources in order to optimize expertise, speed and cost. We provide the ability for development teams to quickly add specialty skills to a development team without incurring long-term costs.Our successful track record speaks for it’s self, and we love to share past work we’ve done. Our global team is made up of a diverse range of experienced professionals, allowing us to work on complex solutions requiring a wide variety expertise. The result for our customers is the capability to far better manage resource capacities and outcomes.

Contact us if for your next IT project. We’re always happy to help.

Contact us now

Top Ten Software Development Outsourcing Trends for 2016

Originally, the primary motivation to outsource software development was to achieve lower labor costs, but continuing and emerging business and technology trends in 2016 are leading to new client requirements on outsourcers. When choosing an outsourcing partner, more and more businesses are looking for closer alignment to their business goals, flexibility demands and quality requirements.

Thus, clients are evaluating outsourcing companies via increasingly sophisticated criteria. The smartest software providers are reciprocating by developing new service models while taking advantage of many of the same technologies driving these current trends.

1) Moving from Hours to Results

In order to ensure that enterprises are getting what they need for their money, most are now seeking out providers who operate on a results-driven model versus rates based on time. Furthermore, clients are demanding that payment schedules be based on satisfactory achievement of those results versus upfront fees or retainers.

2) Greater Flexibility

Clients are looking for providers who provide on-demand services without locking them into long-term contracts or volume commitments. This enables client companies to respond more efficiently to rapidly changing market demands. In response, development providers who are moving operations to cloud resources are the ones most likely to adapt to the increased demand for flexibility.

3) Utilization of DevOps Practices Continues Apace

DevOps continues to attract adherents as it goes mainstream in up to 25 percent of companies this year, according to Gartner. Most of the IT departments in these organizations are transitioning to a service center model. Service providers who already operate in this manner will more easily blend into these organizations’ processes and decision-making apparatus.

4) Security Risk Perception Increases

A key concern within any outsourcing strategy is security. With the growing presence of the Internet of Things and the potential for an exponentially larger attack surface, software development outsourcing companies must ensure that their own security vulnerabilities are addressed in a manner that will win the confidence of client decision makers. Demonstrating solid track records and establish policies is of high importance when selecting a vendor.

5) Managing Infrastructure as Code

Amazon’s AWS has enabled the application of software development change management systems to development and deployment infrastructure. AWS is dedicated to making this paradigm increasingly easier with new APIs and services. Outsourcers who adopt this practice are reaping large benefits in their software support, testing and deployment efficiency by synching servers, storage and networking infrastructure to precise versions of the source code.

6) Multi-Sourcing Technologies Impacts Integration

Client companies are utilizing a more complex mix of software products and services this year. This multi-sourcing of technologies presents in-house management challenges, and a rise of new vendor management offices. The challenge for software providers is meeting new performance and integration standards from VMOs. Compliance failure may result in the outsourcer being dropped in the interests of streamlining operations.

7) Business Process Outsourcing Being Replaced by Robotic Process Automation

The software outsourcing industry in 2016 will continue to feel the influence of the rise of RPA. In fact, one of RPA’s touted benefits is the reduction of outsourcing, especially via cloud-based RPA services. Those outsourcers who can adapt by offering relevant automated services in the most responsive, scalable and efficient manner are the ones who can survive and profit from this trend.

8) Outsourcing Selection is Speeding Up

Along with the adoption of agile methodologies within software development, business decisions are also being made with more agility and higher velocity. Outsourcers will increasingly recognize this trend as more clients endeavor to close smaller deals faster in order to stay ahead of their competition.

9) Adept Companies Are Being More Selective with What They Outsource

Many organizations who originally turned to outsourcing to compensate for a lack of internal expertise and resources have grown more sophisticated over time. They are progressively learning to be more selective regarding what to do in-house versus handing off to an outsourcing provider. Organizations are looking deeper into what their core competencies are and what they can outsource to make themselves more efficient in-house. Their motivations are usually the desires for greater flexibility, responsiveness or cost reductions, all of which software providers need to be sensitive to in contract negotiations.

10) Outsourcing Company Accommodation Increasing

It is no longer the case that companies seek out only the lowest cost provider. Sophisticated outsourcing companies will respond tactically and strategically to all the trends discussed here to grow or to survive. This trend can be seen in the greater tendency for outsourcers to adapt and adjust terms or offer new services in an effort to deliver the best product and service.


The outsourcing industry is more fluid than ever this year with clients focusing less on price per se and more on results, quality, integration, security and agility from software development providers. As you adapt to your own fast-moving markets and the rise of paradigm-shaking technologies such as IoT and on-demand infrastructure, so do we. Optimus stays two steps ahead in order to support your business in all your software and IT requirements.

At Optimus, we consistently stay on top of these trends while leveraging the forces driving them to bring you the solutions you need. Contact us to help with your next development, testing, cloud, BI or mobile project.


Using Context Driven Testing

The premise behind context-driven testing is that software should be viewed first and foremost as providing solutions to real problems. In order to effectively test software, the problems it solves, the end-user needs and usage contexts must be taken into consideration when creating a test plan and applying tests. It recognizes that test processes must adapt to the testing environment itself, which includes the methodologies, skills and resources available within a particular test organization.

Testing if the Problem Is Solved

Functional testing ensures that each feature works as stated in the requirements. However, that only partially addresses how successfully the features taken together provide a complete answer to customer needs. Context-driven testing also takes those needs into account when designing test cases.

For example, say a doctor’s office acquires an application that assists the doctor in making a correct diagnosis. It may have a high accuracy rate and every button and widget in the application works perfectly. If it offers a diagnosis in a non-transparent way, however, the doctor may lose confidence in its diagnostic abilities, which leads to disuse. If the app fails to interface with other office programs, its usefulness is also diminished.

Furthermore, pre-release testing must be extremely rigorous in order to avoid the possibility of a lawsuit for an incorrect diagnosis. These are usage, environmental and perception contexts that must be considered thoroughly when designing the test plan.

Context-Driven Testing Principles

Beyond the factors considered above with regard to software use and its operating environment, “true believers” of the context-driven school of thought consider other, more subjective factors as essential characteristics of context-driven testing. This is evident in the commonly accepted principles of context-driven testing paraphrased here:

  • The end product must solve the intended problem in the intended context. If it does not, then it does not work.
  • The value of any “best” practice depends on the context in which it is applied. The phrase “best practice” should never be considered an absolute statement.
  • he application of judgment and skill applied in a cooperative manner is essential to product testing success.
  • The people cooperating in the testing activity are the most important aspect of a project’s context. This especially applies to the project’s stakeholders.
  • Projects naturally evolve over time in unpredictable ways and thus, flexibility is critical.
  • Software testing, when performed correctly, is a challenging intellectual process.

Several of these principles do not apply directly to the context of software in the field, but to the context of the testing environment including the methodologies, standards, so-called best practices, abilities of the testers, test tool capabilities and stakeholder expectations with regard to what, when and how to test the software under development.

The Practices of Context-Driven Testing

To follow context-driven testing precepts, apply a holistic approach that wraps up testing constraints and goals into processes and practices most appropriate for the problem being solved by the software within its testing context.

Flexibility is a key attribute in context-driven testing, which leads many to believe that agile methodologies and context-driven testing significantly overlap in how they operate and get results, which is true to some extent.

Regardless of the methodologies used by your organization, however, context-driven approaches hold value. In practice, they lead to several useful practices that enhance any testing process:

  • Sharing information and asking lots of questions significantly improves consensus on what precisely is the context of the project from various viewpoints. It increases learning and transparency as well.
  • The test plan should be reviewed by as many stakeholders as possible supplemented by experts in the application or testing domain who are not directly tied to the project. Obviously, this should be done as early as possible in order to minimize the impact of changes.
  • The test plan should be considered adjustable as the project moves ahead because knowledge of the entire project/product context is unlikely to be known a priori regardless of how many questions were asked.
  • Carefully consider the appropriateness of any test tool, testing practice or paradigm, especially if it is being used out of convenience versus applicability to the test problems at hand.
  • Defer the decision of when testing is complete to the project stakeholders. This increases their attention to the project and allows the testers to concentrate on their job without making judgments about product readiness.


In the end, context-driven testing boils down to doing the best job possible for product stakeholders, especially end-users, with the resources you have available. By definition, it is an approach that highly values problem solving, collaboration, intelligent testing and the most efficient use of testers’ skills.

Test Automation in Agile

Although both agile development and automated testing have more or less independently come into more widespread use with each passing year, it seems they were made for each other from the start. The advantages of both are derived from the same desire to increase software production efficiency overall.

Both work best when there is a high degree of collaboration between development and testing although they offer significant benefits to all project stakeholders. You often see a high usage of each approach wherever organizations are growing their continuous integration and continuous delivery capabilities.

Test Automation in Agile versus Traditional Waterfall

There certainly is no reason that teams employing waterfall methods cannot take advantage of test automation too. Typically, however, automation tends to be of lower priority in those scenarios due to the ingrained “throw it over the wall” mentality. Such a back-end loaded scenario is unfortunate because automation, especially test-driven coding, can be used quite effectively during development phases.

In agile, however, the questionable “luxury” of developers shedding testing responsibility is simply anathema to a process of two week scrums in which planning, designing, coding, testing and deployment all happen nearly at once. Without thorough development and testing automation, the productivity and quality of agile cycles would consistently fail to meet project goals.

When Automating Tests, Choose Wisely

Despite the brevity of agile development cycles, there can exist a tendency to over-automate testing. This is where careful planning and in-cycle flexibility are both necessary. Tests will naturally be categorized as to their type, whether that be functional, white box, black box, performance and so on. Within categories, however, individual test tasks need to be sorted by how likely they are to be reused repeatedly in order to optimize automation.

Any test that will be repeated more than twice, especially if it appears it will carry over to the next scrum, is a prime candidate for automation. That pretty much eliminates automation of exploratory testing and perhaps some boundary testing if the boundaries are still in flux and are not easily parameterized. On the other hand, even one-off tests might be amenable to automation in the sense that they are ongoing placeholders within an automation framework test suite for future software revisions.

Avoid Automating for Automation’s Sake

With more sophisticated test automation tools and frameworks, it is uncannily easy to become lost in the forest without seeing the trees. This is especially true of homegrown test frameworks, where the framework development effort might rival that of the applications it is meant to test. It pays at all times to bear in mind that meaningful, reusable tests must be the top priority.

An additional potential trap is not paying enough attention to the ebb and flow of value within your test suites. They naturally become cluttered with marginally valuable test scripts that may have lost relevance and eat up undue cycles of execution time. That is why it is important to set clear criteria for what does and does not get automated plus regular reviews of the test code base.

Staying One Step Ahead

Due to the rapid pace of agile development, anything to gain a leg up will pay dividends. Crisp, detailed planning is one key to getting ahead of the game, but in addition you should implement testing and its automation as early in the development cycle as possible.

The use of pair programming with one developer and one tester simultaneously working on the same code is an innovative practice in this regard. It is especially effective when combined with test-driven development in which the tester first writes the unit tests based on the specification and the developer follows with application code that turns each test “green.”

Actually, the underlying concept of first creating acceptance criteria as test code and then programming the software to make that test succeed can be applied at most points throughout the development cycle. These “live” requirements should be captured in an automated test environment that can be accessed and run by all stakeholders in the project to further increase collaboration.


Test automation is a key tool in agile development project success. As development organizations naturally evolve their entire production team from agile development to continuous integration and continuous deployment, the use of build and test automation becomes ever more critical.

Test automation in an agile development environment, however, has to be finely tuned to a greater degree than when employed in waterfall environments. Precise planning, a shift-left of testing responsibilities, keen maintenance policies and a sharp weather eye for when and when not to use automation are the most important elements for extracting the highest value from test automation.

5 Ways to Improve Mobile App Testing Quality and Efficiency

Mobile device application development faces significant challenges, which pass through to their testing. Though the obstacles appear daunting, there are ways to mitigate such complications, improve your team’s testing effectiveness and also raise app quality.

Mobile App Challenges

  • Mobile device fragmentation is rising. There are a myriad of hardware platforms, OS versions and network connection options across devices. Trying to maximize market coverage for a single app requires spanning this ever-increasing matrix of combinations with a concomitant increase in testing complexity.
  • Testing budgets are not expanding, which means doing more with less, which in turn leads to deciding between in-house or outsourced testing. Outsourcing infrastructure to the cloud often reduces capital and maintenance expenditures, but outsourcing test personnel and processes is a riskier proposition that is usually hard to unwind later. Outsourcing pressure is also increasing because new expertise and sophistication is required by leading edge test frameworks and tools.
  • Adding insult to injury, mobile app development requires ever faster development/release cycles. That is due to heightened competition and to end-users’ progressively shorter attention spans. Users more and more expect defects and improvements to be available in near-real time. That means testing has to constantly play catch-up so as not to become the bottleneck in the next upgrade.

Dealing Effectively with Modern Mobile App Testing

Before deciding you might be in the wrong line or work in the face of these looming challenges, let us consider five approaches that will improve your organization’s abilities to operate more efficiently while improving the quality of released apps.

1. Embrace Agile Development

This step is global to the organization. Fundamentally, it means involving all the stakeholders in the project from business managers to architects to production personnel. Depending on the delta between what you do today and how far you want to take agile development determines the size of the steps you take, but every step gets you closer to a more successful testing organization. Agile methodologies are proven to improve the ability to adapt to change, improve customer and stakeholder engagement, create more useful documentation and higher quality deliverables.

2. Value Your Automation Properly

Are you accurately measuring the value you derive from test automation? Often, more automation simply bloats maintenance tasks and increase test run times in the absence of ongoing evaluation of the tests contained within a framework. For instance, effort is commonly wasted automating tests that are better done manually, such as exploratory testing. The correct approach, customizable to your particular environment, is to measure your tests along a value spectrum. Tests such as build smoke tests are relatively easy to automate and provide high value. On the other end of the spectrum might be compatibility tests, which are necessary but probably should be lower priority.

3. Virtualize Your Testing Resources

Virtualized services and virtualized platforms are both relatively inexpensive in terms of infrastructure, setup and ongoing cost. They can be used on both the client and server sides of mobile applications. You will use less costly hardware and gain the ability to scale almost linearly, which is especially valuable for performance, load and stress testing.

4. Improve Your Ability to Test on Both Emulated and Real Devices

Testing a multi-platform mobile app on real devices is probably a non-starter for in-house deployment depending on the complexity of the instance matrix combining devices, platform capabilities, OS versions and network connectivity options. On the other hand, you cannot achieve a sufficient level of quality confidence with only emulators, which should by the way be deployed as early as possible in the development cycle.

The increasingly obvious answer to this situation is to employ cloud testing services that provide real-time access to a wide collection of new, sometimes unreleased, mobile devices and network operators along with built-in test and collaboration frameworks.

5. Gain Deeper Insights into the End User Experience

Simply reviewing comments and ratings in app stores is not enough anymore to evaluate how apps are performing in the field. Consider compiling in one of today’s sophisticated crash/analytics SDKs, which have vanishingly small run-time overhead. These provide real-time insights into how users are interacting with your app plus detailed reports on crashes that pinpoint problems immediately, especially if used in combination with a hotfix insertion tool.


Testing mobile apps gets harder every day, although new tools and techniques are always coming along to mitigate the difficulties. Seeing ways to improve development/test turnaround times, to optimize test automation capabilities, employing test resource virtualization and cloud-based device testing services are effective methods for maintaining the pace. Above all, keep your app’s user experiences squarely in your sights and utilize new tools for evaluating their behaviors and problems.