Data-Driven Test-Automation Frameworks

Data-Driven Test-Automation Frameworks: How Ubisoft is Generating and Executing Automated Tests from Game Data.


Using a data-driven test approach to enable a test-automation framework provides benefits beyond the simple execution of automated tests. In this presentation, we will look at how these techniques can be used to increase test coverage while also providing valuable data that can be used to better understand application stability and readiness.

Guest Speaker:

Sean Wilson, Worldwide QA/QC Development Director for Ubisoft.
Certificates: PMP, PSM I, PSO I, PSK I

Sean started as a manual tester on a financial treasury application somewhen in the last millennium. His career took him on a winding journey through automated testing, development, project management, and agile evangelism before he abandoned mainstream software and went where he could play games and get paid for it. In his current role as the WorldWide QA/QC Development Director for Ubisoft, Sean is focused on evolving the approach to Quality Assurance through a better application of technology.

Logging and Reporting in Selenium Webdrive Projects

Come learn about Selenium Automation from seasoned professional, Alex Siminiuc. Alex has over 20 years of experience in testing, and has worked with companies such as Best Buy, Trader Corp., YVR, BCAA, and Boeing Canada. Alex will discuss logging and reporting in Selenium webdriver projects.

5:30-6:00 pm – Registration & Refreshments
6:00-7:00 pm – Presentation and Q&A
7:00-7:30 pm – Networking

The Importance of Efficient Automation in the Agile World

We’re excited to have Kevin Butto, Automation Engineer at Plenty of Fish, speaking about “The Importance of Efficient Automation in the Agile World.”

Kevin is an ICAgile Certified Professional in Agile Test Automation and has a background in Computational Intelligence and Design (Cognitive Systems).

The Future of Automation

Automation is the new buzzword these days as organizations are moving to automate more and more.

Our speaker will be Jeff LaPorte, the CTO of Kobalt. He will speak to us about where he sees the Future of Automation.

More details to come.

API Testing: Do It Right and Automate

Software development in 2018 looks nothing like it did a decade ago. Developers have been driven to find faster and more efficient ways to produce a finished application. Customers demand better products, and market pressures mean get your apps out now or risk annihilation from competitors.

This means abandoning the waterfall method of software development where each part of a program was completed before the next was started, often leaving error identification (and subsequent delays) until the end of the cycle. Instead, developers are now embracing Agile development with its highly integrated production methodology, often releasing product in two-week sprints.

APIs Can Get Your App Market-Ready Faster

We live in a fast world. Enterprises that are big users of technology need to keep up with the pace of change demanded by their customers. This can put a high degree of stress on internal IT teams and lead to the inevitable coding errors and resulting delays in new product release.

An advantage is the ubiquitous use of APIs or Application Programming Interfaces. APIs allow nearly limitless possibilities for how applications can interact with each other.

However powerful the use of APIs can be, they also need to be put through a rigorous testing process. Unfortunately, this often places added strain on a company’s IT team that may not be familiar with the most effective testing methodologies.

“Often a client doesn’t know what KPI they should focus on or they don’t define it at all. Then they’d have no way of knowing what the metrics are for the data they’re gathering. But if you do it right, you get the right information back from the metrics to make improvements.”

Ashish Pandey, Optimus Information’s Technical Lead

API Testing Begins with KPIs

API testing can be the most challenging part of software and QA testing because APIs are complicated creatures, using protocols and standards not often seen in other forms of testing.
It’s critical to test all the components of the API – not just its UI or its functionality. Testing performance and security are just as critical.

To test properly, says Ashish Pandey, Optimus Information’s Technical Lead at the company’s location in India, it’s important to begin with Key Performance Indicators (KPIs).

Don’t Forget Security and Performance

In addition to defining your KPIs, you also need to focus on more than UI. Some customers, Pandey continues, concentrate on just the UI portion of the application and ignore testing the other components. Security and performance are two areas often overlooked and, if something breaks, it’s probably because key areas within the API were ignored in testing.

“When we test APIs for our clients”, says Pandey, “we generate a lot of data which gives us and our clients a clear summary of what might need to be improved in the future.” This can only happen when thorough testing is done and nothing is left to chance.

While Ashish admits that thorough testing of the UX is extremely important for the customer,  it’s the KPIs that measure the data that then determine the performance of the application. If there are problem areas turning up, they can be remedied in a matter of seconds.

Fail Forward Faster – Automation is Key to Proper API Testing

Proper testing of an API is accomplished by running test cases which are designed to uncover failures. It can be extensive and time-consuming – the opposite of what agile development tries to accomplish – so Optimus specializes in automating as much of a customer’s testing as possible.

It begins with the test cases, themselves, Pandey explains. “We take the customer’s test cases and analyze them to determine if they are ‘automatable’ or not. This makes it possible for us to suggest the correct technology stack with which the automated test could be performed.” In the future, the customer simply runs automated scripts to test different iterations of their applications, saving a great deal of time and money otherwise spent on manually writing the test scripts themselves.

iStock-870784968 API Testing: Do It Right and Automate

The Optimus Test Harness – Why No Company Should Test Without It

A further advantage for Optimus customers is the use of an open source test harness. Ashish Pandey is one of the creators of the Optimus test harness which uses open source components and is configured to test cloud-based applications.

Optimus estimates that 85% of its customers are technology firms that have cloud-based apps often undergoing testing as new iterations are created. Optimus has designed the harness to perform automated testing at different levels. “If we’re doing test automation at the UI level”, explains Pandey, “we have the ability to create automated test scripts for UI. We also have the capability to test at the full API level as well. In fact, our test harness is efficient enough that customers can perform a wide variety of testing on things such as execution of SQL queries to their database. We have built in to our harness APIs like SoapUI, WebSockets and others.”

Test Feedback in Hours (Sometimes While You Sleep)

What Optimus strives to do is provide customers with the sort of speed and agility that can be achieved through automation. “Many of our customers are into Agile development, so what they want is quick delivery of their app with feedback in a few hours,” says Pandey. “Some of them are also evolving with DevOps practices and they want results fast.”

Automated testing, he points out, means that if a customer has three or four hundred test cases to run and each test takes four hours to perform, automation allows Optimus to test while the customers are sleeping. “The next morning, they have the results in their hands, rather than having to wait several days,” Pandey concludes.

Understand and Implement the Right Methodology for Automated Testing

Optimus has one aim with its clients: to ensure that they implement the right thinking and methodologies around testing. Doing so will improve the customers’ UX, decrease errors and get the app to market faster and on time.

We understand Agile development, DevOps and automated testing and how the combination leads to rapid deployment of new, error-free applications at greatly reduced costs. We also know that this translates into powerful ROI for our customers.

To learn more about using API test automation to make your software better, faster and more secure, download our new eBook now.

More resources:

Cloud-Based Test Automation Tools and Benefits

Acquiring and maintaining in-house testing platforms for even small-market software applications is expensive and time-consuming. If your software products include mobile applications that support thousands or millions of users, then the sheer scale of testing can overwhelm even well-stocked IT departments. Fortunately, with the availability of a multitude of cloud-based testing environments, there is a better way to meet your testing platform requirements that will cost less and improve efficiency.

Multi-Faceted Cost Savings

To start with, renting cloud-based testing environments means your company avoids huge capital equipment expenditures for servers, networking equipment, OS licenses and so on. Additionally, hardware and software maintenance costs plus the cost of personnel to manage such resources is borne by the cloud service. These are meaningful advantages for any company, especially for cash-strapped startups.

Furthermore, you avoid depreciation costs on test platforms regardless of whether they are in-use or not. That advantage, along with knowing that test platform backups, upgrades, provisioning and configurations are taken care of automatically, brings a strong measure of stability to ongoing cost projections.

On-Demand Testing Expedites Planning

Multiple surveys have indicated that, due to the periodic nature of testing, most companies that maintain in-house testing infrastructure achieve less than 50 percent utilization of that infrastructure. Most cloud-based testing services enable scaling test infrastructure up and down at will, so you only pay for what you actually use at any given time.

Instant scalability of hard testing resources provides an enormous benefit from a planning perspective. It eliminates the scenario where test platforms are acquired or re-commissioned in preparation for large-scale performance and load testing only to have them sit idle after development is delayed.

Additionally, many cloud-based mobile-app testing services provide virtual access to large arrays of released or soon-to-be released mobile devices and various OS versions. When it comes time for full-scale compatibility testing, few companies are equipped with the necessary resources to duplicate such a large and diverse testing environment at almost a moment’s notice.

Reducing False Defects Due to Inconsistent Configurations

A large percentage of software defects can be traced to incorrect test environment configuration. For example, provisioning a platform with a specific OS version plus service packs, databases and third-party middleware or applications on which your software depends is an error-prone process to say the least. Cloud-based testing services excel at providing consistent test platform preparation backed by service level agreements, so that you can avoid wasting time and resources tracking down phantom bugs.

Reducing In-House Test Applications

Besides the advantages inherent in renting versus buying your test platform infrastructure, the best cloud-based testing services also offer world-class testing frameworks with which you can create, run and manage your test suites. This avoids the costs of licensing in-house testing tools plus the costs to install and maintain in-house tools.

Increased Collaboration Improves Time-to-Market

Furthermore, most cloud-based tools make a point of being user-friendly and only require access via a browser. Thus, greater accessibility to the tools, test suites and reports is achieved across development, test and business units. Access can be granted outside the company as well to provide transparency to clients.

Most importantly, because testing resources and tools are virtualized, collaboration between developers and testers increases regardless of their physical locations and the time zones within which they perform their work. Testers can share results instantly via screen sharing with developers, which reduces communication failures and delays immeasurably.

As software products become more complex, more distributed and release cycles are increasingly shorter, maintenance of internal testing infrastructure is increasingly impractical in terms of cost, quality and time-to-market. Although virtualization of such resources via cloud-based testing services will never completely replace internal testing, an emphasis on its use brings immense benefits. The reduction of capital expenditures, ongoing cost reductions, test platform scalability, planning flexibility and enhanced collaboration are advantages too great to ignore.

A final key advantage to cloud-based testing services is that they can be sampled without incurring a large commitment by enterprises who desire hands-on experience of cloud-based testing relative to their particular situation. An experienced partner can save time and provide enlightening insights in this regard.


Nothing Found

Sorry, no posts matched your criteria