Trial and error: Why QA and test automation is crucial to egaming

Since the introduction of lean and agile operations, gambling operators have employed much more efficient software testing processes. But as the technology becomes more complex firms are battling between being thorough and automating their services

Techies are renowned for their immaculate attention to detail, but none more so than the gambling industry’s quality assurance (QA) and test automation engineers. They oversee approximately 1,000 API tests a day and work to ensure the hundreds of releases are carried out without any bugs or issues.

Automation is a widespread technical principle used to power a huge number of areas within egaming, from sportsbook trading to data collection and, it appears, quality assurance and software testing.

Gone are the days of carrying out hundreds of manual tests daily to secure the upkeep of the site and its frequent releases. But it seems some operators have only really jumped on the bandwagon in the last few months as mammoth tech corporations have begun to employ DevOps and agile processes and have ridden themselves of the arguably outdated siloed team structure.

Lean, mean testing machine

Through the use of its own Spotify modeled lean and agile operational structure, Leeds-headquartered operator Sky Betting & Gaming (SB&G) has spent 18-months streamlining its QA and test automation processes, and in that time has built an entire team of engineers to work on ensuring its gaming brands Sky Vegas, Sky Casino, Sky Poker and Sky Bingo progress and operate smoothly.

Head of QA for the gaming tribe Paul Lilley says the journey has not been without its complications and failures: “We have a massively service-orientated architecture and we’re very agile, so all the things [at each level of the testing process] change regularly and everything depends on everything else,” Lilley relays to EGR Technology.

“How do we pull a safety net around that and our key journeys and ensure that revenues are protected?” he asks. The operator frequently performs multiple functional changes to its UI, from altering the log-in process and other areas of the customer journey.

In order to ensure minimal downtime on the site and across its apps, speed of delivery is a crucial consideration. “The biggest value is to the business [through] that speed of delivery. Imagine a world where this didn’t exist and we had manual testers in squads running a thousand tests a day themselves.

Now we’ve got testers that focus on the analysis and do really in-depth explorative testing and the leg-work of just keeping an eye on the core journeys is now managed by this process,” says Lilley.

Taking EGR Technology through the seemingly complex journey of its software tests, head of data at SB&G Rik Barker reveals Lilley’s team has created a framework that can manage thousands of tests every day. “[They] are fast to run, cheap to maintain and give us confidence in everything we’re putting out the door,” Barker adds.

Similarly, mobile app and web development agency for the industry, Degree 53, has vastly improved and streamlined its own software testing processes in the last six months by speeding up delivery periods and adopting a more agile methodology.

Head of QA Rick Sedgwick says in nine months the firm has vastly expanded its capabilities across all areas of the business.

“What we are looking at doing is moving towards shorter delivery periods and having a more agile methodology. Within two week sprints we can get delivery of product quality features at the end of each of those sprints so we’re using automation a lot earlier in the lifecycle.”

Central framework

In many areas operators are quick to create more efficient testing environments via other means of streamlining their technology services. Sky Bet, for example, often builds product agnostic services that can be deployed across any product. Elsewhere Lilley’s team adopted a central framework built with Codeception.

“We have multiple products and they generally work in the same way and have the same problems from a UI automation point of view so it didn’t seem right to do this in every team,” explains Lilley.

Interestingly both firms use different coding languages with SB&G operating with PHP and using the open source Codeception framework as a base layer for its in-house offering. Lilley says the real work was in the development of the base layer portal into individual test artefacts for each of the firm’s gaming verticals.

“The Codeception framework uses page object models, so provides the ability to quickly and repeatedly access the things you’re testing.

“It has a concept of our Sky Bet account; if you’ve seen our products when you log-in you get an account side bar, that’s a shared thing managed by our core tribe,” he adds.

“It’s ideal for us that Playtech and OpenBet provide us with APIs and services that allow us to direct those features and functionalities out to each product,” Barker comments, revealing the operator’s close ties with its gaming and sportsbook platform providers.

Although it is widely known to develop and deploy much of its own frontend systems, Barker says the firm works closely with its providers on security management.

“Generally gaming and bet have core teams that work with us individually on the bespoke platform and doing the things we need to do,” Barker says.

Although speed is of the essence, attention to detail is more of a primary consideration for Degree 53’s Sedgwick who explains that although his team sets up nightly automation runs, engineers will scour the system come morning to ensure it has not missed any human errors.

“We like to employ exploratory session-based testing to supplement our automation, that’s where the tester would sit down and run the scripts overnight to find any issues. They are brought up and the manual tester will sit down and produce a mission to exercise that area of the site and go around it fully and use it as a user would do.”

Complex customer journey

It is pivotal to replicate a t ypical customer journey to ensure all potential bugs are found. Sedgwick explains: “We can go around and approach it from more of a user perspective. What happens if you close your laptop or get an incoming call on your device?

“How does your device react to that? We add another layer of complexity on top from the human point of view.”

Specific to the gambling industry is the complexity of the customer journey, as they often flit between bet and casino apps and use the system in unexpected ways.

Also to be expected are huge peaks in traffic, particularly during horseracing festivals and football matches.

But Sedgwick outlines a performance test process that unveils exactly what the present infrastructure can support, what its breaking point is, and at what point the engineers need to re-assess it to enable the system to cope with additional demands.

Reflecting on Sky Bet’s turbulent software testing history, Lilley says the operator formerly used test automation only to support third party game functions.

“We implemented UI automation which is typically very fragile and requires a lot of maintenance,” Lilley relays cautiously.

SB&G’s head of QA, Paul Lilley

“You can’t underestimate the amount of resource that this could suck up if you don’t do it properly. We looked at how we’d failed in the past and spun up a team to support this.”

The Codeception-based system broadly follows the test pyramid model with unit tests at the bottom, integration tests in the middle and UI at the top.

“We’ve retrofitted a good level of coverage across this middle level, but that was never going to be a short journey due to the number of services we maintain or consume, and we had to protect our products.”

Once we entered the mobile era a few years ago, software testing had its Eureka moment as the tech community scrambled to develop appropriate systems to support native mobile apps across varying browsers and operating systems.

Degree 53 uses Appium to test clients’ app-based products as it covers all mobile browsers including Android and iOS. Sedgwick says it enables the team to cover more devices in a shorter period of time. “It gives us a level of confidence that the applications are working across all supported browsers and operating systems.”

But how is a third-party supplier meant to cater to the individual testing needs of any one operator using a centralised testing model? According to Sedgwick, the web development firm adapts its system to cater for smaller clients.

“It depends on what their needs are. If it’s predominantly a desktop web platform they’re not really targeting mobile for, then we’ll use Selenium to test the website, layout and responsiveness. If it’s a native app only we’d focus on the Appium tool to deliver that as well,” he adds.

For a shorter-term project he believes the return of investment from test automation will be relatively low, and much longer projects, spanning over six months, are likely to offer a much greater ROI and retain a higher quality.

The world of testing and automation offers a spectrum of frameworks and additional tools easily integrated into open source platforms. For the egaming industry, operators and third party testers have learnt to adapt and build on their central frameworks to cater to the unique customer journey carried out by users, and to suit the very many multiple layers of software and products each employs.

But as we move into a new technology generation in a world of voice-assisted devices and the Internet of Things, we can expect the complex world of automation to gain even more layers, and for engineers to find more loopholes and quick fixes to help them cater to multiple platforms.