This article was written by Sara Martin.
Test automation is helpful for rapid learning, effective communication, and creating innovative products, as well as boosting collaboration and productivity in the workplace.
Our testing, test strategy and activities have evolved over recent years. Almost 4 years ago, all applications were tested through UI and no shortcuts were applied. It could even be said that the Login feature was the most thoroughly tested one. For all platforms. Different programming languages and frameworks were established. Cypress, TestCafe written in javascript, python and appium for mobile.
The QA team were in the driving seat of functional testing while Dev teams were in charge of unit and integration and some functional testing made use of another framework called Behave, and still in use. The API was primarily tested by QA using Postman. We now automate testing not only for the UI but also for the backend, utilizing the same programming language, TypeScript, across projects, although with different frameworks: Playwright, WebDriverIO, PactumJS, and Pact.
We still have one or two projects that utilize Cypress. I will refrain from going into detail on every single project but rather focus on what we changed and what we learnt. The bumps in the road are not enough to deter us from making strides in our work. And a long way still remains!
Four years ago, our automation QA Engineers developed functional scripts using javascript as the primary language along with frameworks such as Cypress and TestCafe. As Cypress had certain limitations at that point, TestCafe proved to be helpful. Two different frameworks for the same web application turn out to be double maintenance. And any product increase could be reflected in a growth in both automation projects, depending on the criticality of the feature. In addition, data used by tests were mostly static and this led into several problems:
Whenever someone made changes to the dataset, the automated test suite would be affected and fail.
Relaunching was often a Herculean task. If the test had failed at any point, leaving the necessary actions incomplete, a data-cleaning job would be required to ensure the data was returned to its previous state.
Another issue we faced was that tests often took a long time to execute, leading to complaints from all teams, including our own team.
To sum up, here are the main problems we had: several programming languages and frameworks; pipeline time execution and static dataset. How did we solve them? Let’s group them in two big points:
It took multiple steps to move toward what we have today: Removing retries. This was a controversial point. Finally, we agreed that retries only add time to the pipeline when a test fails, masking a flaky test. We’d rather have a failure in our pipeline and invest time to solve it, even though sometimes it is very hard to find out the cause of the flakiness. Google posted some meaningful and inspiring post about flakiness and how to solve it.
Googleblog from december 2020 and Googleblog from march 2021.
Using static data made us launch using a serial mode. Tests weren’t executed in isolated mode and sometimes were even dependent. And let's not forget, we were using the UI to set up the preconditions for the test. So, we decided to incorporate dynamic data in our solution, using API endpoints to generate the preconditions. We have a mixed data test strategy using both: static and dynamic data, having the benefits of both! Having this solution in place, we could even work in a volatile environment which have the minimum data needed to work with it.
After researching online, we discovered that Playwright met all our requirements, such as Safari support, for web applications. While migration could be time-consuming, we decided to use the strangler pattern to stay on the safe side. We leverage Playwright to incorporate page object pattern which made our tests more readable and maintainable.
It has been the same QA team writing the automated scripts. Four years ago, all automated tests were performed on two devices— one for Android and one for iOS — even though we were using a real device cloud provider. As you can imagine, we didn’t have — and still don’t have — a physical device farm in our offices. We had some, but we didn’t make use of them for automation purposes. Python and Appium were the frameworks for automation. And at that point, our apps were mostly native, so we had to test whole flows to ensure everything was working as expected.
Having said that and to sum up: we didn’t take into account the more used devices from our customers; using python instead of JS/TS; our scripts could be illegible and hard to grasp in some cases. Mobile automation testing is challenging by itself, but we needed to solve the issues I mentioned above.
Let's go through it point by point:
We created a coverage strategy which specifies 3 devices + operating systems of both platforms, the most commonly used from our customers. We review the statistics every 6 months and update the strategy accordingly.
Since we were writing scripts for the web using JS/TS as the programming language and had more team members with in-depth knowledge of these languages at the company, we decided to transition to this language and framework.
Cucumber was included at first trying to get the tests more readable for non-technical people, but time later we found out that only IT read the tests, so we got rid of this extra layer.
Pipelines suffer drastic changes as well. We split the execution — one job one test — giving us the chance to relaunch one single test if needed. And we even connect the pipeline with Nexus to upload the artifacts to our real device cloud provider in an automated fashion. Nowadays pipelines are directly connected, so we can get the artifacts directly without any other server in the middle.
Only native parts and transitions from native to webview are now covered in our automated solution. Webview flows are tested in our web automated testing approach changing viewports.
As many other companies, we have services that need to be tested. This kind of testing is something we recently implemented in our SDLC. At the moment of writing this article, we have 164 pacts developed. We have been facing some challenges since we started. The main points are:
1. We struggled to shift our mindset from functional testing to contract testing. As QA engineers, we had rarely worked at the backend level, and at that point, we saw this type of testing as more aligned with a developer's approach, given its similarities to integration testing.
2. Not all contract tests are integrated in the pipeline on the provider side, but it’s something we plan to do in the coming weeks
3. This kind of testing helps us to understand how a consumer and provider works. We have a different vision of our backends.
And that's all! We will continue improving our test scripts and strategy and most probably include new types of testing or, at least, new tools.
We have evolved since we started with test automation. We have introduced new ways of testing such as contract testing, new tools like playwright, webdriverio, and we have improved our pipelines. But a long way still remains.
Therefore, we value your input and opinions on this matter. Please let us know if there are any types of tests you believe should be added or removed or if you have any suggestions for improving our testing process. Your feedback is paramount in helping us create the best possible product for our customers.
Tech Lead
At PromoFarma by DocMorris, there are career opportunities at all levels.
We are looking for diverse talent for different teams.
