Abstract. In software development context, with agile methods, having continuous and fast feedback is crucial for project success. One way to ensure quick bug detection is through automated tests. Given the high value of automated testing, it is necessary to evaluate the various frameworks, currently available in the market, to determine which one suits better the project’s needs. In this work, it’s proposed a comparative analysis between three testing frameworks: Selenium, one of the most traditional automation tools; Cypress, a tool that has gained popularity in recent years; and Playwright, a newer tool that has attracted a lot of attention. To perform this analysis, automated test suites were developed for a web application, with the aim of comparing the generated scripts based on selected criteria. Thus, the pros and cons of working with each of these frameworks will be presented.
Software testing, among other objectives, seeks to detect existing failures and defects in order to reduce software quality risk levels [ISTQB-CTFL]. Myers' Rule of Ten states that the cost of correcting defects tends to increase the later the defect is detected. Defects found during production tend to cost much more than defects found in data models and other software project documents. Therefore, by treating testing as an organized process that is often parallel and integrated with the development process, maintenance costs will be reduced. [Bastos 2007]
Nowadays, many software options are being released constantly, so the cost of replacing software is no longer as high as it used to be. For this reason, the adoption of agile development methods is becoming increasingly common, to meet customer needs with constant improvements and quick fixes. As a result, the deadline for launching an application is becoming increasingly shorter and, consequently, the time available for testing is becoming shorter
Due to the shorter time required to develop and launch an application, automated testing becomes necessary to find defects and ensure quality during development cycles. Test automation is the use of software to control the execution of software tests through the application of strategies and tools, comparing expected results with actual results. Its objectives are to reduce human involvement in mmanual activities, time required and final cost. [Oliveira 2018]
There are currently many options for test automation frameworks, many of them open source, but even selecting an open source framework for adoption in a company is not free of cost. After all, one must take into account the learning curve of a framework, which will affect the time required to train an employee to learn how to use it. In addition, it is necessary to check whether the framework meets all the needs of use, because otherwise it will be necessary to modify it through customizations or installation of third-party plugins. These factors make choosing a framework a complex task, requiring a good analysis.
One of the most traditional and widely known frameworks is Selenium, a comprehensive project for a variety of tools and libraries that enable and support web browser automation [Selenium 2023a]. Although its first version was created in 2004, the framework is currently in version 4.0 and has great community support, as can be seen in Table 1 [Selenium 2024a].
| Framework | Supported Languages | Copies on GitHub | Stars on GitHub |
|---|---|---|---|
| Selenium | Java, Javascript, Python, Ruby, C# | 7.9K | 29K |
Table 1. Data about the Selenium Framework
Cypress is a new front-end testing tool built for modern web applications. It is worth noting that it addresses the main pain points faced by developers and QA engineers when testing modern applications [Cypress 2024a]. Despite having been created in 2017, Cypress currently has a high number of active users, as shown in Table 2. [Cypress 2024b]
| Framework | Supported Languages | Copies on GitHub | Stars on GitHub |
|---|---|---|---|
| Cypress | Javascript | 3.1K | 45.9K |
Table 2. Data about the Cypress Framework
Playwright was created specifically to meet the needs of end-to-end testing. Capable of supporting all modern rendering engines, including Chromium, WebKit, and Firefox [Playwright 2024a], it was conceived in 2020 by Microsoft and has been attracting a lot of attention from the community, as can be seen in Table 3 [Playwright 2024b].
| Framework | Supported Languages | Copies on GitHub | Stars on GitHub |
|---|---|---|---|
| Playwright | Java, Javascript, Python, C# | 3.2K | 60.1K |
Table 3. Data about the Playwright Framework
Although it is more recent, Playwright has already surpassed Selenium in number of downloads and despite having reduced the distance from Cypress in the last year, it is still far from the number of downloads of Cypress, a situation indicated in Figure 1.
Figure 1. Number of downloads of each framework in the last 5 years. [NPM Trends 2024]
Therefore, in view of the above, this work aims to identify the main functionalities and limitations of the selected frameworks, perform the environment configurations, develop the architecture of the automation project, in order to follow the good practices of software engineering, develop the same test scripts in each framewrok, execute and valdiate them to compare the results obtained later.