Introducing Katalon Analytics for Test Automation Analysis & Quality Visualization

Test execution reports are not only one of the most important feature for any automation framework but a valuable deliverable to track dynamic changes over the project timeline. To add value to the stakeholders, automation teams usually rely on third-party libraries to generate visual reports & dashboard which are time-consuming and unreliable sometimes.

Unlike other automation tools, Katalon Studio provides a rich UI for viewing test execution reports as a trusted, built-in feature without any configuration of third parties. The native reports provide the “snapshot” information of each test execution but limited at analyzing data to help make smart decisions about the quality of the application under test.

Offering a solution for this problem, Katalon Studio is going far beyond the basic by introducing Katalon Analytics that brings users to the next level of test reporting with analytics and visualization on aggregated execution historical data.

What is Katalon Analytics?

Katalon Analytics is an intelligent analytics platform with visual and filtering tools to give you a better understanding of your test executions of multiple projects in real time. The Analytics translates test execution results into actionable data which empowers you with the knowledge to pivot your automation strategy, focus on the high-risk areas of your product, detect flakiness tests and to have a better control on the quality.

A product owner could utilize the test results to the fullest only when it is structured and categorized such that the information facilitates decision making and improve the quality of the product. For the purpose of effective analysis, we need to have all the test execution results in one central repository to form a single source of truth for the test. With Katalon Studio v5.0, the test results can be uploaded automatically to Katalon Analytics server to form a seamless solution to provide the benefits of having the full history at your fingertip to be analyzed and filtered in any way you need it.


How Katalon Analytics helps your Test Automation Process?

Flaky tests are noteworthy, sometimes your tests fail unexpectedly for various reasons which could be data synchronization, improper test data management or having the dependency on other tests. As your test suites and functionalities of application grow, flaky tests become harder to debug. There are cases that tests that run perfectly fine in series might get flaky when executed concurrently. Automation test is a vital component to detect new errors introduced as a result of new features development but if tests failure rates are high, time and effort spent in the analysis and validation process lead to unacceptable time loss which deceives the purpose of automated regression tests. Katalon Analytics helps you solve this problem by providing flaky analysis on your test executions with visualization to easily identify this particular problem.

Katalon Analytics also helps you to optimize your code and reduce the regression rate. Execution time mainly depends on how you have designed & configured your tests, Katalon Analytics helps you to identify and benchmark the slowest performing functions by providing an analysis to optimize your test executions.

Benefits of Katalon Analytics:

  • Katalon Analytics is free to use with intuitive charts & reports to interactively navigate through vast amounts of log data
  • Eliminates the need for manually generating reports, reducing the possibility of data
  • Using Katalon Analytics pre-built aggregations and filters, you can easily run a variety of analytics
  • Updating and Collaborating test results in Real Time
  • Easy Distribution, Dashboards can easily share dashboards and reports with anyone in the team
  • Calculated execution time of each test cases
  • Time Series Analytics, Total execution history of the test case and many more features
  • Log Analytics can easily gain valuable insights about the tests executed provided with screenshots and videos


How to start with Katalon Analytics

No code, No installations, it is quite simple to synchronize your test results from Katalon Studio to Analytics server. Setting up your project settings with Katalon Studio v5.0 and you and your team members can view all of the project test executions by access Katalon Analytics using your registered Katalon account.

Whenever you execute your test suites, Katalon Studio automatically uploads the test results to the server along with logs, screenshots and execution videos. Katalon allows users to create multiple projects inside the Analytics dashboard and also from Katalon Studio. It also allows users to share the dashboard with the team members just by sending an invitation. Dashboard not only provides us charts but also gives detailed information of the entire test case history such as a number of times executed, the performance of the test case, console logs, status of the tests. Using the dashboard one can easily analyze the results from test suite level to test case level.

Katalon allows users to upload their test results to the dashboard in several ways. If the user does not want to sync the results automatically, then they can upload the results from the Reports section. By clicking ‘Upload‘ button under Katalon Analytics label, results will be uploaded to Katalon Analytics dashboard. The other way to upload the results is to execute a simple command by providing the project ID mapped in the dashboard. For more information, refer to the documentation.

You can give a try by downloading the latest version on Katalon v5.0.  Katalon Analytics in current in Beta stage and the team is working on additional quality intelligent reports that will be available in future releases. Please submit your feedback at the community forum to make the tool more helpful.

Advertisements

What makes Katalon Studio more powerful than any other Selenium based Frameworks?

As a test automation engineer, we should always consider a few cornerstones before employing a tool to automate our products. As you know the primary benefits of test automation are reusability, scalability, improving test coverage, time and effort saving. Quite many of us say, using Selenium integrated with other third party tools would achieve all the above cornerstones. But still, we have a few drawbacks. Let’s see how Katalon Studio turns to a magic wand solving test automation difficulties.

Challenge to Selenium/Appium based Frameworks

Let’s say we need to automate a modern application which has web UI, a mobile app and web services. Applying test automation for this application takes much time and effort for coding to automate test scenarios, publishing content from the web UI and verifying the same content in the database & from web services, the same process is required for the mobile app on Android and iOS platforms. Making this more challenging, open-source automation solutions like Selenium require various third party tools needed to be integrated and configured. Continue reading

Katalon Studio for Test Automation – Simple & Powerful

Recently I came to know a Tester/Developer friendly Test Automation tool Katalon Studio. I am really impressed with the provided features of Katalon Studio 4.5. I have seen all features which are required for Test Automation. I can say this is a comprehensive Test Automation Tool, a cluster of Selenium, Appium, QTP, SOAPUI for both Web & Mobile Apps testing.

Continue reading