Introducing Katalon Analytics for Test Automation Analysis & Quality Visualization

Test execution reports are not only one of the most important feature of any automation framework but a valuable deliverable to track dynamic changes in the project timeline. To add value to the stakeholders, automation teams usually rely on third-party libraries to generate visual reports & dashboard which are time-consuming and unreliable sometimes.

Unlike other automation tools, Katalon Studio provides a rich UI for viewing test execution reports as a trusted, built-in feature without any configuration of third parties. The native reports provide the “snapshot” information of each test execution but limited at analyzing data to help make smart decisions about the quality of the application under test.

Offering a solution for this problem, Katalon Studio is going far beyond the basic by introducing Katalon Analytics that brings users to the next level of test reporting with analytics and visualization on aggregated execution historical data.

What is Katalon Analytics?

Katalon Analytics is an intelligent analytics platform with visual and filtering tools to give you a better understanding of your test executions of multiple projects in real time. The Analytics translates test execution results into actionable data which empowers you with the knowledge to pivot your automation strategy, focus on the high-risk areas of your product, detect flakiness tests and to have a better control on the quality.

A product owner could utilize the test results to the fullest only when it is structured and categorized such that the information facilitates decision making and improve the quality of the product. For the purpose of effective analysis, we need to have all the test execution results in one central repository to form a single source of truth for the test. With Katalon Studio v5.0, the test results can be uploaded automatically to Katalon Analytics server to form a seamless solution to provide the benefits of having the full history at your fingertip to be analyzed and filtered in any way you need it.

How Katalon Analytics helps your Test Automation Process?

Flaky tests are noteworthy, sometimes your tests fail unexpectedly for various reasons which could be data synchronization, improper test data management or having the dependency on other tests. As your test suites and functionalities of application grow, flaky tests become harder to debug. There are cases that tests that run perfectly fine in series might get flaky when executed concurrently. Automation test is a vital component to detect new errors introduced as a result of new features development but if tests failure rates are high, time and effort spent in the analysis and validation process lead to unacceptable time loss which deceives the purpose of automated regression tests. Katalon Analytics helps you solve this problem by providing flaky analysis on your test executions with visualization to easily identify this particular problem.

Katalon Analytics also helps you to optimize your code and reduce the regression rate. Execution time mainly depends on how you have designed & configured your tests, Katalon Analytics helps you to identify and benchmark the slowest performing functions by providing an analysis to optimize your test executions.

Benefits of Katalon Analytics:

  • Katalon Analytics is free to use with intuitive charts & reports to interactively navigate through vast amounts of log data
  • Eliminates the need for manually generating reports, reducing the possibility of data
  • Using Katalon Analytics pre-built aggregations and filters, you can easily run a variety of analytics
  • Updating and Collaborating test results in Real Time
  • Easy Distribution, Dashboards can easily share dashboards and reports with anyone in the team
  • The calculated execution time of each test cases
  • Time Series Analytics, the Total execution history of the test case and many more features
  • Log Analytics can easily gain valuable insights about the tests executed provided with screenshots and videos

How to start with Katalon Analytics

No code, No installations, it is quite simple to synchronize your test results from Katalon Studio to Analytics server. Setting up your project settings with Katalon Studio v5.0 and you and your team members can view all of the project test executions by access Katalon Analytics using your registered Katalon account.

Whenever you execute your test suites, Katalon Studio automatically uploads the test results to the server along with logs, screenshots and execution videos. Katalon allows users to create multiple projects inside the Analytics dashboard and also from Katalon Studio. It also allows users to share the dashboard with the team members just by sending an invitation. Dashboard not only provides us charts but also gives detailed information of the entire test case history such as a number of times executed, the performance of the test case, console logs, status of the tests. Using the dashboard one can easily analyze the results from test suite level to test case level.

Katalon allows users to upload their test results to the dashboard in several ways. If the user does not want to sync the results automatically, then they can upload the results from the Reports section. By clicking ‘Upload‘ button under Katalon Analytics label, results will be uploaded to Katalon Analytics dashboard. The other way to upload the results is to execute a simple command by providing the project ID mapped in the dashboard. For more information, refer to the documentation.

You can give a try by downloading the latest version of Katalon v5.0.  Katalon Analytics in current in Beta stage and the team is working on additional quality intelligent reports that will be available in future releases. Please submit your feedback at the community forum to make the tool more helpful.


What makes Katalon Studio more powerful than any other Selenium based Frameworks?

As a test automation engineer, we should always consider a few cornerstones before employing a tool to automate our products. As you know the primary benefits of test automation are reusability, scalability, improving test coverage, time and effort saving. Quite many of us say, using Selenium integrated with other third party tools would achieve all the above cornerstones. But still, we have a few drawbacks. Let’s see how Katalon Studio turns to a magic wand solving test automation difficulties.

Challenge to Selenium/Appium based Frameworks

Let’s say we need to automate a modern application which has web UI, a mobile app and web services. Applying test automation for this application takes much time and effort for coding to automate test scenarios, publishing content from the web UI and verifying the same content in the database & from web services, the same process is required for the mobile app on Android and iOS platforms. Making this more challenging, open-source automation solutions like Selenium require various third party tools needed to be integrated and configured. Continue reading

Katalon Studio for Test Automation – Simple & Powerful

Recently I came to know a Tester/Developer friendly Test Automation tool Katalon Studio. I am really impressed with the provided features of Katalon Studio 4.5. I have seen all features which are required for Test Automation. I can say this is a comprehensive Test Automation Tool, a cluster of Selenium, Appium, QTP, SOAPUI for both Web & Mobile Apps testing.

Continue reading

JavaScript Unit Testing with TDD, Jasmine and Karma

JavaScript Unit Testing with TDD, Jasmine and Karma

In this article, we’ll introduce you to a tool combination which will make JavaScript unit testing so much easier. Jasmine and Karma combined will save you hours of testing.

First, let’s understand what exactly are TDD, Jasmine and Karma.

Test driven development (TDD) is one of the main, agile development techniques. The genius of TDD lies in writing your test’s code first, so it can guide your further implementations. This method provides you with an increased quality of code, improved bug detection and overall faster development. The bigger and more important the system you’re building, the more helpful these techniques will be:

  1. Write your test first, and see it fail. (Red)
  2. Write your unit of code and see the test pass. (Green)
  3. Make your code better, using the tests to keep you safe. (Refactor)

Continue reading

Are you still looking for a Test Automation Framework?

We all know, designing a framework is a tedious task. So, this time I came up with an article on new framework RedwoodHQ, a Tester friendly with many built-in features and allows you to automate and manage your tests very quick.

What is RedwoodHQ?
An Open source framework which supports Selenium, Appium and allows multiple users to design test cases and develop test scripts which can be written in a ubiquitous language and also allows you to execute them all using a friendly Web interface.
RedwoodHQ supports Java/Groovy, Python and C#.

Why should I use RedwoodHQ Framework?
Using this framework, a non-tech guy can automate using the keywords available while designing the test case itself. We don’t need any IDE to configure. I guess this can be used for Bigger and Smaller projects as well.

Continue reading

What’s my open source tool for Mobile App test automation?

That small, mobile but not so little device we’re all holding in our hands is a powerful thing. Its ability to completely takeover our lives is astonishing, technology wise, it’s all the talk and rave.  Companies use it, you’re probably using it right now – maybe not to read this article, but it’s definitely not too far away.

Those devices we all use, operate on software that we expect to always work, and when it doesn’t – we get irritated.

In order for any device to work properly, all the software that makes it tick, should be tested. Undiscovered bugs or regressions can cause enormous losses for a company and frustrate your target audience.

However, the need for automating mobile application tests, as well as tests for any other platform (web or desktop) was never a consensus. There are several reasons for automation, such as speed of testing, coverage and scalability – automated tests can be performed by a computer much faster than by human hands, cover much wider scenarios and all this can be done concurrently on various environments, configurations and devices. In addition, automation reduces costs – using computers as resources cost less than using human hands. So if it’s so important, what are the right tools that will allow creating, maintaining, managing, executing and easily analyzing results?

Continue reading

World’s most desirable test automation skills!

With the growing demand for test automation experts and new technologies emerge at a rapid pace, gaining the relevant technical skills is crucial for candidates who are looking to improve their chances to get hired.

In order to answer the question – “What are the most desirable test automation skills in the upcoming year of 2016?”, our research team took over the top job searching websites around the globe looking for the most popular and required technologies in the test automation field.

Most popular test automation frameworksMost popular test automation frameworks

Selenium is considered to be the world’s most popular framework (as also reflected in ‘Test Automation Trends of 2015’). The most surprising fact regarding Selenium in the United States is that it’s almost 300% more demanded than QTP/UTF which has made to second place. This huge gap practically renders all other technologies as less relevant for people looking to learn a new automation framework.

Continue reading