Van Record & Playback naar CI/CD

An test automation programming mind set Harold van Geenhuizen Test tool expert for Sogeti Julianadorp 2016

Intro Wie ben ik Wat gaan we doen Mindset Code uitpluizen Kan het anders/beter 10 Golden rules of Test automation Een framework

Testcase: een begin Als gemeente wil ik een kenteken opvragen bij de RDW, en de brandstof soort en het bouwjaar uitlezen, dit om te bepalen of het voertuig de milieu zone in mag.

Een stukje opgenomen code

Colour Syntax

Parameters In het voorbeeld zien we de kentekens, deze staan nu “hardgecodeerd”. Als we deze parameteriseren, dan kunnen we willekeurige kentekens invoegen. Voorbeeld:

Lekker simpel De browser.page combinatie kunnen we korter maken Voorbeeld:

Herhaling Wat we ook zien, is dat steeds weer hetzelfde gebeurt Voorbeeld:

Meer als een keer? Maak een functie

Sub of Function Een SUB: Geeft niets terug, voert alleen uit. Een FUNCTION: Geeft een waarde terug, na het uitvoeren. Makkelijk voor Flow controle, Error handling, berekingen

Function met error handling

Function met flow control

Kan het beter? Google is your friend: Open Data RDW: Gekentekende_voertuigen_voertuigklasse https://dev.socrata.com/foundry/opendata.rdw.nl/kmfi-hrps https://opendata.rdw.nl/resource/kmfi-hrps.json Via service testen, makkelijker en sneller dan via website. In dit geval via een rest service kan de case ook getest worden. https://opendata.rdw.nl/resource/kmfi-hrps.json?kenteken=77BFZ3

10 golden rules of test automation Test automation is not the same as automating your tests Determine WHAT you want to test/automate Document what you are testing Determine WHEN you want to test what you want to test Think about your action words Choose the right interface Structure your test data Plan your maintenance Keep your test ‘small’ Decide what evidence you want to save Below are 10 ‘Golden rules’ that help you in achieving the most out of test automation. Test automation is not the same as automating your tests This sounds contradictory but what we mean to say here is that merely picking up your existing tests cases and automating them without looking at the quality of the test cases and without looking at a proper test automation strategy can result in a lot of effort spend on test automation without the proper return of your investment and ultimately resulting in negative feelings about the test automation and reluctance to use it or continue with further investments. So in short ‘look before you leap ’. Determine WHAT you want to test/automate Test automation is only useful for repetitive tasks. One time tests, often related to interface changes, are not worthwhile to automate. Test cases that are related to correctness of functionality that is at high risk to be impacted by changes in any part of the application are very valuable to automate because they are bound to be run very frequently in regression tests. Key word in the previous sentence is risk. So before automating your test cases ensure that you have a proper test strategy in place that identifies what are the functionalities that can be considered as having a high risk and thus requires test cases that need to be part of the regression test suite. Document what you are testing  Once you have determined what and when for test automation, you need to ensure that your test ware is properly documented. Even though a large part of your test is or will be automated, it is always required to properly document your test case. Think about description, goal, risk / priority, steps needed to go through the test, the test data that you need to perform the test and of course the expected result. You need all this information if you want to properly automate so you might as well write it down properly in such a way that it is always possible to run manual (might be needed if e.g. the automation tool is not working properly, or changes in the application prevent proper automated execution and the scripts need too much rework) and it makes the automation a lot easier. Determine WHEN you want to test what you want to test CI/CD identifies a number of principles: one of them is “integrate quick and often” another one is “get continuous feedback”. Combined this mean that you need to ensure that you get feedback about the quality of your build as early as possible and throughout all stages. But you also have to think about the amount of testing and depth of testing and thus automation you want to perform in which stage of the build and delivery of your user stories and in which environment. Keeping in mind that finding and solving defects early in the life cycle should lead to more automation in unit test than in the acceptance test: “Shift left”. Next to that you do not have to run all test cases always at all times. It could be useful to determine subsets based on several aspects such as risk / priority, test needed for first indication proper installation (smoke test or intake test) or full regression test, tests only on specific environments, test levels etc.. Think about your action words Setting up actions words is a good way to keep your tests readable and maintainable. Carefully choosing which actions words to set up, will help in obtaining an optimum. Choosing actions words at a too detailed level will make it harder to set up the scenarios, choosing action words at a too general level, will make it hard to reuse them in other scenarios and harder to maintain. Choose the right interface If you need to test a GUI than it makes sense to use the GUI interface, but if you want to test functionality or logic in the middleware or back-office, it might make more sense to use API’s to do your testing. Why? Because a. testing via an API is mostly much quicker than testing via a GUI and b. API’s in general are more stable than user interfaces, reducing the risk of costly maintenance. Structure your test data The success of your automated tests is highly dependent on the availability of the correct test data. Some of this test data will also be ‘consumed’ by the execution of the tests, such that subsequent runs of the test set cannot use the same test data. Make sure you have a strategy to deal with this type of situations (for instance by having a means of restoring the data, or by making sure that you set up your own data as part of the test set.) Plan your maintenance Once you have set up your automated test set, you are not done. At every change of in functionality, at every change in the interfaces, you might have to do some rework to keep your scripts/test suite up to date. If you have set up your actions words and test cases the right way, maintenance is probably easy to do. But in every sprint you should have a user story to update the automated tests to reflect the changes that are developed in that sprint  Keep your test ‘small’ Setting up long test cases that test all kinds of validations in one test case with multiple actions, might seem to be efficient, since executing one test case shows a lot of information. However, by having multiple ‘smaller’ test cases, it easier to determine what goes wrong when a test case fails and it is easier to run multiple test sets in parallel, specifically as soon as the central UFT server, will schedule the execution of test sets/test cases. Decide what evidence you want to save Easy answer would be everything. But logging every step of the test case in your test automation tool influences the run time of the scripts. Logging also screen dumps increases the run time tremendously. So discuss with the team and your stakeholders what kind of evidence is needed in which situations. For most teams it is enough to log results with only pass or fail and only log more information such as screen dumps of the situation in case of an error or unexpected result .  Denken aan reusability?  Also the goal of your testing?  Cleaning up? Zie onder  In het algemeen nog iets zeggen over het clean up principe, het system in de dezelfde staat achterlaten als voor de test?  

A Test framework? One way of working Interchangability Reusability Better maintainable Remove the technical from the testers Needed for CI/CD Oke, maar hoe dan

Wat voor een framework Keyword driven Data driven Hybrid

Keywords Technisch vs Functional Een woord dat een treffende omschrijving van een onderdeel/actie weergeeft: Web_Click : Clickt op een web button SOAP_Send Message: Verstuurt een SOAP bericht Web_Field_Read: Leest een web scherm veld OF Gmail Login: Logt in op een gmail account Gmail count unread mail: telt het aantal ongelezen mails Gmail close: logt uit by gmail

Functional vs technical Scenario Functional layer Technical layer Web Mobile SAP Database

Test Automation programmer Testers Colored is filled with data

Framework design Door Sogeti is de aanpak TAKT (Testen, Automatiseren, Kennis, Tools) ontwikkeld, waarbij testen centraal staat en automatiseringsprincipes en tools op doeltreffende wijze worden toegepast. TAKT is de aanpak waarmee het handmatige testproces omgezet wordt in een geautomatiseerd test-proces waarin sneller en beter getest kan worden. TAKT is een aanpak voor geautomatiseerd testen, gebaseerd op de volgende visie: Toon zo snel mogelijk het rendement van testautomatisering om zo snel mogelijk de verwachtingen te bevestigen of te kunnen bijsturen Bruikbaarheid van de technische oplossing is de belangrijkste succesfactor voor acceptatie van de geautomatiseerde oplossing Begin relatief klein en eenvoudig, zorg daarna voor continue uitbreiding en verbetering Investeer in onderhoudbaarheid van alle producten voor testautomatisering Investeer in beheer van alle producten voor testautomatisering Waarom TAKT? TAKT is een gestructureerde en resultaatgerichte aanpak, die niet alleen aandacht besteed aan de technische aspecten testautomatisering maar ook aan de organisatorische. Met TAKT wordt testautomatisering op een bruikbare manier ingericht en wordt gezorgd dat de geautomatiseerde tests worden gebruikt en onderhouden. Het behalen van de gestelde doelen wordt bij TAKT centraal gesteld.

Stelling “Automating Tests ≠ Test Automation”

Start Thinking as a Developer