Welcome to the Botium Knowledge Center

In Botium Box, open the Test Project by first clicking on the Test Projects menu in the left sidebar, select the Test Project you want to run from Azure DevOps Pipeline, click on the Configuration link in the top right corner, then select the Build Pipeline Integration tab and adapt the parameters as follows:

• Enter ${BUILD_ID} in the Build Id field

• Select JUnit XML as output format

• Enable the Wait with HTTP(S) response until test session completed option

Select the Curl Command Line tab and copy the command line for later use.

You can use other build environment variables in the command line script as well, you can find the list of available environment variables here. You will have to map it later.

Create an Azure DevOps Project. The settings really don’t matter, choose them for your infrastructure.

Azure DevOps Pipeline is configured in a file called azure-pipelines.yml - this file is saved in a repository of your choice. You can host it in your own Github repository, on Bitbucket or in your Azure Git repository - it doesn’t matter for Botium.

If you don’t already have a Git repository, just create a new one in Azure.

Now add a new Build Pipeline, connecting it to your existing repository or to our newly created repository. Select Starter pipeline on the Configure tab.

On the last screen, change the pipeline configuration to this yaml definition:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true

In line 11, replace the curl command with the one from your previous step, adding the --output $(System.DefaultWorkingDirectory)/testresult.xml flag.

What happens in this pipeline ?

• It is bound to the “master” branch of the repository - any change in this repository will trigger the pipeline to run

• First the Botium CI/CD integration is called to run the test project “Echo-Bot-Test-Suite“ and output the test result as JUnit XML

• The file is saved to the working directory

• The final task picks up the JUnit XML and publishes it as test result.

That’s it. Azure DevOps Pipeline is ready for running Botium tests!

Now you can run your first Botium test in Azure. There are several triggers available in Azure DevOps to trigger the test run.

You can fine-tune the triggers in the Triggers section of the pipeline configuration. You can either use the DevOps user interface to configure the triggers in your pipeline configuration, or you can modify your yaml definition.

The best option for your first test run is to use the Queue function to start the Botium tests manually.

A better option for live usage is to trigger the test run from your Git repository

• this is enabled by default: just make a commit and push from the connected Git repository (to one of the convo files, for example), and the test run will start automatically.

You can also add instructions to the pipeline configuration to run the tests regularily - this is a good option for a nightly test run for monitoring purposes, for example. You can either use the DevOps user interface to add a scheduler to your pipeline configuration, or you can modify your yaml definition.

You can find several samples in the DevOps documentation above.

You can see test results and the failed Botium test cases in Azure DevOps builds and drill down to the detailed Botium error message.

Azure DevOps includes a highly configurable notification mechanism. You can use it to send email notifications when tests fail, succeed, or recover. The configuration screen is available in the Project Settings.

By extending your yaml definition you can download the test report from Botium Box (in several formats) and publish/persist it in the Azure DevOps pipeline. This is meant for long-term storage of test results you may need for auditing purposes, for example.

This yaml definition not only runs the test cases on Botium Box, but it also downloads the test report as PDF and as CSV file and attaches it as artifact to the Azure DevOps build:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.pdf "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=pdf"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.csv "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.pdf
    artifact: Test Result (PDF)

- task: PublishPipelineArtifact@1
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.csv
    artifact: Test Result (CSV)

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true

You can then download the test reports in the Related section from the Build Summary:

It is possible to download the Transcript (the conversation) from Botium Box for external processing/storing. You can integrate it with Azure DevOps pipeline, and store Transcript as Artifact.

Supported file formats are json, and csv. (REPORTER=csv or REPORTER=json)

Following yaml definition

• runs test cases, downloads test report as XML,

• creates artifact from test report

• downloads the transcript as csv

• creates artifact from transcript

• publishes the test report:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  displayName: Running tests, downloading test result
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=${BUILD_ID}&WAIT=1&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  displayName: Publish test result as artifact
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.xml
    artifact: Test Result (XML)

- task: Bash@3
  displayName: Downloading transcript
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/transcript.csv "https://box.botium.at/api/transcript/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  displayName: Publish transript as artifact
  inputs:
    path: $(System.DefaultWorkingDirectory)/transcript.csv
    artifact: Transcript (CSV)

- task: PublishTestResults@2
  displayName: Publish test results, check failed status
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true