Welcome to the Botium Knowledge Center

Create an Azure DevOps Project. The settings really don’t matter, choose them for your infrastructure.

Azure DevOps Pipeline is configured in a file called azure-pipelines.yml - this file is saved in a repository of your choice. You can host it in your own Github repository, on Bitbucket or in your Azure Git repository - it doesn’t matter for Botium.

If you don’t already have a Git repository, just create a new one in Azure.

Now add a new Build Pipeline, connecting it to your existing repository or to our newly created repository. Select Starter pipeline on the Configure tab.

On the last screen, change the pipeline configuration to this yaml definition:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true

In line 11, replace the curl command with the one from your previous step, adding the --output $(System.DefaultWorkingDirectory)/testresult.xml flag.

What happens in this pipeline ?

• It is bound to the “master” branch of the repository - any change in this repository will trigger the pipeline to run

• First the Botium CI/CD integration is called to run the test project “Echo-Bot-Test-Suite“ and output the test result as JUnit XML

• The file is saved to the working directory

• The final task picks up the JUnit XML and publishes it as test result.

That’s it. Azure DevOps Pipeline is ready for running Botium tests!

The best option for your first test run is to use the Queue function to start the Botium tests manually.

A better option for live usage is to trigger the test run from your Git repository

• this is enabled by default: just make a commit and push from the connected Git repository (to one of the convo files, for example), and the test run will start automatically.

You can also add instructions to the pipeline configuration to run the tests regularily - this is a good option for a nightly test run for monitoring purposes, for example. You can either use the DevOps user interface to add a scheduler to your pipeline configuration, or you can modify your yaml definition.

You can find several samples in the DevOps documentation above.

Azure DevOps includes a highly configurable notification mechanism. You can use it to send email notifications when tests fail, succeed, or recover. The configuration screen is available in the Project Settings.

By extending your yaml definition you can download the test report from Botium (in several formats) and publish/persist it in the Azure DevOps pipeline. This is meant for long-term storage of test results you may need for auditing purposes, for example.

This yaml definition not only runs the test cases on Botium, but it also downloads the test report as PDF and as CSV file and attaches it as artifact to the Azure DevOps build:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.pdf "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=pdf"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.csv "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.pdf
    artifact: Test Result (PDF)

- task: PublishPipelineArtifact@1
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.csv
    artifact: Test Result (CSV)

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true

You can then download the test reports in the Related section from the Build Summary:

It is possible to download the Transcript (the conversation) from Botium for external processing/storing. You can integrate it with Azure DevOps pipeline, and store Transcript as Artifact.

Supported file formats are json, and csv. (REPORTER=csv or REPORTER=json)

Following yaml definition

• runs test cases, downloads test report as XML,

• creates artifact from test report

• downloads the transcript as csv

• creates artifact from transcript

• publishes the test report:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  displayName: Running tests, downloading test result
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=${BUILD_ID}&WAIT=1&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  displayName: Publish test result as artifact
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.xml
    artifact: Test Result (XML)

- task: Bash@3
  displayName: Downloading transcript
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/transcript.csv "https://box.botium.at/api/transcript/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  displayName: Publish transript as artifact
  inputs:
    path: $(System.DefaultWorkingDirectory)/transcript.csv
    artifact: Transcript (CSV)

- task: PublishTestResults@2
  displayName: Publish test results, check failed status
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true