We prepared a step-by-step guide how to connect Botium to the Azure DevOps Pipeline.
Configure Botium Pipeline Integration
In Botium, open the Test Project by first clicking on the Test Projects menu in the left sidebar, select the Test Project you want to run from Azure DevOps Pipeline, click on the Configuration link in the top right corner, then select the Build Pipeline Integration tab and adapt the parameters as follows:
-
Enter ${BUILD_ID} in the Build Id field
-
Select JUnit XML as output format
-
Enable the Wait with HTTP(S) response until test session completed option
Select the Curl Command Line tab and copy the command line for later use.
You can use other build environment variables in the command line script as well, you can find the list of available environment variables here. You will have to map it later.
Configure Azure DevOps Pipeline
Now there are several small steps to do in the Azure DevOps Pipeline in the Azure management console.
Create a DevOps Project
Create an Azure DevOps Project. The settings really don’t matter, choose them for your infrastructure.

Select a Git Repository
Azure DevOps Pipeline is configured in a file called azure-pipelines.yml - this file is saved in a repository of your choice. You can host it in your own Github repository, on Bitbucket or in your Azure Git repository - it doesn’t matter for Botium.
If you don’t already have a Git repository, just create a new one in Azure.
Create a Build Pipeline
Now add a new Build Pipeline, connecting it to your existing repository or to our newly created repository. Select Starter pipeline on the Configure tab.

On the last screen, change the pipeline configuration to this yaml definition:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash@3
inputs:
targetType: 'inline'
script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
env:
BUILD_ID: $(Build.BuildNumber)
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
mergeTestResults: true
failTaskOnFailedTests: true
In line 11, replace the curl command with the one from your previous step, adding the --output $(System.DefaultWorkingDirectory)/testresult.xml flag.
What happens in this pipeline ?
-
It is bound to the “master” branch of the repository - any change in this repository will trigger the pipeline to run
-
First the Botium CI/CD integration is called to run the test project “Echo-Bot-Test-Suite“ and output the test result as JUnit XML
-
The file is saved to the working directory
-
The final task picks up the JUnit XML and publishes it as test result.
That’s it. Azure DevOps Pipeline is ready for running Botium tests!
Running Botium tests
Now you can run your first Botium test in Azure. There are several triggers available in Azure DevOps to trigger the test run.
You can fine-tune the triggers in the Triggers section of the pipeline configuration. You can either use the DevOps user interface to configure the triggers in your pipeline configuration, or you can modify your yaml definition.

Trigger Manually
The best option for your first test run is to use the Queue function to start the Botium tests manually.

Trigger from Repository
A better option for live usage is to trigger the test run from your Git repository
-
this is enabled by default: just make a commit and push from the connected Git repository (to one of the convo files, for example), and the test run will start automatically.
Scheduling Test Runs
You can also add instructions to the pipeline configuration to run the tests regularily - this is a good option for a nightly test run for monitoring purposes, for example. You can either use the DevOps user interface to add a scheduler to your pipeline configuration, or you can modify your yaml definition.
You can find several samples in the DevOps documentation above.

Viewing Test Results
You can see test results and the failed Botium test cases in Azure DevOps builds and drill down to the detailed Botium error message.

Sending Email Notification
Azure DevOps includes a highly configurable notification mechanism. You can use it to send email notifications when tests fail, succeed, or recover. The configuration screen is available in the Project Settings.

Attaching Test Report to Build
By extending your yaml definition you can download the test report from Botium (in several formats) and publish/persist it in the Azure DevOps pipeline. This is meant for long-term storage of test results you may need for auditing purposes, for example.
This yaml definition not only runs the test cases on Botium, but it also downloads the test report as PDF and as CSV file and attaches it as artifact to the Azure DevOps build:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash@3
inputs:
targetType: 'inline'
script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
env:
BUILD_ID: $(Build.BuildNumber)
- task: Bash@3
inputs:
targetType: 'inline'
script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.pdf "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=pdf"'
env:
BUILD_ID: $(Build.BuildNumber)
- task: Bash@3
inputs:
targetType: 'inline'
script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.csv "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
env:
BUILD_ID: $(Build.BuildNumber)
- task: PublishPipelineArtifact@1
inputs:
path: $(System.DefaultWorkingDirectory)/testresult.pdf
artifact: Test Result (PDF)
- task: PublishPipelineArtifact@1
inputs:
path: $(System.DefaultWorkingDirectory)/testresult.csv
artifact: Test Result (CSV)
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
mergeTestResults: true
failTaskOnFailedTests: true
You can then download the test reports in the Related section from the Build Summary:

Attaching Transcript to Build
It is possible to download the Transcript (the conversation) from Botium for external processing/storing. You can integrate it with Azure DevOps pipeline, and store Transcript as Artifact.
Supported file formats are json, and csv. (REPORTER=csv
or REPORTER=json
)
Following yaml definition
-
runs test cases, downloads test report as XML,
-
creates artifact from test report
-
downloads the transcript as csv
-
creates artifact from transcript
-
publishes the test report:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash@3
displayName: Running tests, downloading test result
inputs:
targetType: 'inline'
script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=${BUILD_ID}&WAIT=1&REPORTER=junit"'
env:
BUILD_ID: $(Build.BuildNumber)
- task: PublishPipelineArtifact@1
displayName: Publish test result as artifact
inputs:
path: $(System.DefaultWorkingDirectory)/testresult.xml
artifact: Test Result (XML)
- task: Bash@3
displayName: Downloading transcript
inputs:
targetType: 'inline'
script: 'curl --output $(System.DefaultWorkingDirectory)/transcript.csv "https://box.botium.at/api/transcript/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
env:
BUILD_ID: $(Build.BuildNumber)
- task: PublishPipelineArtifact@1
displayName: Publish transript as artifact
inputs:
path: $(System.DefaultWorkingDirectory)/transcript.csv
artifact: Transcript (CSV)
- task: PublishTestResults@2
displayName: Publish test results, check failed status
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
mergeTestResults: true
failTaskOnFailedTests: true
Comments
0 comments
Please sign in to leave a comment.