Skip to main content
  1. Botium Support
  2. Technical Reference
  3. CI/CD Pipeline Integration

Add Botium Test Suite to Azure DevOps Pipeline

Note: This task assumes that you have already created a test project. If you have not yet created a test project you can open the Test Suite and follow the steps to create a project for the type of testing you will run from your Azure DevOps Pipeline.

Configure Botium Pipeline Integration

  1. Select a Project: In Botium, open a Test Project by navigating to the Test Suite menu in the left sidebar, then open the project you want to run from the Azure DevOps Pipeline (e.g., NLP, E2E, Regression, etc.)

  2. Configure Project: After you've opened your project, click the CONFIGURATION button in the top right corner, and select Build Server Integration from the tab at the top of the screen. The configure the parameters as follows:

    • Enter ${BUILD_ID} in the Build Id field

    • Select JUnit XML as output format

    • Enable the Wait with HTTP(S) response until test session completed option



  3. Select the Curl Command Line tab and copy the command line for later use.

Tip: You can use other build environment variables in the command line script as well, you can find the list of available environment variables here. You will have to map it later.

Configure Azure DevOps Pipeline

Now there are several small steps to do in the Azure DevOps Pipeline in the Azure management console.

  1. Create a DevOps Project: Create an Azure DevOps Project. The settings really don’t matter, choose them for your infrastructure.

  2. Select a Git Repository: Azure DevOps Pipeline is configured in a file called azure-pipelines.yml - this file is saved in a repository of your choice. You can host it in your own Github repository, on Bitbucket, or in your Azure Git repository - it doesn’t matter for Botium.
    Tip: If you don’t already have a Git repository, just create a new one in Azure.

  3. Create a Build Pipeline: Now add a new Build Pipeline, connecting it to your existing repository, or to our newly created repository. Select 'Starter pipeline' on the 'Configure' tab.

    1. Change YAML Definition: On the last screen, change the pipeline configuration to this yaml definition:
      trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true
      Note: In line 11, replace the curl command with the one you copied from Step 3 of the Botium Pipeline Integration, adding the --output $(System.DefaultWorkingDirectory)/testresult.xml flag.
That’s it. Azure DevOps Pipeline is ready for running Botium tests!

What happens in this pipeline?

  • It is bound to the “master” branch of the repository - any change in this repository will trigger the pipeline to run

  • First the Botium CI/CD integration is called to run the test project “Echo-Bot-Test-Suite“ and output the test result as JUnit XML

  • The file is saved to the working directory

  • The final task picks up the JUnit XML and publishes it as test result.

Running Botium Tests

Now you can run your first Botium test in Azure. There are several triggers available in Azure DevOps to trigger the test run.

You can fine-tune the triggers in the Triggers section of the pipeline configuration. You can either use the DevOps user interface to configure the triggers in your pipeline configuration, or you can modify your YAML definition.

  1. Trigger Manually: The best option for your first test run is to use the Queue function to start the Botium tests manually.

  2. Trigger from Repository: A better option for live usage is to trigger the test run from your Git repository.
    Tip: This is enabled by default. Just make a commit, and push from the connected Git repository (to one of the convo files, for example), and the test run will start automatically.
  3. Scheduling Test Runs: You can also add instructions to the pipeline configuration to run the tests regularly - this is a good option for a nightly test run for monitoring purposes, for example. You can either use the DevOps user interface to add a scheduler to your pipeline configuration, or you can modify your yaml definition.

    You can find several samples in the DevOps documentation above.

Viewing Test Results

You can view test results and failed Botium test cases in Azure DevOps builds, and drill down to see detailed Botium error messages.

Sending Email Notification

Azure DevOps includes a highly configurable notification mechanism. You can use it to send email notifications when tests fail, succeed, or recover. The configuration screen is available in the Project Settings.

Attaching Test Report to Build

By extending your YAML definition, you can download the test report from Botium in various formats and publish or store it in the Azure DevOps pipeline. This allows for long-term storage of test results, which may be needed for auditing purposes.

Example:

This yaml definition not only runs the test cases on Botium, but it also downloads the test report as PDF and as CSV file and attaches it as artifact to the Azure DevOps build:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=$BUILD_ID&WAIT=1&TAG=AzureDevOps&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.pdf "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=pdf"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.csv "https://box.botium.at/api/build/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.pdf
    artifact: Test Result (PDF)

- task: PublishPipelineArtifact@1
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.csv
    artifact: Test Result (CSV)

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true

You can then download the test reports in the Related section from the Build Summary:

Attaching Transcript to Build

You can download the transcript (conversation) from Botium for external processing or storage. It can be integrated with the Azure DevOps pipeline to store the transcript as an artifact. Supported file formats are JSON and CSV (REPORTER=csv or REPORTER=json)

Following YAML definition:

  • Runs test cases, downloads test report as XML
  • Creates an artifact from the test report
  • Downloads the transcript as CSV
  • Creates an artifact from the transcript
  • Publishes the test report"
trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: Bash@3
  displayName: Running tests, downloading test result
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/testresult.xml "https://box.botium.at/api/triggerbuild/Echo-Bot-Test-Suite?APIKEY=yyyy&BUILDID=${BUILD_ID}&WAIT=1&REPORTER=junit"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  displayName: Publish test result as artifact
  inputs:
    path: $(System.DefaultWorkingDirectory)/testresult.xml
    artifact: Test Result (XML)

- task: Bash@3
  displayName: Downloading transcript
  inputs:
    targetType: 'inline'
    script: 'curl --output $(System.DefaultWorkingDirectory)/transcript.csv "https://box.botium.at/api/transcript/$BUILD_ID?APIKEY=yyyy&REPORTER=csv"'
  env:
    BUILD_ID: $(Build.BuildNumber)

- task: PublishPipelineArtifact@1
  displayName: Publish transript as artifact
  inputs:
    path: $(System.DefaultWorkingDirectory)/transcript.csv
    artifact: Transcript (CSV)

- task: PublishTestResults@2
  displayName: Publish test results, check failed status
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)/testresult.xml'
    mergeTestResults: true
    failTaskOnFailedTests: true

Was this article helpful?

0 out of 0 found this helpful