Please read this article about crawler, it's more up to date: How to maintain chatbot regression tests with minimum effort
What is Botium Crawler?
Botium Crawler is able to find all of the conversation flows in your the chatbot.
It detects quick replies and buttons and simulates clicks on all of the options and follows the paths until it reaches the configured depth or found and open-ended question.
In the following points the Botium Crawler is introduced by walking through an example.
Step 1: Create a Crawler project
Create from quick start
In the first step of quick start you are able to chose or register a new chatbot. Then in the second step you can configure some basic property of the Conversation Crawler. In the third step within the Regression Testing set of Test Types you will find the Crawler projects
Register a new Crawler project
If you have already a chatbot, you can register your new Crawler project directly. The Conversation start messages are prefilled with two typical start messages, so the only thing you need to do add the name and to select an existing chatbot. To start a new Crawler Project, click on the Test Data Wizard in the sidebar, then select the Crawler Wizard
Both from quick start and new registration you will ended on the Dashboard of your Crawler project. Here you can see the previous Crawler session results and the current execution settings.
Step 2: Advance configuration
Conversation start messages
These are the messages with the crawler starts the conversations.
Maximum conversation steps
This is the depth of the conversation in the conversation flow. (One depth is a user-bot message pair.)
Number of welcome messages
There are chatbot which initiate the conversation without user interaction. In this case you have to specify how many welcome message will be sent by the bot.
Wait for prompt
Many chatbots answer with multiple bot messages. In this case you can define a timeout until the crawler wait for the bot messages after each simulated user message.
If the text or in the payload of any quick reply/button matches to any of the exit criteria, then that conversation is stopped there and marked as successfully ended conversation.
All text messages are saved as utterance. Crawler can recognize the non-unique utterances and merge them into one utterance.
Optionally you can select a specific agent, which you want to run the crawler jobs on.
Let’s see my Crawler project configuration. This banking example chatbot has two welcome messages, which contains buttons. The crawler can use these buttons to start the conversations, so I removed the conversation start messages. This bot is pretty small so the maximum conversation steps is just 3. This bot has multiple bot messages, so I raised the wait for prompt to two seconds to be able to sure that the crawler doesn’t miss any bot message.
Step 3: Start a crawler session
On the Overview tab you can see the results of the last Crawler Session. Here you also have an option to repeat the Crawler Session. How long a session takes to complete depends on the size/depth of the chatbot and on the "Wait for Prompt" configuration parameter also has a big impact.
In my case the session is finished within a minute.
If you scroll down from the session summary you can see the detailed result. In this case 17 conversations have been generated (12 of the conversations reached the specified depth, but five of them finished earlier).
You can also check the whole conversation model as a graph on the Flow View tab
Step 4: Open-ended questions
In the case of conversations which have stopped with open-ended question you can add user responses. You can also remove the "end of conversation" marker to add additional responses to conversations.
If you rerun the crawler session, the user responses will be handled by crawler like quick replies or buttons would be found on that certain path. So in my example I added ‘Tomorrow' and 'Wendesday’ as user response. In the conversation flow diagram you can see the result that these responses used by crawler on the corresponding path:
Step 5: Create or update a Testset from the crawler result
You are able to copy the crawler result into a new or an existing Test Set by clicking on the 'Copy test scripts into Test Set button.
Botium Crawler will help you to detect automatically all paths in the conversation flow. Additionally all detected conversation flows can be saved as Botium test cases and utterance lists and it can be used as a base for a regression test set.
For more information on how to use Botium Crawler to assist with a regression test, watch the video below;