Performance testing with Botium helps ensure your chatbot can handle real-world usage. Functional testing which answers a simple question: is it working, or not. Performance tests are simulating human users, so they can answer more sensitive questions.
Key Questions Answered:
- Handling Many Users: Use stress tests to find chatbot limits and recovery abilities.
- Memory Leaks: Conduct load tests to check for resource issues.
- Consistent Responses: Use performance tests to verify consistent answers.
- Thread Safety: Perform stress tests to detect issues with parallel conversations.
Performance Testing Introduction
How Does your Chatbot handle Multiple Users?
Before publishing a bot, it's important to know its limits and how it behaves beyond those limits. Can it be overwhelmed? Does it recover after a heavy load?
Stress test can help you to answer this question.
Do we have Some (memory) Leak?
The conversation is stateless, so executing the same conversation on the same server should take about the same time. If the chatbot starts a never-ending process or fails to release resources like memory or background storage, the system will slow down over time or crash suddenly.
Load test is for detecting this.
Are the Conversations Stateless?
Chatbots are mostly static nowadays, providing the same answer to the same question. Repeating a question is a good test. Sometimes, the first attempt works well, but the second attempt might result in an error.
Every Performance test can detect this problem, because the nature of the Performance Tests is repeating a convo.
Are the Conversations Thread-Safe?
Many problems can arise with parallel execution, such as deadlocks, improper resource sharing, or synchronization issues.
Performance tests can detect these issues, but they're not foolproof. A test might run smoothly ten times in parallel and fail the next time due to system complexity. Therefore, while every performance test can identify these problems, stress tests are particularly effective for this purpose. Botium can sometimes uncover these issues as a side effect.
Every Performance test can detect this problem, but a Stress test fits better for this.