• EDITED PAGE

There is no exact number of individual fish in the world, but scientists estimate there are about 3.5 trillion fish in the oceans. It is impossible to know the exact total due to the vastness of aquatic environments and factors like overfishing, which affect population numbers

• Goal: evaluate KBOT’s performance in handling multilingual documents by scoring its Faithfulness and Context Recall metrics as per the RAGAs framework. List of available metrics – Ragas
• Test Scope:
  • Languages: English, Portuguese, French, German, Italian, Turkish, Spanish, Dutch, and Swedish
  • Document Requirements:
    • minimum 100 pages
    • format: PDF
    • varied formatting (tables, pictures, multiple font sizes, etc.)
• Testing Process:
  • Preparation:
    • Select a standardized source document, ideally publicly available in the languages listed above
    • Ensure the same formatting across all language versions
  • Execution:
    • Input each document into KBOT for processing
      • On data ingestion use different chunking strategies
      • Create two different applications where the rephrase flow is activated and deactivated
      • Create two different applications where the data masking is turned on and off
      • Create different applications with different AI models
    • Use the same questions across different languages
      • Also to be tested when the data source is in language A, but on the chatbot page the user sets the “language” in the “Advanced config” a different language
    • Evaluate the output, in each language, for:
      • Faithfulness: Accuracy of the response to the input
      • Context Recall: Ability to maintain context and refer back correctly
      • Use a 1 to 5 scale (1 = Poor, 5 = Excellent)
• Deliverables:
  • Report containing the following:
    • Per-Language Faithfulness and Context Recall scores
    • Identified areas for improvement