Free Practice Questions for ISTQB Certified Tester – Testing with Generative AI (CT-GenAI) Certification

This section provides a foundational understanding of Generative AI (GenAI) and its core concepts. It explains what GenAI is and how it differs from traditional (discriminative) AI. The key characteristics of Large Language Models (LLMs), such as their architecture (e.g., transformers), training process (pre-training and fine-tuning), and scale (parameters), are introduced.

Key terminology is defined, including tokens, context window, temperature, and top-p, which are essential for interacting with LLMs effectively. The section also provides an overview of different types of GenAI models beyond text generation, such as those for creating images, code, and other data formats, highlighting their relevance to various software testing activities.

This section explores the practical application of GenAI in the software testing lifecycle. It outlines the potential benefits, such as increased efficiency, improved test coverage, and accelerated feedback loops. It also addresses the inherent limitations and challenges, including the potential for errors, the need for human oversight, and the probabilistic nature of GenAI outputs.

The section illustrates how GenAI can be applied to various testing activities, from requirements analysis and test design to test data generation and test automation. It establishes core principles for the effective and responsible use of GenAI in testing, emphasizing the importance of critical thinking, domain expertise, and treating GenAI as a powerful assistant or co-pilot rather than a replacement for the professional tester.

This section introduces prompt engineering as the key skill for effectively interacting with and guiding LLMs. It defines what a prompt is and explains the iterative nature of prompt engineering. The core components of an effective prompt are detailed, including defining the role or persona for the AI, providing clear context, specifying the precise task, and defining the desired output format.

Different types of prompting techniques are explained and contrasted, including:

- Zero-shot prompting (asking a direct question without examples) - Few-shot prompting (providing a few examples to guide the model's response) - Chain-of-Thought (CoT) prompting (encouraging the model to explain its reasoning step-by-step to improve accuracy for complex tasks)

This section focuses on the practical application of prompt engineering techniques to specific software testing tasks. It provides examples of how to use GenAI to support and augment various activities throughout the test process.

Use cases covered include:

- Generating test ideas and scenarios from requirements or user stories - Creating detailed test cases in formats like Gherkin (Given-When-Then) - Producing varied and realistic test data (e.g., names, addresses, edge cases) - Assisting in the creation of test automation scripts in different programming languages - Supporting exploratory testing by suggesting areas to investigate - Improving the clarity and completeness of defect reports

This section addresses the critical need to evaluate the output generated by GenAI and the iterative process of refining prompts to improve results. It outlines key criteria for assessing the quality of GenAI-generated content for testing, such as relevance, accuracy, completeness, and clarity.

Techniques for refining prompts are discussed, including adding more context, clarifying constraints, experimenting with different phrasing, and breaking down complex requests into smaller, more manageable steps. The importance of human review and validation is emphasized, ensuring that the tester remains in control and accountable for the final work products.

This section delves into the inherent risks associated with the outputs of GenAI models. It defines and provides examples of 'hallucinations,' where the model generates factually incorrect or nonsensical information. It also covers reasoning errors, where the model fails to follow logical steps correctly. The concept of bias in GenAI is explained, covering how biases in training data can lead to skewed or unfair outputs.

The potential impact of these issues on software testing is discussed, such as generating invalid test cases or biased test data. Mitigation strategies are presented, including fact-checking outputs, using multiple sources for verification, and designing prompts that reduce the likelihood of such errors.

This section focuses on the critical risks related to data privacy and security when using GenAI tools, particularly public-facing services. It highlights the danger of inputting sensitive information, such as personally identifiable information (PII), intellectual property, or confidential business data, into prompts, which could lead to data leaks.

Security vulnerabilities, such as prompt injection attacks, are also discussed. The section outlines essential mitigation techniques, including data anonymization, using private or on-premise GenAI models for sensitive tasks, and adhering to organizational policies on data handling and security.

This section raises awareness of the significant energy consumption and environmental footprint associated with training and operating large-scale GenAI models. It explains the factors contributing to this impact, such as the computational resources required for model training and inference.

The section encourages a mindset of responsible and sustainable AI usage. It discusses potential strategies for mitigation, such as using smaller, more efficient models when appropriate, leveraging models hosted in energy-efficient data centers, and being mindful of the frequency and complexity of GenAI queries.

This section provides an overview of the evolving legal and ethical landscape surrounding AI. It introduces key emerging regulations, such as the EU AI Act, and explains their potential implications for the use of AI in software development and testing. The importance of adhering to relevant international standards, such as those from ISO/IEC (e.g., ISO/IEC 42001 for AI management systems), is highlighted.

Best practice frameworks for responsible and trustworthy AI are also discussed, emphasizing principles like transparency, fairness, accountability, and human oversight. The section underscores the tester's role in ensuring that AI-powered systems and the use of AI in testing align with these regulations and ethical guidelines.

This section explores different ways to architect and integrate LLM capabilities into the testing toolchain and environment. It compares and contrasts various architectural patterns, explaining their respective advantages and disadvantages.

Approaches covered include:

- Using public LLM APIs (e.g., from OpenAI, Google, Anthropic) for general-purpose tasks - Deploying open-source LLMs in a private cloud or on-premise for enhanced data privacy and control - Implementing the Retrieval-Augmented Generation (RAG) pattern, which allows an LLM to access and utilize information from a specific knowledge base (e.g., project documentation) to provide more context-aware and accurate responses.

This section introduces more advanced concepts for tailoring and managing GenAI solutions. It explains the concept of fine-tuning, a process where a pre-trained LLM is further trained on a smaller, domain-specific dataset to improve its performance on specialized tasks. The benefits of fine-tuning, such as increased accuracy and adherence to specific formats, are discussed.

The section also introduces the concept of LLMOps (Large Language Model Operations), which is an extension of MLOps. LLMOps encompasses the practices and tools required to manage the entire lifecycle of LLM-based applications, including data management, model versioning, continuous integration/continuous deployment (CI/CD), monitoring, and governance.

This section provides a strategic framework for introducing and scaling the use of GenAI within a testing organization. It outlines the key steps in creating a roadmap for adoption, starting with defining clear objectives and identifying high-value use cases. The importance of starting with pilot projects to experiment and demonstrate value is emphasized.

The process of evaluating and selecting appropriate GenAI tools is discussed, with criteria such as functionality, integration capabilities, security, and cost. The section provides guidance on how to scale successful pilots across the wider organization, establishing best practices and governance along the way.

This section addresses the human and organizational aspects of adopting GenAI in testing. It highlights the importance of effective change management to ensure a smooth and successful transition. Key activities include providing training and upskilling opportunities for the test team to develop new competencies like prompt engineering.

The need to adapt existing testing processes and workflows to incorporate GenAI tools is explained. The section also emphasizes the importance of fostering a culture that encourages experimentation, collaboration, and continuous learning, allowing the team to discover the most effective ways to leverage GenAI while managing its risks.