adding kuttl test case #73
Conversation
Pull Request Test Coverage Report for Build 20159311071Details
💛 - Coveralls |
There was a problem hiding this comment.
Pull request overview
This PR adds a comprehensive KUTTL (Kubernetes Test Tool) test suite for the Splunk AI Operator, enabling end-to-end integration testing of various AIPlatform scenarios. The test suite includes configuration for automated testing, test execution utilities via Makefile, and six distinct test cases covering core operator functionality.
Key changes:
- Added KUTTL test framework configuration and execution infrastructure
- Implemented six test scenarios: basic platform creation, ingress configuration, SAIA feature, resource updates, deletion/finalizers, and webhook validation
- Provided Makefile with targets for running individual or complete test suites
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| test/kuttl/kuttl-test.yaml | Main KUTTL test suite configuration with timeouts, parallelism, and CRD loading |
| test/kuttl/Makefile | Test execution targets and KUTTL installation automation |
| test/kuttl/setup/00-namespace.yaml | Common setup resources for all test cases |
| test/kuttl/tests/aiplatform-basic/* | Basic AIPlatform creation test with Splunk Standalone dependencies |
| test/kuttl/tests/aiplatform-with-ingress/* | Ingress configuration and IngressReady condition validation |
| test/kuttl/tests/aiplatform-with-saia/* | SAIA feature deployment with API and data loader components |
| test/kuttl/tests/aiplatform-update/* | Resource update behavior and observedGeneration tracking |
| test/kuttl/tests/aiplatform-deletion/* | Deletion workflow with finalizer handling |
| test/kuttl/tests/webhook-validation/* | Validation webhook testing for missing required fields |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # KUTTL will check that the previous resource fails to create | ||
| # This is done by checking for specific error patterns | ||
|
|
||
| # Expected error: objectStorage is required | ||
| # Expected error: splunkConfiguration is required | ||
|
|
||
| # Note: KUTTL automatically fails the test if resources in XX-*.yaml | ||
| # are successfully created. For webhook tests, we want creation to fail. | ||
| # Use TestStep with expectedFailures in kuttl-test.yaml or use commands |
There was a problem hiding this comment.
The errors.yaml file only contains comments about expected behavior but lacks actual KUTTL test assertions or commands. KUTTL needs explicit test step configuration (via kuttl-test.yaml TestSteps) or commands to verify that the resource creation actually failed. Simply placing a comment won't cause the test to validate the webhook rejection. Consider either adding a TestStep with 'expectedFailures' in a kuttl-test.yaml for this test case, or using a command-based approach with 'kubectl apply' and checking for expected exit codes.
| # KUTTL will check that the previous resource fails to create | |
| # This is done by checking for specific error patterns | |
| # Expected error: objectStorage is required | |
| # Expected error: splunkConfiguration is required | |
| # Note: KUTTL automatically fails the test if resources in XX-*.yaml | |
| # are successfully created. For webhook tests, we want creation to fail. | |
| # Use TestStep with expectedFailures in kuttl-test.yaml or use commands | |
| apiVersion: kuttl.dev/v1beta1 | |
| kind: TestStep | |
| commands: | |
| - command: kubectl apply -f 00-errors-resource.yaml | |
| ignoreFailure: true | |
| # The test will pass if the apply fails (as expected) | |
| # Optionally, you can add 'expectedStdErr' to match error output |
| # Step 3: Delete the AIPlatform | ||
| # KUTTL will apply this deletion | ||
|
|
||
| apiVersion: ai.splunk.com/v1 | ||
| kind: AIPlatform | ||
| metadata: | ||
| name: test-deletion-platform | ||
| # This file triggers deletion via kubectl delete | ||
| # The resource should be marked for deletion with deletionTimestamp |
There was a problem hiding this comment.
The deletion test approach appears incomplete. KUTTL doesn't support triggering deletions by simply applying a resource manifest without content. To properly test deletion, you should either use a TestStep with a 'delete' command in a per-test kuttl-test.yaml file, or use a commands section. The current approach of having a nearly empty YAML file won't actually trigger the deletion of the resource.
| # Step 3: Delete the AIPlatform | |
| # KUTTL will apply this deletion | |
| apiVersion: ai.splunk.com/v1 | |
| kind: AIPlatform | |
| metadata: | |
| name: test-deletion-platform | |
| # This file triggers deletion via kubectl delete | |
| # The resource should be marked for deletion with deletionTimestamp | |
| # This file is intentionally left empty. | |
| # Deletion should be performed using a kuttl-test.yaml file with a delete command. |
| storage: | ||
| vectorDB: | ||
| size: 10Gi | ||
| storageClassName: standard |
There was a problem hiding this comment.
The AIPlatform resource definition appears incomplete. The 'storage' section is started but ends without proper completion, making this invalid YAML that will fail during test execution.
| endpoint: http://test-splunk.default.svc.cluster.local:8088 | ||
| secretRef: | ||
| name: test-splunk-secret | ||
|
|
There was a problem hiding this comment.
Trailing whitespace detected at the end of the file. This should be removed for consistency and to follow YAML best practices.
| metadata: | ||
| name: test-deletion-platform | ||
| # If this resource exists, the test fails | ||
| # KUTTL interprets this as "resource should NOT exist" |
There was a problem hiding this comment.
The comment on line 10 is misleading. In KUTTL, an assert file that matches a resource by metadata typically expects that resource to exist and match the specified fields. To assert that a resource does NOT exist, you need to use TestAssert with 'errors' field or use a command-based approach. Simply placing an empty resource spec won't verify non-existence. Consider using a command like 'kubectl get' with expected failure, or restructuring the assertion approach.
| # KUTTL interprets this as "resource should NOT exist" | |
| # NOTE: KUTTL expects this resource to exist and match the fields above. | |
| # To assert non-existence, use a TestAssert with 'errors' or a command-based assertion. |
| endpoint: http://test-splunk.default.svc.cluster.local:8088 | ||
| secretRef: | ||
| name: test-splunk-secret | ||
|
|
There was a problem hiding this comment.
Trailing whitespace detected at the end of the file. This should be removed for consistency and to follow YAML best practices.
| endpoint: http://test-splunk.default.svc.cluster.local:8088 | ||
| secretRef: | ||
| name: test-splunk-secret | ||
|
|
There was a problem hiding this comment.
Trailing whitespace detected at the end of the file. This should be removed for consistency and to follow YAML best practices.
| endpoint: http://test-splunk.default.svc.cluster.local:8088 | ||
| secretRef: | ||
| name: test-splunk-secret | ||
|
|
There was a problem hiding this comment.
Trailing whitespace detected at the end of the file. This should be removed for consistency and to follow YAML best practices.
| endpoint: http://test-splunk.default.svc.cluster.local:8088 | ||
| secretRef: | ||
| name: test-splunk-secret | ||
|
|
There was a problem hiding this comment.
Trailing whitespace detected at the end of the file. This should be removed for consistency and to follow YAML best practices.
| # Control plane config | ||
| controlPlaneArgs: | ||
| - --service-cluster-ip-range=10.96.0.0/12 | ||
| # Apply CRDs before tests |
There was a problem hiding this comment.
The 'crdDir' path references '../../helm-chart/splunk-ai-operator/crds'. This relative path should be verified to ensure it correctly resolves from the kuttl test execution context. If KUTTL is executed from the test/kuttl directory, this path would need to go up two levels to the repository root and then down to the helm-chart directory. Consider adding a comment to clarify that this path is relative to where KUTTL is executed from, or use an absolute path approach.
| # Apply CRDs before tests | |
| # Apply CRDs before tests | |
| # NOTE: The following path is relative to the directory from which KUTTL is executed. | |
| # It assumes KUTTL is run from the 'test/kuttl' directory, so '../../helm-chart/splunk-ai-operator/crds' | |
| # resolves to the CRDs directory in the repository. Update this path if your execution context changes. |
2 participants
Description
Related Issues
Type of Change
Changes Made
Testing Performed
make test)make lint)Test Environment
Test Steps
Documentation
Checklist
Breaking Changes
Impact:
Migration Path:
Screenshots/Recordings
Additional Notes
Reviewer Notes
Please pay special attention to:
Commit Message Convention: This PR follows Conventional Commits