-
Notifications
You must be signed in to change notification settings - Fork 1.2k
feat(ci): add Bedrock integration tests with record/replay #4292
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
305be8e to
0667de8
Compare
| providers: | ||
| inference: | ||
| - provider_id: bedrock | ||
| provider_type: remote::bedrock |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remote::bedrock is part of ci-tests/run.yaml , can this be used instead (i.e. no need for a new run.yaml)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point! I'll check if ci-tests/run.yaml can be used directly. The separate file was created to have a minimal bedrock-only config, but if the existing run.yaml already includes bedrock provider, I can simplify by using that instead.
tests/integration/ci_matrix.json
Outdated
| { | ||
| "default": [ | ||
| {"suite": "base", "setup": "ollama"}, | ||
| {"suite": "bedrock", "setup": "bedrock", "allowed_clients": ["library"], "stack_config": "ci-tests::run-bedrock.yaml"}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we can just add this to CI here, anybody updating tests that need new recording would need a bedrock key (or we'd need a key in CI for new recordings, is that the plan?)
69eb2c4 to
51121ae
Compare
d36bfe4 to
800ef86
Compare
- Create dedicated bedrock suite with 3 compatible test functions - Add run-bedrock.yaml stack config for CI - Enable config resolution for distro::file.yaml format in library mode - Add test recordings for streaming, non-streaming, and inference store - Skip tool-calling tests for Bedrock (not supported by AWS) - Add recording guide documentation for contributors Tested with GPT-OSS model on us-west-2 region.
The bedrock suite uses specific test function paths like "test_file.py::test_function" in its roots. The pytest_ignore_collect hook was treating these as filesystem paths, causing 0 tests to be collected. Changes: - Strip "::test_function" suffix when checking file paths - Add pytest_collection_modifyitems to filter to specific tests Without this fix, cleanup_recordings.py marks bedrock recordings as unused and deletes them in CI.
- Skip embedding model validation for bedrock setup (empty string forces override) - Exclude stream_options from hash computation for consistent recordings The bedrock provider adds stream_options for telemetry but this was causing hash mismatches since recordings were made before this was added. Now infrastructure/telemetry fields are excluded from the hash computation.
- Remove separate run-bedrock.yaml in favor of modifying templates - Update bedrock provider config: dummy API key for replay mode, us-west-2 region - Pre-register bedrock model in ci-tests template (Bedrock /v1/models returns empty) - Update ci_matrix.json to use default stack config
800ef86 to
bd9f2dd
Compare
What does this PR do?
Adds Bedrock integration tests to CI using a record/replay mechanism. Tests run against pre-recorded API responses, without the need for AWS credentials in CI.
The main challenge was that Bedrock's OpenAI-compatible API doesn't support everything - no tool calling, no embeddings, no dynamic model listing. So instead of running the full base suite (which would fail on ~40 tests), I created a dedicated bedrock suite with just the tests that actually work.
Changes:
docs/source/providers/inference/bedrock_recording_guide.mdso contributors with AWS access can re-record tests when neededCloses #4095
Test Plan
Run from tests/integration/inference:
Expected: 6 passed