WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content

Conversation

@skamenan7
Copy link
Contributor

What does this PR do?

Adds Bedrock integration tests to CI using a record/replay mechanism. Tests run against pre-recorded API responses, without the need for AWS credentials in CI.

The main challenge was that Bedrock's OpenAI-compatible API doesn't support everything - no tool calling, no embeddings, no dynamic model listing. So instead of running the full base suite (which would fail on ~40 tests), I created a dedicated bedrock suite with just the tests that actually work.

Changes:

  • New run-bedrock.yaml stack config with dummy API key for replay mode
  • bedrock suite in suites.py pointing to 3 test functions (6 parametrized tests total)
  • Config resolution fix for distro::file.yaml format in library mode
  • Added a docs/source/providers/inference/bedrock_recording_guide.md so contributors with AWS access can re-record tests when needed

Closes #4095

Test Plan

Run from tests/integration/inference:


  uv run pytest -v \
    test_openai_completion.py::test_openai_chat_completion_non_streaming \
    test_openai_completion.py::test_openai_chat_completion_streaming \
    test_openai_completion.py::test_inference_store \
    --setup=bedrock \
    --stack-config=ci-tests::run-bedrock.yaml \
    --inference-mode=replay \
    -k "client_with_models"

Expected: 6 passed

test_openai_completion.py::test_openai_chat_completion_non_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_com
 pletion:non_streaming_01] PASSED [ 16%]
 test_openai_completion.py::test_openai_chat_completion_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_complet
 ion:streaming_01] PASSED [ 33%]
 test_openai_completion.py::test_inference_store[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-True] PASSED                          [
  50%]
 test_openai_completion.py::test_openai_chat_completion_non_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_com
 pletion:non_streaming_02] PASSED [ 66%]
 test_openai_completion.py::test_openai_chat_completion_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_complet
 ion:streaming_02] PASSED [ 83%]
 test_openai_completion.py::test_inference_store[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-False] PASSED
 [100%]

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 3, 2025
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 7 times, most recently from 305be8e to 0667de8 Compare December 3, 2025 22:46
providers:
inference:
- provider_id: bedrock
provider_type: remote::bedrock
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remote::bedrock is part of ci-tests/run.yaml , can this be used instead (i.e. no need for a new run.yaml)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I'll check if ci-tests/run.yaml can be used directly. The separate file was created to have a minimal bedrock-only config, but if the existing run.yaml already includes bedrock provider, I can simplify by using that instead.

{
"default": [
{"suite": "base", "setup": "ollama"},
{"suite": "bedrock", "setup": "bedrock", "allowed_clients": ["library"], "stack_config": "ci-tests::run-bedrock.yaml"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we can just add this to CI here, anybody updating tests that need new recording would need a bedrock key (or we'd need a key in CI for new recordings, is that the plan?)

@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 3 times, most recently from 69eb2c4 to 51121ae Compare December 4, 2025 20:13
@skamenan7 skamenan7 marked this pull request as ready for review December 4, 2025 21:36
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch from d36bfe4 to 800ef86 Compare December 5, 2025 15:10
- Create dedicated bedrock suite with 3 compatible test functions
- Add run-bedrock.yaml stack config for CI
- Enable config resolution for distro::file.yaml format in library mode
- Add test recordings for streaming, non-streaming, and inference store
- Skip tool-calling tests for Bedrock (not supported by AWS)
- Add recording guide documentation for contributors

Tested with GPT-OSS model on us-west-2 region.
The bedrock suite uses specific test function paths like
"test_file.py::test_function" in its roots. The pytest_ignore_collect
hook was treating these as filesystem paths, causing 0 tests to be
collected.

Changes:
- Strip "::test_function" suffix when checking file paths
- Add pytest_collection_modifyitems to filter to specific tests

Without this fix, cleanup_recordings.py marks bedrock recordings as
unused and deletes them in CI.
- Skip embedding model validation for bedrock setup (empty string forces override)
- Exclude stream_options from hash computation for consistent recordings

The bedrock provider adds stream_options for telemetry but this was causing
hash mismatches since recordings were made before this was added. Now
infrastructure/telemetry fields are excluded from the hash computation.
- Remove separate run-bedrock.yaml in favor of modifying templates
- Update bedrock provider config: dummy API key for replay mode, us-west-2 region
- Pre-register bedrock model in ci-tests template (Bedrock /v1/models returns empty)
- Update ci_matrix.json to use default stack config
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch from 800ef86 to bd9f2dd Compare December 8, 2025 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Bedrock Provider to CI Integration Tests

2 participants