[ENH] Make it possible to deploy multiple chromas in Tiltfile

[jasonvigil]

Description of changes

Also, remove hardcoded sysdb namespace. And, update our existing configs to fully specify hostnames (including namespace). The *2.yaml variants of our configs should only differ from the original configs in the various places we configure the namespace for the chroma deployment. In the future, we can consider generating the new chroma region's configs based on the original configs.

To deploy (and tear down) a single chroma:

tilt up
tilt down

To deploy (and tear down) multiple chromas:

export MULTI_REGION=true
tilt up
tilt down

Test plan

CI

Migration plan

N/A

Observability plan

N/A

Documentation Changes

N/A

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[github-actions]

⚠️ The Helm chart was updated without a version bump. Your changes will only be published if the version field in k8s/distributed-chroma/Chart.yaml is updated.

[propel-code-bot]

Enable multi-region Chroma deployments via Tilt

This PR extends the Tiltfile and Kubernetes manifests to support bringing up two isolated Chroma environments (namespaces chroma and chroma2) from the same Tilt session, gated by the MULTI_REGION environment variable. It also aligns existing single-region configs with fully qualified cluster DNS hostnames and removes hard-coded namespace assumptions across Rust worker/front-end configs and the Go memberlist manager.

Key Changes

• Expanded Tiltfile to template both region-specific Helm releases, define duplicated k8s_resource groups, and toggle resource sets based on MULTI_REGION using config.set_enabled_resources.
• Added second-region configuration files (distributed2.yaml, chroma_config2.yaml, values2.yaml, postgres2.yaml, etc.) mirroring primary configs with namespace- and bucket-specific overrides.
• Updated existing configs to reference fully qualified service hostnames (*.chroma.svc.cluster.local) and to declare memberlist custom-resource namespaces explicitly.
• Generalized CRMemberlistStore.UpdateMemberlist to respect the injected namespace instead of assuming chroma.

Affected Areas

• Tiltfile
• rust/worker/chroma_config.yaml
• rust/worker/chroma_config2.yaml
• rust/frontend/sample_configs/distributed.yaml
• rust/frontend/sample_configs/distributed2.yaml
• k8s/distributed-chroma/values.yaml
• k8s/distributed-chroma/values.dev.yaml
• k8s/distributed-chroma/values2.yaml
• k8s/distributed-chroma/values2.dev.yaml
• k8s/test/postgres2.yaml
• go/pkg/memberlist_manager/memberlist_store.go

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[Maintainability] The addition of multi-region support has introduced significant duplication in this Tiltfile. For example, the k8s_resource blocks for k8s_setup and k8s_setup2, as well as the service definitions for chroma and chroma2, are nearly identical.

While this works for two regions, it will be hard to maintain and scale to more regions. As a best practice to keep the configuration DRY, consider refactoring this to use a loop over a list of region configurations. This would allow you to generate the resources for each region dynamically, making the Tiltfile more concise and easier to extend.

Context for Agents

The addition of multi-region support has introduced significant duplication in this `Tiltfile`. For example, the `k8s_resource` blocks for `k8s_setup` and `k8s_setup2`, as well as the service definitions for `chroma` and `chroma2`, are nearly identical.

While this works for two regions, it will be hard to maintain and scale to more regions. As a best practice to keep the configuration DRY, consider refactoring this to use a loop over a list of region configurations. This would allow you to generate the resources for each region dynamically, making the `Tiltfile` more concise and easier to extend.

File: Tiltfile
Line: 269

[propel-code-bot]

[Documentation] In the PR description, you mentioned that the *2.yaml configs should only differ in namespace. However, I noticed that this file also changes the S3 bucket name to chroma-storage2 (e.g., here and for other services). This seems like a reasonable approach for isolating regions, but it would be helpful to clarify this in the PR description to align the stated intent with the implementation.

Context for Agents

In the PR description, you mentioned that the `*2.yaml` configs should only differ in namespace. However, I noticed that this file also changes the S3 bucket name to `chroma-storage2` (e.g., here and for other services). This seems like a reasonable approach for isolating regions, but it would be helpful to clarify this in the PR description to align the stated intent with the implementation.

File: rust/worker/chroma_config2.yaml
Line: 30