TFX Tuner on Vertex AI fails with DEADLINE\_EXCEEDED, no custom job launched

[Wolff-Lucas]

System information

• Have I specified the code to reproduce the issue: Yes (pipeline with TFX Tuner on Vertex AI)
• Environment in which the code is executed: Google Cloud Vertex AI Pipelines
• TensorFlow version: 2.16
• TFX Version: 1.16.0
• Python version: 3.10

---

Describe the current behavior

When running a pipeline with the TFX Tuner component on Vertex AI, the tuning job gets stuck.
No Vertex AI custom job or study for hyperparameter tuning is launched.

It gets stuck right after returning from:

return TunerFnResult(tuner=tuner, fit_kwargs=fit_kwargs)

I have tested with AI Vizier, Cloud Tuner, RandomSearch, and all options listed in the [TFX Tuner documentation](https://www.tensorflow.org/tfx/guide/tuner).
The result is always the same: the job eventually fails with a grpc._channel._InactiveRpcError caused by a Deadline Exceeded error.

As a workaround, I had to force Keras tuning to run locally (so results are written to /tmp) and then manually copy results to GCS afterwards.

---

Describe the expected behavior

The Tuner component should successfully launch a Vertex AI hyperparameter tuning job (CustomJob/Study) and report the results back to the pipeline, instead of failing with a timeout.

---

Standalone code to reproduce the issue

A minimal pipeline containing a Tuner component configured to run on Vertex AI.
(tuner_fn similar to the official TFX [Tuner example](https://www.tensorflow.org/tfx/guide/tuner)).

tuner = Tuner(
    module_file=module_file,
    examples=example_gen.outputs['examples'],
    train_args=trainer_pb2.TrainArgs(num_steps=1000),
    eval_args=trainer_pb2.EvalArgs(num_steps=200),
)

The component runs fine locally, but fails when orchestrated on Vertex AI Pipelines.

---

Other info / logs

Full traceback from Vertex logs:

Traceback (most recent call last):
  ...
  File "/opt/conda/lib/python3.10/site-packages/keras_tuner/src/distribute/oracle_client.py", line 81, in create_trial
    response = self.stub.CreateTrial(
  ...
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2025-09-12T08:10:49.033761848+00:00"}"

[pritamdodeja]

@Wolff-Lucas are you using the TFX tuner or the one in extensions for Vertex AI? I believe there are two versions. I had contributed an example some time ago that might be of use:

#6953

[Wolff-Lucas]

Yes i'm using the extension. I started from your example elsewhere. I think that my problem could also come from the fact that I am using a custom TFX image where I add my packages or from a permissions problem that is not correctly reported in the errors. These are only assumptions.

[pritamdodeja]

One debugging step I'd recommend is see if your trainer component is working, because that's a simpler situation that probably uses the same tfx container image. I have seen problems like this in other settings where the python sdk harness is not available for the runner to send beam DoFn related work to, although those are situations where the Executor is powered by Beam. There's also some debugging features in Vertex that might be helpful that enable you to get inside containers etc. and look around. Best of luck!

[Wolff-Lucas]

Yes my trainer is working and it is using the same image. I'll continue to use the local Keras Runner version for now and come back to it later to investigate further. Thanks for your help.