How to accept HTTP/2 request and proxy it upstream as HTTP/1

[banool]

I have built a proxy making use of Hyper. Normally we run in GCP Cloud Run, which by default downgrades HTTP/2 requests to HTTP/1. This is nice because then the request we make to our upstreams is always HTTP/1, which is always supported.

For various reasons, we want to turn on end-to-end HTTP/2 now (ref), which means TLS is terminated and the request continues as h2c. This means that the proxy can receive either HTTP/2 or HTTP/1 requests. I would like to be able to set some configuration on a per upstream basis to declare whether the upstream can handle HTTP/2 requests. If not, when we receive a HTTP/2 request, we now need to handle proxying it upstream as HTTP/1, then proxying the response back to the client still as HTTP/2.

I was hoping that Hyper would automagically handle this, and indeed it seems like it should be able to. Unfortunately I can't figure out the correct way to build the client to make this happen. First off, my dependencies:

axum = { version = "0.7.4", features = ["http2", "ws"] } 
http = "1.1.0"
http-body = "1.0.0"
http-body-util = "0.1.1"
hyper = { version = "1.2.0" }
hyper-tls = { version = "0.6.0" }
hyper-util = { version = "0.1.3", features = ["client-legacy"] }

Currently I'm doing this:

use axum::body::{Body, Bytes};
use http_body_util::BodyExt;
use hyper_util::{
    client::legacy::{Builder, Client, connect::HttpConnector},
    rt::TokioExecutor,
}

pub type HyperClient = Client<hyper_tls::HttpsConnector<HttpConnector>, axum::body::Body>;

/// Opionated function to return a client with a https connector.
pub fn build_http_client() -> HyperClient {
    Client::builder(TokioExecutor::new())
        .build::<_, axum::body::Body>(hyper_tls::HttpsConnector::new())
}

When I receive a HTTP/2 request, no matter whether trying to proxy to a HTTP/1 and HTTP/2 or just HTTP/1 compatible upstream, if I try to proxy it with this client I get this error:

Connection is HTTP/1, but request requires HTTP/2

This led me to this issue. I added the alpn feature to hyper-tls but it didn't help. I see from looking at the code that this only affects the https:// path, but in testing the upstream is just https://. I see how this makes sense, since ALPN is specifically related to TLS. But unfortunately this means it didn't solve my problem.

(Though perhaps I can tolerate the restriction that if the request is HTTP/2, the upstream must be https. I'll need to do more testing to see if everything works with this restriction).

Next I tried to build the client with http2 only:

pub fn build_http_client() -> HyperClient {
    Client::builder(TokioExecutor::new())
        .http2_only(true)
        .build::<_, axum::body::Body>(hyper_tls::HttpsConnector::new())

After this, HTTP/2 requests to upstreams that support HTTP/2 work, but obviously this doesn't work with HTTP/1 upstreams, nor can I proxy HTTP/1 requests anymore.

At this point I'm at a bit of a loss. There is another option entirely, in which we'd run two HTTP gateways in Cloud Run, one that can handle HTTP/2 directly and one where Cloud Run downgrades requests to HTTP/1 for us. We then configure routing rules in the GLB that makes it that requests for upstreams that we know can handle HTTP/2 go to the HTTP/2 gateway, otherwise they go to the HTTP/1 gateway. But I'd love to get this all to work with a single gateway.

So in short, is it possible for me to configure a client / multiple clients / set flags on a per request basis such that if we receive a HTTP/2 request, and I know the upstream can't handle HTTP/2, it gets proxied upstream as HTTP/1?

[seanmonstar]

Can you change the version, like *req.version_mut() = Version::HTTP_11, before passing it on the to client?

[banool]

I actually considered that, but thought it'd surely never be so easy hahah. I'll try it out and report back.

[banool]

Remarkably that actually works! If you receive a HTTP/2 request you can downgrade it to HTTP/1 like this:

if !state.get_can_handle_h2c() && request.version() == http::Version::HTTP_2 {
    *request.version_mut() = http::Version::HTTP_11;
}

Then if you send a request as HTTP/2, it'll get proxied upstream as HTTP/1 and the original client will be none the wiser:

$ curl -v --http2-prior-knowledge http://localhost:9898/v1
* [HTTP/2] [1] OPENED stream for http://localhost:9898/v1
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: http]
* [HTTP/2] [1] [:authority: localhost:9898]
* [HTTP/2] [1] [:path: /v1]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /v1 HTTP/2
>
* Request completely sent off
< HTTP/2 200
* Connection #0 to host localhost left intact
<happy response>

Beautifully written library, I did not expect this magic to work, kudos @seanmonstar.

This leads to a different question regarding h2c, but I'll start a fresh discussion for that to keep this clean.

[banool]

Here's that other question: #3957.