Flight DoGet multiple batch DictionaryArray deltas

[jskowronski-factset]

I have a FlightServer written in c++ using subclasses of flight::RecordBatchStream and arrow::RecordBatchReader. For efficiency, I'm trying to convert some StringArray fields to DictionaryArrays. It all seems to work if all dictionary values are known with the first RecordBatch streamed back, but if new values occur in subsequent RecordBatches the client doesn't see them. As far as I can tell the client only ever sees the dictionary associated with the first RecordBatch streamed back. I checked both c++ and python clients.

It seems like the IPC protocol itself supports this, but flight doesn't seem to support it automatically, and I don't see any obvious way to enable it.

Does anybody know if this is even possible, and if yes how I can accomplish it? I already tried using the same Dictionary32Builder between batches, but it didn't help.

[lidavidm]

Which Arrow version are you using? Dictionary replacement is available: #45056

[jskowronski-factset]

It looks like we're still on 21.0.0 and that was added to 22.0.0. I'll have to discuss upgrading with the team. Thank you for pointing that out.

[raulcd]

As a minor note the Arrow 22.0.0 release is currently on-going. The feature freeze has been done and we are still solving some issues. The PRs pointed there will be available once we release.

[lidavidm]

Ah my bad, I had thought 22 was out for some reason 😅 please bear with us!