OK, this is probably user error and a mistake on my part but I need some please…
I’m working on adding support for Gemini LLMs to AIKit and have hit an issue.
AIKit communicates with LLMs through various vendor APIs essentially using a URLConnection. Specifically the URLConnection makes a GET request to the API and expects a streaming response of JSON. For Anthropic, OpenAI and Ollama I had been using the URLConnection.ReceivingProgressed event to handle the streaming.
The structure of the data received various by vendor but typically it was in a format similar to:
data: {SOME VALID JSON}
data: {MORE JSON}
I would split the string in the ReceivingProgressed event by a delimited and handle each individual JSON string separately. No issues.
With the Gemini API however, what seems to be happening is the string passed into the ReceivingProgressed event is a carved up JSON string. The first time the event fires I might get the opening part of the JSON and then the next time the event fires it’s a continuation of the JSON object and then maybe the start of another.
I don’t know how best to handle this - does anyone have a recommendation?
I considered using the ContentReceived event of the URLConnection class but if I do that I essentially lose the ability to display streaming responses.
3 posts - 3 participants