Currently available in beta, the Vertex AI SDK for Firebase enables the creation of apps that go beyond the simple chat model and text prompting. Google has just made available a colab to help developers through the steps required to integrate it into their apps.
The new colab covers several key topics, including prompt design, how to set up a Firebase project to use Vertex AI, configure your Android Studio project, and integrate it into your code using Kotlin.
To build design effective prompts, explains Google engineer Thomas Ezan, developers can use Vertex AI Studio, a Cloud-based tool aimed at rapidly prototyping and testing prompts with Gemini models.
A feature of Vertex AI that promises to make it easier for developers to specialize the behavior of their apps is System instructions.
System instructions serve as a "preamble" that you incorporate before the user prompt. This enables shaping the model's behavior to align with your specific requirements and scenarios.
Using system instructions developers can predefine once and for all the desired output style or tone, a persona or role (e.g, “explain like I am 5”), the goals or rules for the task (e.g, “return a code snippet without further explanation”), and any additional context that is relevant to the app user.
System instructions are set at initialization time, as shown in the following snippet:
val generativeModel = Firebase.vertexAI.generativeModel(
modelName = "gemini-1.5-flash",
...
systemInstruction =
content { text("You are a knowledgeable tutor. Answer the questions using the socratic tutoring method.") }
)
The SDK also allows developers to specify a responseMimeType for the generated output. This can be useful, for example, when generating JSON output in order to exclude any undesired non-JSON content.
Integrating the Gemini API into a mobile app, says Ezan, is not just limited to providing conversational interfaces, thanks to Gemini multi-modal capabilities. Indeed, Gemini can process diverse input besides text, including images, audio, and video. This means, for example, you can generate captions for images, summarize audio files, describe scenes from videos and so on.
Another powerful feature that Ezan highlights is the possibility of creating functions to extend the capabilities of the model. For example, a function could fetch some data from an SQL database and add it to the prompt context. Alternatively, you could define a series of tools that the model can use to generate its output, as it is shown in the following diagram.
Here, the model uses two calls into your data to retrieve the date and the order list to answer a question from a user.
All of these features are nicely supported by Vertex AI in Firebase, so developers can leverage them using their preferred programming language, including Swift, Kotlin, Flutter, and JavaScript.
The Vertex AI in Firebase SDK allows Android and iOS developers to access the Gemini API directly from their apps without requiring an intermediate backend service layer written in Python, Java, or Go.