Mandy Gu spoke at QCon SF 2024 about how Wealthsimple, a Canadian fintech company, uses Generative AI to improve productivity. Her talk focused on the development and evolution of their GenAI tool suite and how Wealthsimple crossed the "Trough of Disillusionment" to achieve productivity.
Gu, a senior software development manager at Wealthsimple, highlighted the company's LLM journey over the past two years. Their main effort was building an LLM gateway for their employees to use, which was motivated by the need for "guardrails", especially for preventing the leakage of personally identifiable information (PII). Gu called out a high-profile example of Samsung's employees accidentally leaking sensitive data when using ChatGPT. Wealthsimple's initial launch of the gateway was a proxy for commercial LLMs like ChatGPT, with an audit trail to track what data was sent and by whom.
To drive adoption, Wealthsimple used a combination of carrots and sticks. The carrots included features that made life easier for developers: the gateway implemented retry logic for the LLM APIs, and Wealthsimple was able to get their collective API rate-limits increased; Wealthsimple also paid the commercial API fees, making it free for their employees to use. Sticks included a "nudge" feature that would automatically send Slack notifications to users who bypassed the gateway.
Over the course of 2023, the team added several features. First, they added automated PII redaction, to reduce the potential for leaking sensitive info. They added self-hosting for smaller, open-source LLMs such as Llama and Mistral as well as for OpenAI's open-source speech recognition model Whisper. They also added retrieval augmented generation (RAG) by importing the company's knowledge bases into a vector DB. The team wrapped up 2023 by shipping a tool called Boosterpack, a personal assistant for Wealthsimple employees, which supported personalized private knowledge bases as well as shared company-wide ones.
At the beginning of 2024, Gu said that the Wealthsimple team felt that they had reached a low point in the LLM "hype cycle." They realized not all their bets had paid off and began to rethink their strategy, "focusing a lot more on the business alignment." Whereas in 2023 they spent a lot of time trying to support the latest state-of-the-art model, in 2024 they decided to focus on higher-level trends.
This led to several changes in their tooling. First, they removed the "nudge" feature, since they found it affected user behavior very little. They added multi-modal support, allowing users to upload PDFs and images when chatting with LLMs. They also re-evaluated their build-vs-buy strategy and began to adopt Amazon Bedrock for several of their features.
Gu wrapped up her talk with some lessons learned. She shared some metrics on the categories of LLM use at Wealthsimple: about 40% of the usage is for code generation, 30% for content generation, and 27% for information retrieval. About 80% of LLM usage does use the company's LLM gateway, and about 50% of employees use it every month. Most importantly, they found that users tended to prefer having a single tool, and tools are "most valuable when injected in the places [they] do work." They also found that "almost everyone" who used their tools reported that it improved their productivity.
