Lessons from the Trenches in a LLM Frontier: An Engineer’s Perspective - Apidays Australia 2024

I, along with my colleagues Jason Goodsell and Juan Burckhardt, had the opportunity to present our key insights and learnings from the rapidly evolving world of Large Language Models (LLMs) at Apidays Australia 2024 in October. The talk, titled “Lessons from the Trenches in a LLM Frontier: An Engineer’s Perspective,” shared our experiences from the front lines of developing LLM-powered solutions. Our team has been deeply immersed in creating and integrating LLM solutions, observing firsthand the industry’s intense focus and the eagerness of engineering teams to incorporate this technology into their products. This often involves developing “Copilot-like” features to augment user workflows through natural language interaction. The drive to innovate with LLMs is immense, especially with the technology becoming more accessible beyond big tech corporations. However, this rapid adoption brings challenges. While the potential is huge, the risks of failed integrations can be significant, leading to increased caution. Furthermore, the rush to build can sometimes mean critical aspects for robust, production-ready systems are overlooked. Many online guides that promise quick expertise often don’t cover these advanced but crucial topics. In our talk, we aimed to provide an engineer’s viewpoint, developed from collaborating within a multi-disciplinary team that includes data scientists. We focused on practical considerations that teams might want to adopt, especially concerning content safety, compliance, preventing misuse, ensuring accuracy, and maintaining security – all vital for successful and responsible LLM deployment. The video of our presentation is available on YouTube, and the slides can be found on Speaker Deck: Video of the talk: Apidays Australia 2024 - Lessons from the Trenches in a LLM Frontier: Engineer’s Perspective. Slides: Lessons from the Trenches in a LLM Frontier: An Engineer’s Perspective on Speaker Deck The talk abstract is as follows: For the past year or so, our industry has been intensely focused on large language models (LLMs), with numerous engineering teams eager to integrate them into their offerings. A trending approach involves developing features like “Copilot” that augment current user interaction workflows. Often, these integrations allow users to engage with a product’s features through natural language by utilizing an LLM. However, when such integrations fail, it can be an epic disaster that draws considerable attention. Consequently, companies have become more prudent about these risks, yet they also strive to keep pace with AI advancements. While big tech corporations possess the infrastructure to develop these systems, there’s a notable movement towards wider access to this technology, enabling smaller teams to embark on building them without extensive knowledge or experience, potentially overlooking critical aspects in the rapid development landscape. Most online guides that promise quick expertise typically fail to account for these advanced topics. For robust production deployment, issues such as content safety, compliance, prevention of misuse, accuracy, and security are crucial. Having spent significant time developing LLM solutions with my team, we’ve gathered key insights from our practical experience. I intend to offer my point of view as an engineer collaborating with data scientists within a multi-disciplinary team about certain factors your teams may consider adopting. Recording Slide Deck If you have any thoughts or comments please leave them here. Thanks for taking the time to read this post.