Day 9 | 👻 The Three Ghosts Haunting Your AI This Holiday Season
Overview
In this video, we explore the three ‘ghosts’ haunting AI in production: the model layer, knowledge layer, and user layer.
The model layer is affected by unexpected updates from AI providers, causing abrupt changes in prompt behavior. The knowledge layer deals with data that becomes outdated if not constantly updated. The user layer evolves as users become more familiar and start asking more complex questions over time.
To combat these issues, AI observability tools such as evals and session replay can provide essential real-time visibility across all layers, preventing customer complaints, viral tweets, and potential lawsuits.
