ScyllaDB as a backend for LangChain addresses a key issue: preserving dialogue context in RAG systems and scaling without losing state.
The problem does not manifest immediately — until the moment the RAG chatbot exceeds a single process. LLMs are inherently stateless: each request begins with an empty context unless the history is manually reproduced. In LangChain, this is partially hidden behind BaseChatMessageHistory, but the standard in-memory implementation disappears upon restart. Once restarts, horizontal scaling, or multiple replicas occur, the system loses dialogue consistency. As a result, responses become fragmented, and behavior becomes unpredictable.
RAG itself does not close this gap. It adds external context through retrieval but does not store user interactions. These are two different axes: knowledge (documents) and state (conversation state). Without a persistent storage layer, the dialogue history does not survive failures and is not shared between instances. For production scenarios, this limitation becomes critical.
The solution revolves around replacing ephemeral memory with persistent storage. In this case, ScyllaDB was chosen — a NoSQL database optimized for high throughput and predictable low latency. The key trade-off here is the abandonment of the simplicity of in-memory in favor of network storage. This adds a network dependency and requires a data schema, but in return provides fault tolerance, scalability, and the ability to audit dialogues. In the industry, such a shift is considered pragmatic: state is moved to the infrastructure layer.
The integration relies on the existing Cassandra connector, which ScyllaDB can reuse. Through cassio.init(), a session and keyspace are registered, after which all LangChain components use a single connection without additional configuration. Importantly, cassio is not tied to a specific DBMS: it works with any compatible session, including scylla-driver. This reduces integration costs and eliminates the need to write a custom adapter.
The architecture is divided into two streams: vector storage and message history. For RAG, a table rag_docs is used with a column vector<float, 384>, corresponding to the embeddings of the model all-MiniLM-L6-v2. Indexing is implemented through HNSW, allowing for nearest neighbor searches upon request. Documents are loaded via WebBaseLoader, then chunked into 500-token segments with a 50-token overlap. This reduces the risk of losing semantics at the boundaries.
The second stream is chat history. CassandraChatMessageHistory saves each message as a string with the key session_id. This is a simple but effective model: one session_id — one logical session. All replicas working with the same key see the same history. Changing the session_id creates a new session without additional logic. Thus, the mechanism for isolation and state recovery boils down to key management.
RunnableWithMessageHistory closes the loop. Before each call, it pulls the history from ScyllaDB, and afterward, it writes the new turn back. Both operations go through the database, making the system’s behavior deterministic concerning state. Even in the event of failures or restarts, the chain continues from the same point.
A separate aspect is the cost of computations. Embeddings are calculated once during the initial data load. Subsequent runs use the already created table unless re-indexing is triggered. This reduces the load and makes the system more predictable in terms of resources.
The result is a resilient RAG chatbot that maintains context between calls and scales horizontally. Performance metrics are not provided in the original material, but architecturally, the system eliminates key points of failure: loss of state during restarts and context divergence between replicas. Additionally, the ability to audit dialogues emerges, which is important for production and compliance.
This approach can be seen as an evolutionary improvement to the LangChain ecosystem. It does not change the interaction model with the LLM itself but stabilizes its behavior under real-world conditions. In systems where dialogue is not a one-off request but a long session, moving memory to ScyllaDB becomes a necessity rather than an optimization.