Making LLMs “learn” from the past mistakes: EMNLP 2023 Overview Part 2

The Large Language Models (LLMs) has traditionally been a statically learned models, which would often falter in their capacity to learn and adapt from past errors by themselves. Some of the works (as briefly discussed in this blog) at EMNLP conference propose mechanisms that not only allow LLMs to recognize and correct their mistakes but also to internalize these corrections for future interactions.

“Cooperative Study Assistant for LLMs” and “Enhancing LLMs through Rule Accumulation” present methods focusing on dynamic learning from past interactions and mistakes, akin to mirroring human learning capabilities.

Learning from Mistakes via Cooperative Study Assistant for Large Language Models

Problem Statement:

The paper addresses two main limitations of existing self-reflection methods in LLMs: reliance on the correctness of guidance, and the absence of an efficient mechanism to learn from mistakes.

Solution Blueprint:

This framework is inspired by the methods human study assistants use to support students, by identifying common errors and providing guidance.

• Study Assistant for Large Language Model — SALAM, a framework, includes an auxiliary agent that aids the main LLM in learning from mistakes.
• It works in two phases: a ‘gathering phase’ wherein the student assistant agent probes the main LLM, analyzes its errors, and collects the interaction in a mistake memory, and an ‘examination phase’ where the study assistant provides guidelines by retrieving relevant cases to help the main LLM anticipate and avoid similar errors.
• A general study assistant is initially utilized and then customized to provide LLM specific guidance through imitation learning from successful guidance experiences.

Key Results:

SALAM showcased substantial improvements in LLM performance, with accuracy margins of up to 6.6% on BBH and 12.6% on BBQ frameworks​​, which evaluate two crucial aspects of LLMs reasoning ability and potential social bias.

Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation

Problem Statement:

This research focuses on overcoming the challenge of frozen LLMs’ repeating of similar mistakes by proposing a “tuning-free” approach i.e, LLM weights itself is not updated in the process and hence can be used in a deployment setup.

Solution Blueprint:

Tuning-free Rule Accumulation — TRAN, is a tuning-free framework that enhances the self-alignment capabilities of LLMs in a streaming setting, without additional training data. TRAN utilizes an iterative process of generating and accumulating rules based on observed mistakes, enabling LLMs to avoid repeating similar mistakes.

• The TRAN framework accumulates rules from incorrect cases, gradually forming a rule collection. Process for rule-collection:
• (a) generate rules based on the current mistake; (b) test and preserve rules that are successful; (c) include the mistake in a collection of mistakes when no successful rule is found; (d) extract related mistakes from the mistake collection; (e) formulate rules by analyzing both current and previous mistakes; (f) add the newly effective rules to the rule collection.
• These rules are then applied to new inputs, enabling LLMs to avoid repeating past mistakes.
• A key remark of TRAN is that these rules are independent of primary prompts thus complementing prompt design strategies like Chain-ofThought.

Key Results:

TRAN achieved a significant increase in accuracy, reaching about 91.6% in zero-shot settings on BBQ-Lite, and outperforming Zero-Shot CoT by 6.3%. It also demonstrated a performance boost of 8.8% over SALAM, the method outlined just above​​.

You can connect with me at LinkedIn