Relevant Past Papers

How is it done today, and what are the limits of current practice?

Human-assisted Robotic Policy Refinement via Action Preference Optimization

Link: https://openreview.net/forum?id=dlQ1iUpQNf

  • Summary: It showcases the simulation and real-world scenarios of Vision Language Action (VPA) models to prove superior generalization and robustness of our human-assisted framework across a variety of manipulation tasks through Action Preference Optimization (APO) models.

  • Gap: the model requires active human involvement during deployment due to failure to grasp an object, identify errors, and regrasp. The pipeline will be in a loop which requires correct signals and expensive simulation.

Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs

Link: https://openreview.net/forum?id=956KYtqwcU

  • Summary: They introduce CorrectBench to compile Chain of Thoughts to demonstrate LLM evaluation, efficiency, and accuracy by modeling mathematical reasoning, commonsense reasoning, and code generation.

  • Gap: It focuses only on language-only tasks and does not address physical action space. It needs self-correction in robotics models and settings, where spatial and physical consequences have not been studied.

Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones

Link: https://openreview.net/forum?id=1t4hR9JCcS

  • Summary: They illustrate the performing command of LLM exhibit systematic failures through parentheses generation as structural tasks to express the internal mechanism errors, and introduces RASteer - a ranking-and-steering method for LLM to attention heads and feed-forward neurons to correct the failures at near -100% accuracy.

  • Gap: The RASteer model requires synthetic data that inflate results and only uses simple hypotheses. It also depends on data-driven correction and offers no mechanistic understanding of how the model detects whether an error occurred without empirical tuning.

From Virtual to Real: A Framework for Verbal Interaction with Robots

Link: https://aclanthology.org/W19-1603/

  • Summary: This paper demonstrates the experiment through ACL’s SpLU-RoboNLP workshop as a simulation of 3D-physics based virtual environment to develop, test, and validate language model to robots before implementing into hardware with natural language reference, robot command execution, perceptual memory, and factual knowledge.

  • Result: The primary metric is Competency - the percentage of user utterances that the system responded to meaningfully - which was around 69%, and the parser achieved more than 95% precision on well-formed robot commands.

  • Gap: The simulation was completed but remained unsolved since it was virtual and none of them were reliable to implement in the real world. The metric shows that there is still a 31% chance that it could not respond meaningfully or have failures in language application and vocabularies. Potentially, the results through experimental products and knowledge were still lacking LLM understanding and having no formal evaluation benchmark, leaving the product to have no statistically significant conclusions and agreements.

Back to the Blocks World: Learning New Actions through Situated Human-Robot Dialogue

Link: https://aclanthology.org/W14-4313/

  • Summary: The paper addresses the classic problem of robotics learning an action using natural language without programming knowledge through the setting of a tabletop with colored blocks and a SCHUNK robotics arm, where its key technique is through continuous planner, discrete planner, and high-level action knowledge.

  • Gap: The model contains a small sample size to examine, and the setting of the simulation is limited, leaving the solution of the vision system partially sidestepped. Furthermore, we cannot recover execution failures due to the fact that it is purely based on NLP and no learning.

Error Reflection Prompting: Can Large Language Models Successfully Understand Errors?

Link: https://aclanthology.org/2025.insights-1.15/

  • Summary: The researcher proposes an Error Reflection Prompting (ERP) model where they extended the CoT model to explain more vividly the incorrect solutions alongside where and why they are wrong and then provide the correct solution, making the motivation of the model quickly adapt to common mistakes and resolve the problem itself through its generated model.

  • Gap: ERP works way better with GPT-3.5 than GPT-4o since it contains a strong model, leading to less room for ERP to support. This requires a power mechanism when the baseline modeling is at stake. It also suffers from overfitting to a particular dataset due to the prompt to be tailored towards commonsense or arithmetic reasoning. It is also limited in token cost which would frequently increase its cost due to the CoT model rapidly used.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Link: https://arxiv.org/pdf/2201.11903

  • Summary: This paper introduces the Chain-of-Thought (CoT) Prompting and Reasoning, where it scans the question and prompts a solution through reasoning steps, explains as it understands the question and provides an answer. 

  • Gap: The model only works on massive scales since its limit is approximately 100B+ parameter models, making it inaccessible in most cases for running large models. The model was written by authors which showed the robustness of annotators, but the process is still subjective, and a limited error analysis of 50 correct and 50 incorrect examined tasks.

Motivation

What limitation or problem are you solving and how do you know it exists?

  • The VLA model is the approach that could potentially enhance the control and optimization of robotics, but it has not been a success in physical tasks or handling commands from humans and AI/LLM. Throughout the papers, there has been a trend of approach to correction has been the APO method, and such setbacks require human knowledge to label which actions were preferred, which can be expensive and become a restriction to resolve at scale.

  • Within recent times, LLMs have been showing correction and improvement in understanding general knowledge of language domains through ERP or CorrectBench, and the RASteer method was seen as a failure that steered even within a model’s internal computation.

  • However, would there be a self-correction that can be transferred to a physical domain model in VLA models through adjusted simulation and appropriate human controls to recreate correction itself rather than a human?

Why is this limitation important?

  • As a big step to connect LLM with hardware models and robotics control, we have seen the appearance of the current VLA model which cannot perform self-correction without human supervision. As a result, autonomous and robots would still be never deployed in a real-world environment. Based on multiple flaws and mistakes with relevant connections, such limitations can be acknowledged and resolved to recover from failure without human intervention.

  • Best case: If it works, then the result would display as significant as ERP errors validated on language tasks (commonsense, arithmetic, code). Demonstrating this result from the simulation oracle transfer to physical action spaces in a VLA model would establish a new pattern/space for autonomous robotics action. Thus, the common solution of robotics autonomy can be applied in the real world.

  • Worst case: If it doesn’t work, then the result will explicitly show to us that the self-correction model does not naively generalize to embodied physical settings. The flaw could potentially be missing state grounding, simulation feedback, or a fundamentally different interface between LLM reasoning and token generation. As a result, the setback of this failure would directly come from the limit of LLM self-correction.

  • Why does your idea solve it?

  • This idea would work and require no human control/data, as the simulation oracle fully replaces the human labeler with APO. The APO will run a physical environment that provides success/failure signals and state variables in the simulation. Using the simulation-based signal, we apply the VLA model’s language to the ERP-style error reflection, the previous flaw, to generate a sufficient preference label. This closes the human-in-the-loop bottleneck while preserving the APO training framework.

  • The only dataset we would rely on would be the human-assisted APO from the original APO paper [1] as a means of result comparison, without interaction with human annotators. 

  • Why would your idea probably work?

  • Every component is validated individually through its own limitations. For instance, APO works for VLA fine-tuning, ERP shows that LLM can reason about errors and perform solutions with sufficient structures and simulation. Furthermore, the robot-language grounding work can be driven into physical action. The approach is connecting these components into a loop as such: 

Simulation -> ERP and LLM diagnosis -> preference label -> APO fine-tuning. 

Key Ideas/Contributions/Novelty

  • What we have not done before is running the simulation without APO’s labeling loop from humans and replacing it with physics movement of the virtual model through success and failure with structured description.

  • We would like to extend ERP failure diagnosis to language tasks that could apply to robots failure scenarios. At the same time, using the VLA model to identify where and why the action or sequence failed, then learn the mistake as a structured error description to correctly perform an action.

  • One thing to acknowledge is the self-correction error diagnosis through ERP. It should compare failed and correct attempts of an action through a loop and feed into APO fine-tuning iteratively, teaching the model to recover from its previous failure description.

  • Contribution: provide a method alongside empirical evaluation.

  • Research question: Can we perform a simulation of LLM self-correction that matches or approaches the human-assisted APO performance as a manipulation benchmark without human labels?

Methods

  • VLA Model -> Simulation -> LLM failure diagnosis (ERP diagnosis) -> Preference label generation -> APO fine-tuning

  • VLA model executes an action in simulation at start

  • Simulation oracle returns success/failure with state variables

  • If Result == Failure, then LLM applies ERP-style diagnosis

  • Preference label generated (failed vs. corrected trajectory/attempt)

  • APO fine-tuning updates the policy

  • The loop continues until the simulation returns success (Result == Success), completing the iteration and creating the task to be complete

Experimental Setup

  • Start with human-APO as the baseline to train the non fine-tuning VLA (OpenVLA) through a manipulation task with human-labeled preferences.

  • Then, we proceed with the simulation by replacing the human labels with a robot simulation. Within the loop, after each rollout, the simulation returns the success/failure state and the state variables, which includes the object position, robot contact, and error.

  • The ERP diagnosis will be simulated in parallel alone as a signal to feed the simulation and the VLA model’s language. The model will be prompted with the description: “The robot attempted [task]. Result: [success/failure]. Robot contact: [A]. Object position: [B]”. Thus, it will operate and identify what went wrong and how to resolve it whenever the model ran into a structure failure description.

  • Compare between failed and correct attempts, label the preferred action, and deliver it to the APO fine-tuning.

  • The iteration repeats over multiple rounds -> rollout -> diagnosis -> label -> fine-tune. The iteration ends after a fixed number of iterations.

  • Models and tools:

  • VLA: non fine-tuning VLA (OpenVLA). This supports LoRA fine-tuning

  • Simulation: RoboSuite (robot simulation and manipulation benchmark)

  • Tasks: Pick-and-place, nut assembly — chosen for clean binary success signals

  • Task: grab an item, move and place it in a different position. Several actions to consider such as nut/bolts/gears assembly, grasp an item for a period of time, adjust its position from left to right or high to low. This checks for clean binary success/failure signals.

  • Ablation:

  • VLA without fine-tuning: Does the base model self-correct without any fine-tuning?

  • No LLM and ERP diagnosis: Does raw success/failure suffice without ERP-style reasoning?

  • Human-APO baseline: How close does our method get to human-labeled preference data?

Datasets and Evaluation

  • RoboSuite: robot simulation and manipulation benchmark 

  • Human-APO preference dataset as a baseline comparison of success/failure

  • Primary metric: Competency. Rate a task success through N rollouts in the simulation; failure-recovery rate from the set of failure scenarios

  • Secondary metric: numbers of human labels required

Benchmarks/Evaluation Sets

  • Human-assisted APO - the main paper to research and analyze human-APO model to understand physical actions for robots to learn

  • Non VLA fine-tuning (OpenVLA) - the simulation without fine-tuning

  • APO Simulation (without ERP diagnosis) - the simulation of robots without ERP diagnosis to run its learning model through CoT, and ERP runs individually as trials and errors

Ideal Results

  • Best case scenario: when we complete the experiment loop, our simulation-driven method reaches around 5-10% of human-APO success rate via manipulation with no human labels. Meanwhile, the failure-recovery rate from the previous set can be utilized to improve significantly over the baseline modeling. Furthermore, without ERP diagnosis, the result demonstrates almost no LLM-based errors, adding measurable value beyond raw binary feedback.

  • Hypothesis: a VLA model when fine-tuned with simulation ground and ERP preference labels will show higher task success and failure-recovery rates than the baseline, and will approach the human-APO success performance throughout the simulation iteration. The ERP performance will contribute to the simulation alone.

Potential Limitations

  • Computation limits, generalization limits, dataset limits, ethical limits?

  • Simulation to real limitation: results validated through simulation may be limited to approach in real life due to cost and not transferable to physical robots. Therefore, it can only be through simulation.

  • ERP diagnosis quality: depending on the CoT prompt and multiple tokenization within the simulation, VLA language models may produce inaccurate diagnosis which can result in insufficient solution or failure modes onto the physical robot. This relates to the failure rate from the ERP performance in the simulation signal alone.

  • Token cost: As mentioned in the ERP token cost, error prompting may significantly increase token usage per step.

  • The experiment is a simulation, so there is no accurate result for implementing in the real world, without sensitive data.

**