Knowledge- and memory-based interactive task learning for human robot collaboration

1 Introduction

Robots that autonomously and safely support users in household environments remain a long-standing goal in robotics research. Over the past decades, household robots—such as vacuum cleaners, lawnmowers, and voice assistants—became increasingly cost-efficient and widely adopted. However, we are yet to see a robot capable of preparing any given meal in any kitchen, adapting dynamically to different environments, tools, and ingredients. The challenge lies not only in executing predefined tasks but in enabling robots to generalize their actions across diverse contexts and objects.

A prominent approach to addressing this challenge is imitation learning, where robots learn new skills by observing and replicating human demonstrations, often through videos or Virtual Reality (VR) simulations. Unlike traditional programming paradigms that rely on explicit, hard-coded instructions, imitation learning allows agents to acquire complex behaviors without manual intervention or exhaustive trial-and-error processes. Policies derived from imitation learning typically focus on learning either task-relevant motion trajectories (Schaal, 1999) or task-specific parameters (DelPreto et al., 2020; Hamon, 2011) from human task demonstrations. Despite its potential, imitation learning faces significant limitations in dynamic and unstructured environments. Household tasks require real-time adjustments to parameters such as angle, velocity, and stopping conditions. Traditional imitation learning methods struggle with task generalization. They replicate observed motions without understanding the underlying task semantics. For example, a robot trained to cut a cucumber may fail when asked to quarter an apple, as it lacks the semantic knowledge to infer task variations.

By observing how children learn tasks, we notice that they do not simply imitate observed movements. Instead, they form structured, experience-based memory representations that connect actions, objects, goals, and contextual constraints, allowing them to grasp the essence of a task and refine their behavior over time (Laird et al., 2017). Inspired by this perspective, we argue that robotic task learning should extend beyond reproducing demonstrated trajectories toward constructing adaptable task representations grounded in both semantic knowledge and episodic experience.

Task learning goes beyond imitation by enabling robots not only to mimic actions but to comprehend their purpose and adapt task execution accordingly (Nicolescu and Mataric, 2005). It involves task understanding that is linked to memory structures, including demonstrations within their situational context, considering the roles of involved objects, and reasoning about actions to infer the required task parameters. For instance, a robot should recognize that quartering an apple involves a center cut followed by object rotations, while slicing a cucumber requires repeated end cuts. In this work, we define interactive task learning as the process through which a robot incrementally acquires, contextualizes, and refines task representations through situated interaction with humans and the environment.

We further define the essential task knowledge of an action as the set of semantically meaningful properties and relations required to reproduce and adapt a task across contexts. This includes object roles, affordances, action constraints, parameter dependencies, and expected task outcomes that remain invariant across individual demonstrations.

Based on this perspective, this work proposes a conceptual framework for interactive task learning that integrates Actionable Knowledge Graphs (AKGs) (Kümpel, 2024), structured representations linking objects, actions, and environments that contain essential task semantics (Kümpel, 2025), with Narrative-Enabled Episodic Memories (NEEMs) (Beetz et al., 2020), which provide structured and explainable episodic representations of interactions and experiences. The framework is visualized in Figure 1. We hypothesize that linking semantically grounded task knowledge with episodic interaction memories enables robots to construct adaptable and explainable task representations that support contextual task variation and interaction-driven refinement.

The proposed framework conceptualizes human-robot interaction as a continuous process of memory formation and refinement in which the robot can act both as a learner and as a teacher. Through interaction, NEEMs are created that capture task parameters, involved agents, objects, and contextual information. These episodic memories are linked to the essential task knowledge represented in the AKG, connecting objects and tasks through affordances, dispositions, and parameterized action relations. The resulting memory structures support embodied simulation (Haidu et al., 2014), allowing robots to reason about, narrate, explain, and execute task variations in context-sensitive ways.

The main contributions of this work are as follows:

• We propose a framework for interactive task learning based on NEEMs and AKGs.
• We outline how interaction-driven refinement of contextual task representations, parameterized through AKGs, facilitate task learning by linking task semantics to motion execution.
• We elaborate how robots can use our framework to perform task variations.

AKGs and NEEMs exist as separate concepts in prior work. The novelty of this work is their combination for interactive task learning. The proposed framework consists of three elements: first, a mechanism that creates NEEM entities with identifiers from the AKG, so that episodic observations and structured task knowledge are linked. Second, interaction loops in which the robot acts either as the learner or as the teacher and refines its task representation across interaction episodes. Third, a parameter-inference process that grounds the parameter ranges defined by the AKG in the concrete values observed in NEEMs. To illustrate the operationalization of the proposed framework, we outline two interaction scenarios in which the robot acts either as a learner or as a teacher: (a) with the iCub robot learning from observation and interaction of a human and (b) with the iCub as a teacher, giving feedback to the human. In (a), participants perform pouring tasks with different liquids and containers. Then they ask the iCub to perform a pouring task while giving feedback such as directions (“Pour there”), or corrections (“Not there, you will spill”). In (b) participants perform cutting tasks on various (wooden) fruits and the iCub verbalizes what it sees (e.g. “I see you taking the carrot and knife for cutting”) but also corrects and gives feedback (like guidance “Next you might want to cut the carrot”). Both study setups use NEEMs as experiment memories, which are linked to the AKG that classifies tasks (e.g., halving, slicing, or pouring), enabling robots to execute them using derived parameters. Since this is a hypothesis paper, we present the framework and two planned study designs; we do not report robot experiments or task-success measurements, which will be part of future works.

2 Related work

The field of imitation learning, also known as Learning from Demonstration (LfD), has been extensively explored to teach robots by leveraging human demonstrations. Within this domain, existing research focuses on two questions: What aspects of human behavior should be imitated? and How can robots effectively generalize what they learn from these demonstrations? While imitation learning excels at quickly training robots to replicate specific actions, its limitations become evident when robots face task variations or dynamic environments. Some approaches learn policies rather than motion trajectories. French et al. (2019) learn behavior trees from demonstration, capturing the structure of a task as a reactive policy rather than a fixed trajectory. Our work differs from these approaches by integrating a formal knowledge representation, which links task structure to object and action semantics.

This section focuses on two prominent branches of imitation learning: virtual reality (VR)-based task demonstrations and knowledge-based learning, highlighting the strengths and limitations, and positioning our approach, which integrates AKGs and NEEMs, as a solution for task generalization.

2.1 Learning from VR task demonstrations

Virtual reality (VR) environments have become a popular medium for robot training due to their intuitive interfaces, cost-effectiveness, and ability to safely simulate a variety of complex tasks. VR-based demonstrations allow humans to guide robots by providing motion trajectories and interaction cues, offering an effective way to teach task execution.

Kazhoyan et al. (2020) introduced a framework that translates human activities observed in VR into real-world robot actions, focusing on parameterizing mobile pick-and-place tasks. The work showcases how VR data can be mapped to robot motion, yet it remains limited to direct replication rather than task adaptation. Similarly, Hamon (2011) proposed an interactive approach that uses real-time simulations and neural networks to train robots through human-controlled virtual characters. While effective for motion imitation, their method struggles to extend learned tasks beyond the demonstrated context, as it lacks semantic reasoning about the task.

Kunze et al. (2013) and Walther-Franks et al. (2015) explored crowdsourcing task models via Games with a Purpose, using data gloves to capture human motions in virtual environments. These approaches creatively gather human input but also focus on imitating observed behaviors without abstracting task semantics.

More recently, DelPreto et al. (2020) proposed a master-apprentice model, where a human teleoperates a robot in VR and intervenes only when the robot fails autonomously. Although this method bridges imitation and reinforcement learning, it does not fully address task generalization, as the robot still relies heavily on reactive behavior rather than proactive task inference.

2.2 Knowledge-based learning

Knowledge-based learning integrates semantic knowledge into robot training, allowing agents to leverage structured information about tasks, objects, and environments. This approach mitigates the data inefficiency of imitation learning by reducing the need for exhaustive demonstrations and incorporating prior knowledge to guide robot actions.

Ramírez-Amaro et al. (2013) proposed extracting semantic rules from human demonstrations, allowing robots to infer the intent behind human actions. Although their approach aids in interpreting motion goals, it does not dynamically link these rules to task parameters for real-time adaptation.

Lopes et al. (2007) employed an affordances model, demonstrating that object properties can guide robot decisions, reducing learning complexity. Similarly, Koppula and Saxena (2016) used an anticipatory temporal conditional random field (ATCRF) to model spatial-temporal relationships between objects and human activities. While these methods incorporate semantics, they primarily focus on predicting future actions rather than adapting current tasks based on real-time conditions.

Recent advancements blend knowledge representation with imitation learning. Geng et al. (2023) combined affordance learning with reinforcement learning (RL), using contact frequencies to represent object affordances. Wang et al. (2021) extended Generative Adversarial Imitation Learning (GAIL) by integrating knowledge graph reasoning, but their focus remained on question-answering rather than dynamic task execution.

Zheng et al. (2024) introduced a Recurrent Graph Convolutional Network (RGCN) for multi-task planning, leveraging knowledge graphs to decompose tasks into reusable subtasks. Although promising, their method primarily addresses task planning, not the real-time adaptation of robot actions during execution.

3 Knowledge- and memory-based interactive task learning

While VR-based and knowledge-based approaches have advanced robot learning, they often fall short in enabling robots to generalize learned tasks. VR methods capture motion trajectories but lack semantic depth, while knowledge-based methods focus on task planning without dynamic adaptation during execution.

Our work bridges this gap by integrating AKGs with memory-based task learning using NEEMs. Unlike static knowledge graphs, AKGs link objects, actions, and environmental contexts (Kümpel, 2025) with task parameters. This allows robots to distinguish between task variations (e.g., slicing vs. quartering) by inferring motion parameters like cutting positions and repetitions, but also dynamically adapt actions in real-time using semantic enrichment of episodic memories. Beyond demonstrations, AKGs enable robots to infer additional necessary actions (e.g., peeling or core removal). Recent studies have shown that AKGs created from web-sourced knowledge can be used to parameterize robot action plans, facilitating flexible execution of meal preparation tasks like pouring and cutting (Kümpel et al., 2025; Kümpel et al., 2024). We refer to the linked object to action knowledge as essential task knowledge of an action. In contrast to low-level trajectories, essential task knowledge captures the abstract structure underlying a task in such a way that a robot can use it for action execution.

By semantically enriching episodic memories of human task demonstrations, i.e., augmenting episodic observations with structured task knowledge, AKGs can empower robots to move from mere imitation to proactive task execution. We hypothesis that this approach reduces reliance on exhaustive training data but also enhances task generalization—a crucial step toward autonomous household robots capable of adapting to new tasks and environments.

This section presents our methodology for integrating AKGs with episodic memories to enable task learning in robots. We describe how NEEMs that integrate essential task knowledge can be created and how this semantic information translates into task parameters for dynamic robot execution.

3.1 Narrative-enabled episodic memories (NEEMs)

Narrative-enabled episodic memories (NEEMs) provide structured logs of human task demonstrations, capturing both high-level actions and low-level sensory data. Unlike simple trajectory-based models, NEEMs record actions, object interactions, and goals, offering a richer understanding of tasks.

Following the socio-physical model of activities (SOMA) (Beßler et al., 2022), a NEEM consists of:

• NEEM-background: records the physical environment, object properties, and spatial configurations. For this, we reuse the concept of semantic digital twins (Kümpel et al., 2021) that combine the perception based scene graph of the robot with semantic annotations and object information.
• NEEM-narrative: a sequence of actions and events, contextualizing object-agent interactions. Actions reuse concepts of the AKG and thus provide a seamless integration of task knowledge.
• NEEM-experience: time-series data (e.g., force, position, and visual inputs) in tf format for low-level motion details and trajectories.

3.2 Essential task knowledge: the meals AKG

The Meals AKG (Kümpel et al., 2025) combines two categories of semantic knowledge for task learning:

(a) Object knowledge: we integrate structured food taxonomies from sources like FoodOn (Dooley et al., 2018) and anatomical information extracted from Recipe1M+ (Marín et al., 2021). This classification identifies relevant object properties, such as whether a fruit has a core or peel, for inferring task parameters like cutting strategies.

(b) Action knowledge: using SOMA (Beßler et al., 2022), we model actions, agent participation, and object roles. To distinguish between task variations (e.g., slicing vs. dicing), we group actions with similar motion patterns, aligning with unstructured lexical data for these tasks from WikiHow and FrameNet (Baker et al., 1998).

To link object and action semantics in the AKG, we use the concepts of dispositions and affordances. A disposition defines what an object enables (e.g., a knife can cut) (Turvey, 1992), while an affordance captures what an agent can do with an object (e.g., an apple affords cutting) (Bornstein and Gibson, 1980). The Meals AKG models these relationships using SOMA's affordsTask, affordsTrigger, and hasDisposition relations. For example, a core removal-task is linked to an apple's anatomical core and the required cutting tool, allowing robots to infer the necessary steps.

3.3 Logging AKG-enhanced NEEMs

Our framework logs NEEMs during human-robot interaction with a structured logging process that captures the evolving task context. At the beginning of an interaction, a new NEEM is instantiated together with a semantic digital twin (semDT) (Kümpel et al., 2021) of the initial environment configuration that contains positions of objects. Semantic digital twins extend traditional digital twins by enriching perception-based scene graphs with semantic knowledge about objects, their properties, and their relations. The effectiveness of creating semDTs for dynamic environments has been demonstrated in a retail setting (Beetz et al., 2022). The semDT includes static background information as well as all relevant entities in the scene. While the overall environment remains fixed, movable objects and agents are continuously updated with their poses over time, ensuring that the NEEM reflects the dynamic aspects of the interaction without redundantly storing unchanged state.

The system monitors for the onset of task-relevant activity. This can either be explicitly triggered by a user request or implicitly detected through rule-based recognition of actions, as proposed by Haidu and Beetz (2016). Whenever such a task or action is initiated, a new episode is created within the active NEEM (see more in Algorithm 1). In this way, a single NEEM may contain multiple temporally ordered episodes. Within each episode, entities are instantiated such that their identifiers are consistent with the concepts used in the underlying AKG, enabling a direct linkage between episodic observations and structured task knowledge.

Algorithm 1 Interactive Task Learning loop for the robot-as-teacher scenario. Numbers at the beginning of a line in square brackets are referencing a specific step of Section 3.6. Require: Prefilled cooking AKG A 1: while True do 2: if No movement detected then 3: Wait 4: else 5: Create a new NEEM N. 6: Start episode e₀. 7: Perceive the scene and all objects. 8: Build the semantic digital twin and create scene graph G. 9: Recognize robot agents Agent_*R* and human agents Agent_*H* in the environment. 10: Store G, Agent_*R*, and Agent_*H* in N. 11: [1] Observe the demonstrated activity. 12: Record perception and demonstration in e₀. 13: Synthesize the observable parameters P_*Obs*. 14: [2] Map P_*Obs* to action core parameters in A. 15: Use initial query Q₀ to retrieve best fitting action core AC from A. 16: Store P_*Obs* in e₀. 17: if P_*Obs* map to an Action Group AG from AC then 18: Record action in N. 19: else 20: i←0 21: P_*Learned*←{} 22: Add special Q&A_*Start*-Tag to N 23: while AG not completly specified do 24: i←i+1 25: Start episode e_*i*. 26: [3] Select the most informative open or ambiguous parameter p_*open* from AC. 27: [4] Ask the human about parameter p_*open*. 28: [5] Interpret the utterance utt_*i* and the demonstration manner demo_*i* to infer p_*Learned*. 29: Set value of p_*open* to p_*Learned*. 30: Store p_*Learned* in P_*Learned*. 31: Update the query to A, creating query Q_*i*. 32: Record utt_*i*, demo_*i*, and the resulting Q_*i* in e_*i*. 33: end while 34: Add special Q&A_*End*-Tag to N. 35: end if 36: State the learned AG and the parameters in P_*Learned*. 37: Record the closing statement, AG, and P_*Learned* in e_*i*+1. 38: return N. 39: end if 40: end while 41: return

By integrating AKG concepts and parameters directly into NEEMs, a robot can access task relevant knowledge directly from the NEEM. For task learning, this means that we include key variables that influence action execution.

For cutting tasks, the following parameters are extracted:

• Tool used: determines motion type (e.g., sawing vs. slicing).
• Cut object: Object properties affect cutting strategies (e.g., stone fruits can not be cut at center while citrus fruits usually need their peel removed).
• Cut position: distinguishes actions like halving (center cut) or slicing (end cut). The actual position in relation to the object is calculated by the robot based on object type and shape.
• Repetitions: differentiates single cuts (cutting on slice, halving) from multi-step slicing and dicing. Again, the actual number of repetitions is calculated based on object size.
• Object rotations: captures whether the object is rotated mid-task (e.g., for dicing).

For pouring tasks, key parameters include:

• Tilt angle: adapted based on liquid viscosity and container shape.
• Pour duration: adjusted for viscosity and task context (adding milk to coffee vs. filling a whole cup).
• Required tools: e.g., sieves for draining or shakers for sprinkling.
• Stopping condition: informed by container capacity or desired volume.

NEEMs are recorded during every interaction between the robot and the human. Thus, when the human performs a demonstration of an action such as cutting, the record is semantically annotated in a NEEM. Incorporating the AKG into the NEEM yields a linkage between a human demonstration and the associated task parameterization. This NEEM can be used downstream to let the robot perform a demonstration with the same parameterization.

3.4 Embodied simulation using semantic digital twins

To operationalize the concept of embodied simulation, we created a Virtual Research Laboratory (VRL) (Beetz et al., 2025) based on the concept of semDTs. The semDT provides a semantically enriched 3D model of the environment, including object classifications (e.g., fridge, shelf, cupboard) and their relations (e.g., stored-in, placed-on). This enables us to simulate recorded NEEMs, including agents and objects.

The VRL serves as an interface for embodied simulation of episodic memories. Specifically, NEEMs recorded during human-robot interaction can be queried, and replayed in a simulated environment that mirrors the physical setup. We created a VRL that allows a user to load an example NEEM and query for task-relevant action parameters to better understand our framework: https://vrb.ease-crc.org/explore-labs/interactive-task-learning-laboratory/.

3.5 Task execution: parameterizing general action plans

We conceptualize robot actions as parameterized action spaces rather than fixed motion trajectories. A parameterized action space defines a structured set of executable task variations whose concrete realization depends on contextual parameters such as object properties, spatial relations, repetitions, force requirements, and task goals. To execute learned tasks, we map enriched NEEM parameters onto general action plans following Flanagan's cognitive model of task decomposition (Flanagan et al., 2006).

As shown in Figure 2, an action (e.g., cutting) is broken into:

• Pick-up: approach, grasp, and lift tool.
• Cut: move tool to object, apply force, and lift tool.
• Place: move tool to surface, release, and retract arm.

The AKG enriches these motions with task-specific parameters, allowing the robot to dynamically adjust execution based on the contextual knowledge from human demonstrations.

3.6 Interaction workflow

The framework operates as a loop that connects observation, memory, inference, and execution. This subsection states the steps of the loop and how the AKG and NEEM interact at each step. The loop proceeds as follows:

• Observe and link: the robot observes a human action and creates a NEEM episode; each entity is instantiated with the identifier of the matching concept in the AKG, which is the link between observation and task knowledge; coherence follows from the shared identifier.
• Infer parameters: the robot queries the linked AKG for the parameter space of the recognized action group; geometry-dependent parameters are computed from object type, context-dependent parameters are retrieved as abstract ranges.
• Detect missing parameters: the robot compares parameters required by the action plan with those present in the NEEM and verbalizes any gap.
• Integrate feedback: human feedback supplies the missing value and is written into a new NEEM episode.
• Execute or simulate: the robot maps parameters onto the general action plan and executes it in reality or in simulation; the execution is recorded as another NEEM episode. A refinement of the parameters follows from repeating the loop.

The loop that is used in the robot-as-a-teacher scenario is shown in Algorithm 1 in more detail. In the algorithm, numbers at the beginning of a line in square brackets are referencing a specific step to the numbers in Figure 3.

An example for this interaction loop is shown in Figure 3. Notably, the concept which is stored in the NEEM as the narrative part of the record is using the same identifier as it is being used in the AKG. Thereby, the system can leverage the structured information about the parameter space of an action from the AKG when it is constructing a question to ask the human interaction partner.

4 Using the interactive task learning framework in practice

To illustrate the practical operationalization of the proposed framework, this section outlines two study setups for future human-robot interaction experiments focusing on task learning and task refinement.

4.1 Study setup for interactive task learning

The two possible study setups are visualized in Figures 4, 5. They assume both a human and a robot as actors in the world, which is mirrored into a semDT. In both setups, the robot observes the human performing an action (Algorithm 1, line 7). The scene is logged into a semDT (line 10) and performed actions are recorded in NEEM episodes (line 12). In the robot as a learner setup, the robot uses the NEEM and the linked knowledge from the AKG to execute the task itself. In the robot as a teacher setup, the robot uses its observations to check the parameter space for existing actions (line 14) in order to execute task parameters or verbalize missing ones (else loop starting in line 19).

4.2 Interactive task learning through human-guided demonstration

To evaluate our framework, we propose a human-robot interaction study using the iCub robot in which the robot acts as a learner. The goal of this study is to investigate how our framework enables a robot to acquire, parameterize and execute a task.

The study scenario is illustrated in Figure 4. A human instructor and the iCub robot are positioned at a table containing everyday kitchen objects, including cups and bowls. The task considered in this setup is a pouring action.

The interaction is planned to proceed in three phases:

• Human demonstration: the human performs the pouring task while verbally grounding the action (e.g., “This is pour”). During this phase, the robot observes the demonstration through its perceptual system, capturing both symbolic and sub-symbolic data such as object identity (here: cup) and motion trajectories, which are recorded as a NEEM.
• Interaction: the robot acts as an observer and only reacts to instructions of the human. If the human gives direct instructions (e.g., “iCub, pour!”), the robot will plan to execute the pouring action. It will do an embodied simulation using the logged NEEM and the AKG parameters. In case it is still missing parameters for the task (such as the goal location) and can not perform the simulation, it will verbalize this. In this case, a new NEEM episode with the additional parameters will be generated.
• Task execution: the robot attempts to reproduce the task based on its internal representation and repertoire of NEEMs, using the presented framework in this work. The task execution itself is recorded as a new NEEM episode. If required, the human provides corrective feedback (e.g. “Stop! You will spill!”).

The creation of consecutive NEEM episodes enables an iterative refinement of the learned task representation.

The NEEM serves as the basis for task execution. By querying the linked AKG, the robot retrieves the necessary action parameters and maps them onto a general action schema for pouring. This enables the robot to instantiate a parameterized action plan rather than replaying a fixed trajectory.

4.3 Interactive task refinement through task teaching

To complement the learner scenario, we designed a second human-robot interaction study in which the iCub robot takes on the role of a teacher. The goal of this study is to investigate how the same framework enables a robot not only to execute tasks but to guide a human user through task completion while refining its own task representation through interaction.

The study scenario is illustrated in Figure 4. The iCub robot and a human user are positioned at a table containing a set of fruits, along with the required tools (e.g., a knife and cutting board). The task considered in this setup is a cutting activity involving multiple objects.

The interaction proceeds in three phases:

• Task instruction: the robot initiates the interaction by issuing a high-level instruction to the human (e.g., “Cut those fruits”). This instruction is grounded in the robot's existing task knowledge, represented through NEEMs linked to the AKG. The instruction itself reflects an abstract task description that may underspecify certain parameters, such as the order of objects or specific cutting strategies.
• Observation and guidance: as the human begins to execute the task, the robot observes the actions through its perceptual system, capturing object identities (e.g., cucumber, carrot) and motion characteristics. These observations are continuously recorded as a NEEM. During this phase, the robot verbalizes its interpretation of the ongoing activity (e.g., “You are slicing the cucumber”) and compares the observed actions with its internal task representation. If relevant, the robot provides guidance or suggestions based on its knowledge (e.g., “Now you might want to cut the cucumber again”), effectively shaping the human's behavior toward task completion.
• Task completion: the interaction continues iteratively until the robot infers that the task goal has been achieved. The robot recognizes achievement by matching the observed parameter values against the parameterization of the target action group in the AKG. For example, the AKG defines quartering as two center cuts with one object rotation, which yields four quarters, while it defines slicing a fruit into four pieces as three cuts without rotation. The robot counts observed cuts, their positions, and rotations, and reports the goal as achieved when these match the target action group. A count of three cuts without any rotation of the object therefore signals four completed slices, whereas two center cuts with a rotation signal four completed quarters. Throughout the process, the robot creates new episodic memories episodes.

Similar to the learner scenario, the creation of consecutive NEEM episodes enables an iterative refinement of the task representation. However, in this setup, refinement is driven by observing human actions. This means that the NEEM contains less time-series position data than first-person NEEMs recorded by the robot.

The NEEMs generated during the interaction serve two purposes. First, they extend the existing task knowledge base. Second, they allow for a change of perspective from first-person internal representation to external observation, making the task representation more robust.

4.4 Parameter space of the framework

To illustrate the expressiveness and existing generalization capabilities of our framework, we analyze the number of task-relevant parameters that can be instantiated through the combination of AKGs and NEEM-based episodic memories. We focus on the two task domains evaluated in this work, namely cutting and pouring.

Within the Meals AKG, actions are organized into semantically coherent action groups that share similar motion patterns and parameterizations (Kümpel et al., 2025). For cutting, we consider the action groups cutting, slicing, dicing, julienning, halving and quartering, which differ in their parameter configurations. Across these groups, the parameter space includes tool selection, object properties, cutting position, number of repetitions, and object rotations. For pouring, we differentiate between the action groups of pouring, pouring through and sprinkling. These action groups are parameterized by tilt angle, pouring duration, stopping condition, and required tools.

Table 1 summarizes the parameterization of these action groups.

Table 1. Overview of action groups and associated parameter spaces for cutting and pouring tasks.

This structured parameter space defines the number of parameters available for task execution through the meals AKG. Across the considered action groups, the framework exposes between two and four execution parameters per action, in addition to object-dependent parameters and task preconditions.

4.5 Optimization of parameter space

The proposed framework enables the refinement of task parameters through the accumulation of NEEMs generated during human-robot interaction. While the AKG provides an initial semantic parameter space, NEEMs provide concrete observations of parameter instantiations in specific task contexts. Over multiple demonstrations, these observations can be used to optimize the parameter values associated with an action group. For each parameterized action, the robot can collect corresponding parameter instances from recorded NEEMs and estimate task-specific preferences or distributions. Discrete parameters, such as the required tool, cut position, or number of repetitions, can be refined by analyzing their occurrence frequency under specific object and task conditions. Continuous parameters, such as pouring angle or duration can be optimized by aggregating observed values and relating them to contextual factors such as object properties, container geometry, or task goals. The optimized parameters can then be written back into the AKG, creating a continuous refinement loop.

This parameter optimization process is currently conceptual and not yet implemented since it requires large amounts of acquired NEEMs. Future work will investigate suitable optimization strategies and learning methods for updating AKG parameters from accumulated NEEM experiences while maintaining the semantic structure and generalizability of the knowledge representation.

5 Discussion

The presented study setups illustrate how combining AKGs with NEEMs could enable a shift from imitation-based learning toward interactive, knowledge-grounded task learning. A robot using our framework would incrementally build and refine its internal task representations through interaction, which results in adaptable action knowledge. While the current work focuses on the conceptual integration of semantic task knowledge and episodic memory structures, future studies will systematically evaluate task generalization, interaction quality, and adaptive behavior across larger task domains and user populations.

A key observation across both study setups is the role of NEEMs as a unifying memory structure. In the learner scenario, NEEMs capture demonstrations and progressively integrate missing task parameters through interaction. In the teacher scenario, NEEMs extend existing knowledge by incorporating observed human behavior from an external perspective. This highlights that task learning in our framework is not tied to a specific role distribution between human and robot, but emerges from the continuous accumulation and refinement of episodic experiences.

The integration of AKGs is intended to provide the semantic backbone required for this process. As shown in the parameter analysis, tasks are not represented as fixed trajectories but as structured parameter spaces, including object dependencies and execution parameters. This would allow the robot to reason about actions at a higher level of abstraction, for example by identifying prerequisite tasks (e.g., slicing before dicing) or adapting execution parameters such as repetitions or cut positions. Importantly, several of these parameters are not directly observable from demonstrations alone but require semantic inference, underlining the limitations of purely imitation-based approaches.

At the same time, NEEMs complement this structured knowledge by grounding parameters in concrete experiences. Through repeated interactions, abstract parameter ranges provided by the AKG can be refined into empirically informed values. This is particularly evident in continuous parameters such as pouring angle or duration, which depend on context-specific factors and benefit from experiential optimization.

The number of episodes required to learn a task depends on the parameter type and the dimensions of the parameter space. Structural parameters of an action, such as cut position, and repetitions, are given by the AKG, so a single demonstration is sufficient to instantiate and execute a task variation. Continuous parameters, like pouring duration or tilt angle, are grounded and refined across episodes, so they benefit from many repetitions. Quantifying the relationship of those influencing factors on the resulting task learning performance is part of the planned studies.

The VRL further supports this process by enabling embodied simulation of recorded experiences. By querying NEEMs and instantiating their parameters in a simulated environment, the robot can analyze and validate task executions before performing them in the real world. This closes the loop between perception, memory, and action, allowing the system to leverage past experiences for both reasoning and planning.

Despite this promising approach, several limitations remain. The system is limited in scale, focusing on a small set of tasks and interactions. While this is sufficient for an experiment to show the feasibility of the approach, larger and more diverse studies are required to assess robustness and generalization. Furthermore, the construction of AKGs involves manual effort, which limit scalability. Automating the extraction of task knowledge from web resources or language data is a key direction for future work. Finally, while the framework supports parameterized task representations, more complex multi-step task compositions and long-horizon dependencies remain to be explored.

Another limitation is that the current AKG and NEEM representations do not consider uncertainty in the perception. E.g. an object is classified as one type, and a pose is a single value. Real perception carries uncertainty. Object classification is correct by a certain confidence, and pose estimation carries a covariance. The NEEM structure can hold these values, since perceptual annotations can store a confidence or covariance alongside each value, and the inference rules can use thresholds on these values rather than static matches. We have not yet modeled uncertainty in this way, and doing so is required before the framework can be applied to noisy real-world perception.

6 Conclusion

This work presents a framework for interactive task learning that combines actionable knowledge graphs (AKGs) with Narrative-enabled Episodic Memories (NEEMs) to enable robots to learn, refine and execute task knowledge through interaction. Moving beyond imitation learning, the proposed approach allows robots to represent tasks as structured, parameterized action spaces grounded in both semantic knowledge and data from past experiences.

We layed out how NEEMs serve as a central memory structure that captures task-relevant interactions over time, linking perceptual observations to semantic task representations. Through a continuous logging process, NEEMs accumulate episodic experiences that can be queried and reused across different contexts. The integration with AKGs provides the necessary task knowledge to generalize across task variations by defining object dependencies and execution parameters.

Our study setups highlight the bidirectional nature of the framework. In the learner scenario, the robot acquires and completes task representations through human demonstration and feedback. In the teacher scenario, the robot guides human behavior while refining its own understanding through observation. In both cases, task knowledge emerges incrementally through interaction, rather than being predefined or statically learned. The analysis of task parameterization further shows that the framework exposes structured parameter spaces that go beyond directly observable features of demonstrations. While AKGs define these parameter spaces, NEEMs ground and refine them through experience, enabling context-sensitive adaptation. Additionally, the VRL supports embodied simulation by allowing the robot to query and replay episodic memories, linking past experiences to future action planning.

While we have not yet evaluated the framework empirically, the conceptual analysis indicates its potential. Future work will address the scalability of knowledge acquisition, for example through automated knowledge extraction, and extend the framework to more complex, multi-step tasks and long-horizon interactions.

Overall, this work contributes to advancing robot learning from static imitation toward interactive, memory-driven task understanding, enabling more adaptive and collaborative behavior in everyday environments.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. An AI tool was used for spell checks and to improve readability.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Mechelle Gittens, The University of the West Indies, Cave Hill, Barbados

Reviewed by: Li Yin, Dalian Jiaotong University, China

Matteo Morelli, CEA LIST, France

Footnotes

Resource website: https://vrb.ease-crc.org/explore-labs/interactive-task-learning-laboratory/