Designing an LLM-driven mobile AR companion for social interaction support in children and youth with autism: a pilot study

1 Introduction

Autism Spectrum Disorder (ASD) affects more than 1% of children worldwide, with core characteristics typically emerging in early childhood Nicholas et al. (2008). Children with ASD often experience persistent difficulties in social communication, reciprocal interaction, and adaptive participation in everyday environments. These challenges may manifest as reduced eye contact, limited conversational initiation, difficulty interpreting social cues, and discomfort in unfamiliar social situations. As a result, many children with ASD face barriers to community participation and independent functioning Lavelle et al. (2014).

To address these challenges, intervention approaches such as Applied Behavior Analysis (ABA) Cooper et al. (2020) and Individualized Education Programs (IEPs) have been widely adopted. IEPs, in particular, provide structured and personalized educational and developmental plans tailored to each child’s needs, emphasizing measurable goals, continuous assessment, and adaptive intervention strategies Wilczynski et al. (2007); Rotter (2014). However, despite their effectiveness in structured settings, these approaches often face limitations in providing continuous, adaptive, and context-aware support in everyday environments. Existing digital tools, including mobile applications, are frequently confined to screen-based or pre-scripted scenarios, offering limited support for real-world interaction and dynamic adaptation Boucenna et al. (2014).

Recent advances in immersive technologies and artificial intelligence offer new opportunities to address these limitations. Augmented reality (AR) enables the integration of virtual elements into real-world environments, supporting situated interaction and contextualized learning Azuma (1997). At the same time, large language models (LLMs) provide powerful capabilities for generating context-aware responses, allowing systems to dynamically adapt interaction content based on user input and behavioral patterns. Together, these technologies create the potential for more flexible and personalized support for children with ASD.

Immersive technologies such as virtual reality (VR) have also been explored for social skill training due to their ability to create controlled and engaging environments. However, designing immersive systems for children with ASD presents unique challenges, particularly in relation to sensory sensitivity and responses to wearable devices. Prior research indicates that children with ASD exhibit a high prevalence of comorbid psychiatric conditions, such as specific phobias, which may contribute to heightened sensitivity to unfamiliar or intrusive sensory experiences, including wearable equipment Leyfer et al. (2006). In this context, head-mounted VR devices, which require sustained wearing and introduce immersive visual stimuli, may present barriers to acceptance and practical use. To reduce such barriers and improve usability, this study adopts a mobile AR approach, allowing virtual elements to be embedded into the real-world environment without requiring intrusive wearable devices.

In response to these challenges, we present Stand By Me, an LLM-driven mobile AR application designed to support social interaction in children with ASD. Grounded in IEP principles, the system integrates AR-based virtual companionship, multimodal interaction (speech and gesture), and adaptive task generation to provide personalized and context-aware support. Through interaction with an AR virtual pet, children are encouraged to engage in communication and participate in real-world activities such as guided travel tasks and simple social scenarios. Unlike static applications, the system dynamically adapts its responses and task difficulty based on the child’s profile and interaction history.

This work is designed as a pilot study to explore the feasibility, acceptability, and interaction patterns of such an approach, rather than to establish causal effects on social skill development. Accordingly, we investigate the following research questions:

RQ1: How can an AI-driven intervention system grounded in IEP principles support social interaction in children with ASD?

RQ2: To what extent is an LLM-driven mobile AR system feasible and acceptable for guiding communication and real-world interaction in children with ASD?

To address these questions, we design, implement, and evaluate the Stand By Me system in collaboration with a rehabilitation institution. The contributions of this work are as follows:

• We present the design of a mobile AR intervention system grounded in IEP principles to support social interaction in children with ASD.
• We explore the integration of LLM-based adaptive interaction and AR-based embodied interaction as a feasible approach for supporting communication and real-world engagement.
• We report a real-world pilot study with 9 children with ASD, providing preliminary evidence on feasibility, acceptability, and interaction patterns.

2 Related work

2.1 Intervention foundations for social skill development in autism

Autism Spectrum Disorder is a neurodevelopmental condition characterized by deficits in social communication, limited interaction abilities, and repetitive behaviors Brentani et al. (2013). Interventions for children with ASD typically require long-term and structured support. Applied Behavior Analysis (ABA) has been widely adopted and has demonstrated its usefulness in supporting communication and adaptive behaviors in children with ASD Cooper et al. (2020); Yu et al. (2020).

In addition, the Individualized Education Program (IEP) provides a structured framework for planning personalized intervention strategies for children with special education needs, including children with ASD. The core goal of an IEP is to develop targeted educational goals and intervention strategies based on each child’s current abilities and developmental needs. A typical IEP includes the child’s present levels of academic achievement and functional performance, measurable annual or short-term goals, special education services and related supports, participation in general or real-world settings, and procedures for assessment and progress monitoring Wilczynski et al. (2007); Kurth and Mastergeorge (2010). In the context of ASD, IEP-based approaches are particularly important because children may differ substantially in language ability, social communication, behavioral regulation, and functional independence.

IEP-based interventions can support multiple developmental domains for children with ASD. For example, social skill goals may focus on eye contact, turn-taking, cooperation, and reciprocal communication, while behavioral goals may help reduce problem behaviors and improve emotional regulation. IEPs may also include functional goals related to daily living skills, self-care, mobility, and community participation Bellini et al. (2007); Wehman et al. (2014). By defining individualized goals, monitoring progress, and adapting support over time, IEP-based approaches aim to address the diverse developmental needs of children with ASD Wilczynski et al. (2007); Silva et al. (2024).

However, these intervention approaches often rely on sustained involvement from professionals and caregivers and are typically conducted in structured environments, making it difficult to provide continuous and adaptive support in everyday contexts. Effective IEP implementation also requires regular documentation of the child’s performance and timely adjustment of intervention strategies, while families and institutions may face limitations in time, expertise, and resources Fish (2008). Furthermore, the transfer of learned behaviors from structured settings to real-world situations remains limited Bellini et al. (2007). Therefore, providing continuous, personalized, and context-aware support in real-life environments remains an important research challenge.

With the development of artificial intelligence technologies, intelligent systems have been explored as a way to optimize the implementation of IEPs through automated data recording, personalized intervention design, and real-time progress monitoring. These technologies may improve the flexibility and timeliness of individualized support, while reducing part of the workload for parents, educators, and intervention professionals.

2.2 Animal-assisted therapy in autism intervention

Animal-Assisted Therapy (AAT) has been widely explored as a complementary intervention for children with ASD. It has been shown to help alleviate certain behavioral symptoms, such as aggression, irritability, attention deficits, and hyperactivity O’Haire (2013).

However, animal-assisted interventions face practical limitations, including high cost, limited accessibility, limited animal resources, and safety concerns, which restrict their scalability in real-world applications. With the development of mobile and immersive technologies, an increasing number of applications have been proposed for autism intervention. These systems often incorporate key ideas from animal-assisted therapy, such as emotional engagement, sensory stimulation, low-pressure interaction, and task-oriented activities, through virtual animals, simulated environments, and gamified designs Hüsgen et al. (2022).

For example, Cai et al. (2013) developed a virtual dolphinarium for children with autism as a VR-based intervention inspired by animal-assisted therapy. The system transformed traditional dolphin-assisted interaction into a controllable virtual environment by simulating dolphin sonar feedback, swimming trajectories, and haptic vibration. The study showed that children with autism exhibited higher engagement and more active interaction behaviors in the virtual environment, with improvements in eye-tracking and joint-attention tasks.

Overall, virtual animal-based interaction digitally reproduces several core values of animal-assisted therapy, such as emotional connection, sensory stimulation, and task-oriented interaction. Compared with traditional animal-assisted therapy, it may reduce limitations related to cost, safety, and limited animal resources, while also enabling personalized adaptation and real-time feedback for autism intervention.

2.3 Applications for social interaction and mobility support in ASD

In recent years, a growing number of applications have been developed to support social interaction training for children with ASD. These systems typically use scenario-based training, visual prompts, or gamified approaches to guide children in practicing basic communication skills, such as emotion recognition, simple dialogue, and behavioral norms Hourcade et al. (2012). However, many of these applications rely on predefined content and fixed interaction patterns, limiting their ability to adapt to individual users and dynamic environments.

In terms of mobility and real-world support, some systems aim to assist children with ASD in completing everyday tasks, such as navigation and routine activities. However, these applications often face challenges in handling complex environments and providing continuous interactive guidance in real-world contexts Koegel et al. (2003).

At the same time, immersive technologies such as AR and VR have been introduced into autism intervention. AR overlays virtual information onto the real world, enabling context-aware interaction and guidance, while VR creates fully immersive environments for controlled training scenarios Howorth et al. (2019). Recent studies have found that VR interventions have positive effects on social skills in children and adolescents with ASD Yang et al. (2025), while an empirical study reported that VR-based training targeting safety and social communication skills was acceptable and usable, and supported the generalization of learned skills to natural environments Gayle et al. (2024). However, when designing systems for children with ASD, it is important to consider sensory sensitivities and responses to different forms of interaction. Prior research indicates that children with ASD often exhibit a high prevalence of comorbid psychiatric conditions, such as specific phobias, which may contribute to increased sensitivity to unfamiliar or intrusive sensory experiences Leyfer et al. (2006). In this context, wearable devices such as head-mounted VR systems may introduce additional barriers to acceptance and sustained use. In contrast, mobile AR does not require intrusive wearable equipment and allows interaction within real-world environments, which may reduce usage barriers and improve accessibility.

2.4 Large language models in autism intervention

With the rapid development of artificial intelligence, large language models have been increasingly applied in educational and assistive technologies. LLM-based systems can generate context-aware responses based on user input, enabling more flexible and adaptive interaction experiences Deng et al. (2025); Aydemir (2025).

Recent studies have increasingly explored the potential of LLMs in autism-related assistive contexts. For instance, research involving autistic workers indicates that LLMs can support communication in socially complex workplace situations Jang et al. (2024). In clinical intervention settings, ASD-iLLM demonstrates the feasibility of adapting LLMs with real dialogue intervention data and intervention strategies for supporting autistic children Lai et al. (2025).

In the field of autism intervention, recent studies have explored the use of LLMs for conversational support, behavioral prompting, and task guidance Singla et al. (2024). Compared with traditional rule-based systems, LLMs offer advantages in handling complex language input and generating personalized responses, which may better accommodate users with diverse abilities.

However, most existing applications focus primarily on text- or speech-based interaction within screen-based environments, with limited integration into real-world contexts. The combination of LLM-driven adaptive interaction with embodied and context-aware systems in real-life environments remains underexplored.

3 Stand by me application

Stand By Me is designed as an AI-driven mobile AR interactive system that supports social communication and real-world participation in children with ASD. Drawing on IEP principles, the system aims to translate individualized goals into interactive, situated, and repeatable activities that can be used under caregiver supervision.

3.1 Design goals

Building upon our earlier research, which involved an iterative design process and an initial user study, we formulated three design goals.

• DG1: Promote Social Engagement through Embodied Interaction. The system should encourage both verbal and non-verbal communication through a socially responsive AR companion. Rather than acting as a passive interface element, the AR pet should respond to gestures, speech, and task progress in ways that create a sense of companionship, responsiveness, and low-pressure social participation.
• DG2: Generate Personalized Real-World Training. The system should use LLM-based reasoning together with contextual and profile data to generate personalized travel and socialization tasks aligned with IEP logic. These tasks should adapt to the child’s language ability, mobility, confidence, and prior performance.
• DG3: Optimize the Individualized Plan through Adaptive Feedback. The system should continuously record interaction and travel data, enabling longitudinal refinement of prompts, task difficulty, and support strategies. In this way, the application should function as an adaptive companion rather than a fixed instructional tool.

3.2 User flows

The primary user interfaces and interaction flow of Stand By Me are shown in Figure 1.

3.2.1 Multimodal interaction with the AR virtual pet

After launching the application, the child enters an AR interface where a virtual pet is anchored in the physical environment. The pet serves as a socially approachable, embodied companion. The system supports two main channels of interaction:

• Gesture interaction. Using the rear camera, the application recognizes predefined gestures such as waving. The pet responds with corresponding animations, encouraging non-verbal communication and reinforcing reciprocal interaction.
• Verbal dialogue. The child speaks to the pet through the microphone. The speech signal is transcribed into text and processed by the LLM, which generates a context-aware, empathetic response presented through both text and synthesized voice. Conversation histories can be reviewed by caregivers to monitor social progress.

3.2.2 Intelligent travel training

To support social adaptation beyond conversational interaction, Stand By Me includes an intelligent travel guidance loop. Based on the child’s profile and previously recorded interaction indicators, the system generates personalized travel plans that gradually increase in distance and complexity. During the initial exploration phase, baseline data are recorded, including access to core functions, speech and gesture interaction attempts, prompt requirements, caregiver assistance level, and willingness to continue use. These data are preserved for longitudinal comparison and combined with static profile information, such as language ability, mobility skills, and activity preferences, to support subsequent task personalization.

Tasks may include simple real-world objectives such as navigating to a nearby place, recognizing landmarks, or completing basic shopping-related errands. During the initial setup, caregivers enter the child’s preferred reward items, such as chocolate, toys, or other motivating objects. These preferences are used to personalize the gamified reward interface. A reward is triggered when the child completes one outing task, with completion confirmed through navigation status, caregiver observation, or task-recording logs. After task completion, the preferred item is displayed as a virtual object emerging from a treasure chest, and the child also receives an achievement medal as positive feedback.

3.2.3 Data-driven analysis and plan refinement

The application records interaction frequency, app usage, task completion, and other behavioral indicators. These data support progress analysis for caregivers and professionals, while also enabling the system to adapt future training plans. In this sense, the system creates a feedback loop between child behavior, caregiver observation, and AI-generated intervention content.

3.3 System implementation

Stand By Me was implemented as an iOS mobile application developed in Swift. The prototype was designed to run on iOS devices equipped with a camera, microphone, speaker, and network connectivity. The camera is used for AR rendering and gesture interaction, the microphone and speaker support speech-based interaction, and network access is required for privately deployed server-side AI services, including speech recognition, LLM-based response generation, and text-to-speech synthesis, as well as map-based travel planning.

3.3.1 System architecture

The system architecture (Figure 2) employs a large language model as the central inference engine to coordinate multimodal data streams. The system integrates three main data sources: (1) children’s static profiles, including cognitive level, language ability, and action skills, which are provided by the rehabilitation institution according to the structured descriptions in the VB-MAPP assessment and entered into the system during the initial use of the application; (2) real-world contextual data obtained through geographic information processing to identify points of interest (POIs); and (3) real-time interaction data collected from the child’s speech, text, and gestural inputs. To protect user privacy, child-related profile information and longitudinal interaction records maintained by the application are stored locally on the device, while only the minimum data required for runtime inference or task planning are transmitted to private server-side services.

The implementation is structured into three key modules, which are described in the following sections.

3.3.2 Dialogue interaction module

The dialogue module enables naturalistic interaction between the child and the AR pet. Speech from the child is captured through the mobile microphone and transcribed into text using an automatic speech recognition (ASR) module. The transcribed text is then combined with contextual data from recent interactions and task status. These inputs are passed to the Large Language Model (LLM) for context integration and response generation.

The conversational module is powered by Qwen2.5–72 B, which is used to generate developmentally appropriate, empathetic, and context-aware responses. The system utilizes customized prompt engineering to control the tone, difficulty, and relevance of the replies according to the child’s linguistic comprehension and text-based emotional cues. After the child’s speech is transcribed into text by the ASR module, the transcribed utterance and recent dialogue context are passed to the LLM for lightweight emotion classification, such as positive, neutral, confused, anxious, or frustrated. The resulting category is used to adjust the response style and support intensity, rather than to provide a clinical assessment of the child’s emotional state. Subsequently, the textual response is converted into speech using a text-to-speech (TTS) API, allowing the virtual pet to answer in a friendly and consistent voice.

Compared with scripted chat flows, this approach supports greater variability, continuity, and personalization. For children with ASD, such responsiveness is important because communication support must often be adapted in real time.

3.3.3 Personalized travel planning module

The travel planning module integrates geospatial data processing with LLM-based contextual reasoning to generate personalized real-world tasks. The system first collects autism-friendly travel scene preferences and caregiver-provided contextual information as local personalization constraints. For map-based retrieval, only the location or destination query information required for route and POI search is sent to the map service. Child profile fields, diagnostic information, dialogue transcripts, and interaction records are not included in map requests.

The retrieved route and POI information is then combined with child-specific constraints by the LLM to produce personalized task plans. Example tasks include simple activities such as “go to the nearby store and buy one apple,” which can be progressively extended to more complex real-world scenarios. The system adjusts destination distance, task sequence, and language complexity according to the child’s cognitive level, language ability, mobility skills, and usage progression, enabling tasks to follow a gradual and individualized progression.

By combining the spatial precision of the Google Maps API with the contextual reasoning capabilities of the LLM, the system connects real-world navigation with adaptive task generation. Destinations may range from familiar nearby environments to more complex or distant locations, depending on the child’s profile and prior use. As shown in Figure 3, the system adopts a three-layer prompting structure (system constraints, user context, and structured output) to support task consistency and reliability.

3.3.4 Adaptive memory and longitudinal personalization

Stand By Me leverages the contextual memory and continuous reasoning capabilities of the large language model to sustain personalized and dynamically evolving interaction across sessions. During each interaction, the model retains short-term conversational context, such as the child’s emotional state, task progress, and prior choices, ensuring coherence and continuity in dialogue and task guidance. Beyond immediate context, the system incorporates long-term user profiles and historical interaction data to adaptively optimize prompt templates and behavioral response strategies.

Through continuous analysis of behavioral patterns, linguistic progress, and text-based emotional cues, the model dynamically adjusts the tone, complexity, and pacing of subsequent dialogues and training tasks. For example, when a child shows improved verbal fluency or task confidence, the system increases linguistic variety and introduces more complex scenarios. When the ASR-transcribed utterance and recent dialogue context indicate confusion or frustration, the LLM performs a lightweight text-based emotion classification and simplifies the instructions or increases supportive feedback frequency accordingly. The inferred emotion category is used as an auxiliary adaptation signal rather than as a clinical measurement of frustration. This adaptive memory mechanism forms a feedback loop in which each interaction enriches the user profile and improves the model’s understanding and predictive accuracy, enabling adaptation to interaction characteristics across different stages of use.

3.3.5 Data privacy and safety constraints

The system distinguishes three types of data. First, child profile data include cognitive level, language ability, action skills, reward preferences, interaction records, task records, and caregiver feedback. These data are stored locally on the device or processed only by privately deployed server-side services when required for personalization. Before storage or analysis, directly identifying labels are removed or replaced with pseudonymized participant IDs, and the system does not include names, contact information, or complete rehabilitation records in model prompts.

Second, AI interaction data include speech input, ASR-transcribed text, recent dialogue context, LLM-generated responses, and TTS text. The ASR, LLM, lightweight text-based emotion classification, and TTS components are deployed on private cloud servers controlled by the research team or collaborating institution. Therefore, these AI interaction data are not transmitted to third-party commercial AI services. Runtime data transmitted to these private services are used only for the corresponding inference process and are not used for external model training.

Third, map-related data include only the location or destination query information required for POI retrieval and route planning. These map requests do not include the child’s name, diagnostic information, static profile, interaction records, or dialogue transcripts. Raw camera streams are used only for real-time AR rendering and gesture interaction and are not stored as long-term video records. Locally stored study records are retained only for the period necessary for pilot analysis and are then anonymized or deleted according to the study protocol.

To reduce risks associated with LLM-generated content, the system uses constrained prompts and structured output requirements to keep responses short, supportive, child-friendly, and task-relevant. The LLM is not permitted to provide medical diagnosis, clinical treatment advice, unsafe navigation instructions, or suggestions for unsupervised outdoor activities.

4 Evaluation of the application through user testing

To explore the feasibility and practical use of Stand By Me, we conducted a real-world user study in collaboration with a local autism rehabilitation institution. The study focused on children’s engagement with the system, ease of interaction, and the system’s ability to support social interaction and participation in real-world tasks.

The evaluation was designed to address the following questions in relation to our research objectives:

• Whether the application provides accessible and meaningful embodied interaction for children with ASD; and
• Whether it is feasible and acceptable in supporting communication behaviors and real-world participation.

4.1 Participants

We recruited 9 participants with ASD from a collaborating rehabilitation institution. All participants were at VB-MAPP Level 2, indicating that they possessed sufficient basic verbal ability to engage with the application. Participants ranged in age from 6 to 22 years. Written informed consent was obtained from the participants’ legal guardians or authorized caregivers. Participants were informed about the study procedures in an ability-appropriate manner, and their willingness to participate was confirmed before and during the sessions.

4.2 Procedure

The user study procedure was developed in consultation with caregivers and staff from the collaborating rehabilitation institution and consisted of three phases. Before the sessions, caregivers received a brief orientation on the purpose of the study, the basic operation of the application, safety precautions, procedures for pausing or terminating a task, and their role in supervising participants during system use.

4.2.1 Phase 1: initial exploration

After obtaining consent, participants explored the application independently or with minimal guidance. Researchers recorded how participants located and began using the main features, in order to understand their initial interaction patterns and usage behaviors.

4.2.2 Phase 2: continued use

During the pilot period, Stand By Me was embedded into participants’ regular one-to-one ABA training schedule at the collaborating rehabilitation institution. Each participant attended a daily ABA training block of 3 h, during which the application could be used as part of structured indoor interaction practice and supervised outing-related activities. This 3-h block represented the maximum daily exposure opportunity rather than continuous active screen time. Active use of the application occurred intermittently within this block, depending on the participant’s behavioral state and caregiver judgment.

Children first used the application in indoor settings under caregiver supervision to become familiar with the AR pet interaction, speech dialogue, gesture interaction, and task guidance functions. During this phase, children interacted with the AR pet through speech and gestures, while researchers observed engagement, interaction behaviors, and task-related responses.

Children then used personalized LLM-generated travel tasks under caregiver supervision. Outing activities were conducted within the same ABA training block and were always supervised by caregivers or institutional staff. For safety reasons, caregivers held the mobile device during outdoor tasks, while children followed the system’s voice prompts, AR cues, and caregiver-mediated instructions. The duration of each outing was not fixed because it depended on destination distance, task complexity, weather, participant condition, and caregiver judgment. In practice, each outdoor task typically lasted approximately 1–2 h, and caregivers could pause, adjust, or terminate the task when necessary.

4.2.3 Phase 3: feedback collection

After the usage cycle, a brief unstructured feedback form and short interviews were conducted with caregivers. The feedback form contained one open-ended question asking caregivers to describe their overall experience with the system, including helpful aspects, difficulties, and suggestions for improvement. It was used only for exploratory feedback collection and was not designed as a structured or validated questionnaire. Follow-up interviews were then conducted to further understand participants’ and caregivers’ user experience.

4.3 Metrics and data collection

Data were collected from system logs generated by the application and caregiver feedback collected after the usage cycle: The training cycle lasted from September 1 to October 31, 2025, during which up to 31 system-use records were collected for each participant. Given the exploratory nature of this real-world pilot and the small sample size, the evaluation focused on descriptive indicators of feasibility, acceptability, and observed interaction patterns rather than inferential efficacy outcomes. Data were collected from system logs generated by the application and caregiver feedback collected after the usage cycle.

The usage time was defined as the cumulative duration for which the application was open and actively used during the daily ABA training block. Interaction frequency was defined as the number of child-initiated communicative acts directed toward the AR virtual pet during a session. These included verbal utterances captured through the speech interface and predefined gesture interactions recognized by the camera module. Repeated system prompts or caregiver instructions were not counted as child-initiated interactions unless the child produced a verbal or gestural response.

Task completion was defined at the outing-task level. A task was coded as completed when the child reached the planned destination or completed the assigned real-world objective with system guidance and caregiver supervision. Tasks that were paused, interrupted, or terminated for any reason were recorded as incomplete, depending on whether the task had already started.

Engagement was assessed as an observational indicator rather than as a validated clinical outcome. Caregivers and researchers considered whether the child maintained attention to the AR pet or task, responded to prompts, voluntarily continued interaction, and showed willingness to proceed with the activity. These observations were used to support qualitative interpretation of acceptability and feasibility.

5 Results

5.1 Engagement with Stand By Me

Table 1 summarizes participant-level exposure and data completeness. All nine participants attended each scheduled training day during the pilot period. System use varied across participants according to individualized training arrangements, readiness for outing tasks, caregiver judgment, and the child’s daily behavioral condition. Some outing tasks were not completed because the session was paused, the outing was not scheduled, or caregivers judged that continuing the task was inappropriate. These cases were recorded as task non-completion. No serious adverse events were observed during the pilot. Caregivers retained the authority to pause, modify, or terminate any task throughout the study.

TABLE 1. Participant characteristics and system-use summary during the pilot study.

The training cycle lasted from September 1 to 31 Oct 2025, during which up to 31 system-use records were collected for each participant. App-use duration refers to cumulative active app-use time within the ABA, training block. Interactions refer to child-initiated speech or recognized gesture responses.

Participants also showed sustained use of the application within the scheduled ABA training blocks. The mean recorded app-use duration ranged from 59.0 to 73.0 min per valid log record, with an overall mean of 67.7 min. The mean number of child-initiated interactions with the AR virtual pet ranged from 3.0 to 4.3 per valid log record, with an overall mean of 3.8 interactions. For supervised outing-related tasks, participants completed 137 of 200 assigned tasks, corresponding to an overall completion rate of 68.5%. These results suggest that Stand By Me could be integrated into supervised institutional routines and used repeatedly across the training cycle.

Figure 4 presents daily descriptive trends across the full participant group. Daily app usage and AR-pet daily interaction were averaged over all nine participants, while the daily task completion rate was calculated as the proportion of completed outing tasks among the outing tasks assigned or initiated on the corresponding day. These trends indicate increased engagement with the system and gradual adaptation to its interaction mechanisms and task structure. During system use, participants showed a gradual increase in both language-based and gesture-based interaction frequency, reflected in more frequent verbal communication and gesture-based interactions with the AR pet. Caregivers reported that some children demonstrated a relatively high level of willingness to participate in outdoor activities and actively engaged in system-guided tasks. In addition, the step-by-step guidance and personalized structure provided by the application were considered to align well with existing institutional training practices. During interviews, caregivers also noted that children performed tasks more smoothly and were able to complete them with the support of prompts and guidance.

5.2 Communication and interaction behaviors

Communication and interaction behaviors were mainly reflected in children’s speech- and gesture-based engagement with the AR virtual pet. Across the training cycle, participants showed an overall increase in AR pet interaction frequency. These interactions included child-initiated verbal utterances captured through the speech interface and predefined gesture responses.

More specifically, children actively interacted with the AR pet through speech and gestures. The large language model in the system generated contextually relevant responses based on the children’s input, allowing the interaction to continue. In interviews, when asked about the virtual pet, children generally gave positive responses, for example, describing the pet as “It’s cute!” These findings suggest that the AR pet, as an interaction partner, was able to attract children’s attention during use and support their participation in communication activities. This interaction format also showed good acceptability in real-world use and indicates the potential of the system in supporting communication-related behaviors.

5.3 Task completion and real-world participation

For real-world task participation, this study used outing task completion as the main observational indicator. The system generated personalized tasks based on children’s basic profiles and prior interaction data. Tasks began with nearby and structurally simple destinations and then gradually extended to more complex real-world scenarios.

During the training cycle, outing task completion rates showed an overall upward trend. Although task distance and complexity gradually increased, children were still able to complete the assigned tasks with the support of voice prompts and route guidance. Caregivers also reported that children showed good cooperation during outings and were willing to participate in goal-oriented activities guided by the system.

6 Discussion and future work

6.1 Interpretation of findings

This study explored the application of Stand By Me for children with ASD in real-world usage contexts. The results showed that over a period of use, children exhibited observable trends in engagement, interaction behaviors, and task participation. These observations provide a basis for understanding how the system functions in practice.

At the interaction level, verbal and gesture-based interactions between children and the AR virtual pet were sustained throughout use and showed a gradual increase. This pattern may be related to the low-pressure interaction style supported by the system. Compared to traditional task-oriented training approaches, the virtual pet serves as an embodied interaction partner, providing a more natural entry point for communication and enabling children to participate in interaction within a less demanding context.

In terms of real-world task participation, children were able to complete assigned tasks with the support of voice prompts and route guidance, while maintaining relatively stable performance as task complexity increased. Embedding interaction support within real-world activities helps connect in-app interaction with physical-world engagement, thereby supporting participation in everyday contexts.

In addition, the context-aware responses generated by the large language model allow the system to continuously adapt to children’s input and sustain interaction over time. This dynamic interaction differs from traditional scripted systems and supports a more continuous interaction experience. Combined with the spatial presence of AR in the physical environment, the system demonstrates characteristics of embodied and situated interaction, which may contribute to maintaining engagement.

6.2 Limitations and future work

Despite these observations, several directions remain for further exploration. First, future studies could involve larger and more diverse participant groups to better capture variations in usage across individuals. Second, children with ASD exhibit substantial variability in cognitive abilities, language skills, and sensory sensitivities, which presents challenges for system design. Future work could further enhance adaptability to diverse user needs, for example, by dynamically adjusting interaction strategies and task structures based on individual profiles.

Another limitation concerns measurement precision. The current pilot relied on system logs, observation, and did not include standardized clinical outcome measures, or formal inter-rater reliability checks. Therefore, the reported engagement, interaction frequency, and task completion outcomes should be interpreted as descriptive feasibility indicators rather than validated efficacy measures. Future controlled studies could incorporate predefined behavioral coding schemes, independent raters, inter-rater reliability analysis, and standardized engagement or social communication scales.

In addition, some system functions currently rely on caregiver assistance, such as device handling and task support during outdoor activities. Future work could improve usability and interaction design to better support independent use, thereby enhancing accessibility and practical applicability in real-world settings.

7 Conclusion

This study presents Stand By Me, a mobile augmented reality system integrated with a large language model to support interaction and task participation for children with ASD in real-world contexts. By combining a responsive virtual companion, multimodal interaction through speech and gesture, and personalized task generation, the system embeds individualized goals into everyday activities.

Through a real-world user study conducted in collaboration with a rehabilitation institution, we explored the system’s performance in practice. The results indicate observable trends in children’s engagement, interaction behaviors, and task participation over a period of use. Children were able to sustain interaction with the AR virtual pet and participate in real-world tasks with the support of system guidance.

Overall, this work suggests that LLM-supported interactive virtual characters have potential as adaptive social partners in situated environments. The findings provide a basis for future research on AI-supported embodied interaction, virtual companionship, and real-world social participation for children with ASD.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Wei Song, Beijing Institute of Graphic Communication, China

Reviewed by: Shuaibing Lu, Beijing University of Technology, China

Apurba Ghosh, Daffodil International University, Bangladesh