Validation of a computer-vision milking parlour monitoring system

Artificial intelligence can help identify suboptimal milking practices and provide reports without time-consuming visits. The objective of this study was to validate a computer-vision milking parlour monitoring system (CVS, Cattle Care Inc.) by comparing its reports against trained human observers (n = 2, comparison method). A camera was installed in the parlour of two dairy farms in Ontario, Canada, one operating a herringbone parlour and the other a parallel parlour. Four one-hour segments (2 per farm), selected by the CVS, were independently analysed by human observers and the CVS. Prep-lag time, average unit-on time, turn time, and milking protocol deviations were assessed in each shift and compared between methods. For continuous variables, Lin’s concordance correlation coefficients (CCC) were calculated to evaluate the agreement between CVS and human observers; Bland–Altman plots were used to assess systematic bias. For categorical variables, the F-score was calculated to assess event-detection performance. The CCC for unit-on time (n=44 milking turns) was 0.999 (95% CI: 0.999-0.999), for prep-lag time (n = 214 cows) was 0.995 (95% CI: 0.994–0.996) and for turn time (n = 25 turns) was 0.998 (95% CI: 0.997–0.999). All protocol deviations flagged by the human observers (n = 51 events) were detected by the CVS; six towel misuse events detected by the CVS were missed by the human observers. An F-score of 1 was obtained for the CVS method. Under the conditions tested, the CVS demonstrated high agreement with human observations for prep-lag time, unit-on time, turn time, and milking-protocol deviations, while also identifying some events not detected by human observers. The assessed technology may complement human expertise by enabling more efficient and streamlined assessment of milking parlour performance.

Optimization of the milking routine is a key strategy for reducing the incidence of intramammary infections and for improving milk quality, milk yield, and overall milking efficiency (Erskine et al., 2019; Watters et al., 2012). An effective routine is characterized by a low-stress environment, adequate stimulation prior to unit attachment, and consistent pre-milking preparation and post-milking teat disinfection (National Mastitis Council, 2013). The extent to which these outcomes are achieved depends largely on routine compliance, underscoring the importance of consistent and optimal execution of each step (Beekhuis-Gibbon et al., 2011; Rodriguez et al., 2025).

Milking parlour audits are widely used to monitor compliance and identify deficiencies in milking procedures (Flores-Miyamoto et al., 2014). However, these audits require prolonged in-person observation, which may not always be feasible. Additionally, a single observation may not accurately represent routine performance due to the Hawthorne effect (behavioural changes that occur when individuals are aware they are being observed, McCambridge et al., 2014), or the absence of employees working different shifts (Douphrate et al., 2012; Sedgwick and Greenwood, 2015).

Remote video auditing (RVA) offers an alternative when in-person assessments are impractical or cost-prohibitive. These systems have been implemented in hospital and food preparation settings to evaluate worker performance (Almqvist et al., 2023; Overdyk et al., 2016), with evidence indicating improvements through increased task awareness and timely feedback (Lacey et al., 2020). Adoption of RVA on dairy farms may similarly identify opportunities for improvement and inform targeted training initiatives, which have been shown to enhance worker performance and adherence to milking protocols (Rodriguez et al., 2025). Furthermore, computer vision, a branch of artificial intelligence concerned with the automated interpretation of images and video (Aloimonos and Rosenfeld, 1991), offers additional potential to reduce the time and costs associated with video-based assessments (Svanberg, 2023). As with other artificial intelligence technologies, validation is required to assure adequate precision and accuracy (Xie et al., 2022). A recently developed computer vision software (CVS, Cattle Care Inc., Marina del Rey, CA, USA) enables automated evaluation of the milking routine by generating efficiency metrics and identifying deviations from protocol. Its performance in detecting reattachment and manual detachment events, as well as prep-lag time has been evaluated on a commercial farm using a parallel parlour, where it demonstrated high accuracy (Wieland and Heuwieser, 2025). However, its ability to detect additional milking routine deviations and the accuracy of other parameters included in the generated report (turn time, unit-on time) have not yet been assessed. The objective of this study was to validate the performance of the CVS, namely regarding prep-lag time, unit-on time, turn time, as well as detecting milking protocol deviations by comparing its automatically generated reports with trained human observers. We hypothesized that the CVS method would perform comparably to human observers.

This validation study was reported using the STARD-AI guidelines (Sounderajah et al., 2025). It was conducted between August and September 2024 and was approved by the University of Guelph Research Ethics Board (REB #114).

Farm characteristics and milking protocols

Two commercial dairy farms in Southwestern Ontario, Canada were recruited based on their proximity to the University of Guelph and willingness to participate in the study. The average daily production was 41 kg/d per cow on Farm A and 40 kg/d per cow on Farm B. Farm A milked 350 cows in a 10 × 2 herringbone parlour, and Farm B milked 110 cows in an 8 × 2 parallel parlour. Both farms milked three times daily with a single operator per shift. The standard milking protocol for Farm A was done in groups of five cows, applying teat pre-dipping, teat cleaning with a cloth towel, unit attachment, and post-dipping with a dip cup. For Farm B, cows were milked in groups of four and followed a similar sequence, with the addition of forestripping prior to teat pre-dipping.

The computer vision software and training of observers

The CVS incorporates multiple convolutional neural networks developed using video from a multiparlour setup (Wieland and Heuwieser, 2025). Object detection was performed using YOLO (You Only Look Once, version 8.0) alongside modified detection models similar to this method for the identification of key elements, including humans, cows, and milking equipment. Vision-language models and convolutional neural networks classify specific milking events. Cases in which the CVS flags a critical event with low confidence or identifies an unusual farm-specific event are subject to human verification; however, no human verification was required for the footage evaluated in this study. The CVS monitors continuous variables by capturing key events throughout the milking process, allowing for the derivation of performance metrics at the cow, turn, and parlour levels. Additionally, according to the manufacturer, the system can detect deviations from the farm’s standard milking protocol (e.g., lack of forestripping or teat-end cleaning, missed pre or post-milking teat dipping, the use of one towel on more than one cow), as well as specific operator behaviours (e.g. smoking, cellphone usage, and interactions that may compromise cow welfare).

Human observers (n = 2, comparison method) were trained by assessing a one-hour segment of practice footage from one of the study farms. Training included review and feedback from a veterinarian experienced in milking routine evaluation. Observers were trained to identify deviations from each protocol and to capture key events required to calculate parameters of interest. Inter-observer comparisons during the training exercise showed no differences in the detection of behaviours that deviated from the established milking protocol, while prep-lag time and turn time differed by ≤2 seconds.

Data collection and processing

A camera (RLC-510WA, Reolink, City of Industry, CA, USA; 2560 × 1920 resolution at 20 frames/sec) was installed by the research team on the side opposite to the cows’ entrance to ensure full visibility of the milking units and the operator. Cameras remained in place for two days. The research team assessed an hour of video to confirm that camera placement allowed accurate evaluation of required variables. Video files were stored on an SD card (Lexar, San Jose, CA, USA; 128 GB) and, after card retrieval, files were transferred by the research team to the CVS via Google Drive (Google, Mountain View, CA, USA) for processing. The CVS analysis was blinded to human observations and reports. The CVS reporting results for the respective video footage assessed were shared with the research team after human observation was completed. Recorded footage was analysed retrospectively using the two methods under comparison (human observation vs. the CVS). One-hour segments of uninterrupted milking shifts (n = 4, two per farm) were selected by the CVS for analysis. The two fixed-hour periods on each farm were separated by 24 hours to assess consistency under similar conditions. The hours were not selected based on any other specific criteria. Footage from the selected shifts (morning for Farm A and afternoon for Farm B) were shared with the human observers who independently assessed the 4 video segments while blinded to the CVS results. The assessment of footage by human observers was conducted independently before discussing discrepancies between human observers. The technical personnel from the CVS company screened the footage for quality before further assessment. If its quality was deemed inappropriate by the research team or the CVS company, then additional footage was collected.

Data from the CVS were recorded in Google Sheets (Google, Mountain View, CA, USA) and exported to Microsoft Excel (version 16.95.4; Microsoft Corp., Redmond, WA, USA) for analysis. Data from human observations were recorded in a Microsoft Excel spreadsheet (version 16.95.4; Microsoft Corp., Redmond, WA, USA) developed in the training phase of the study. For each continuous variable assessed, the average of both observers was defined as the comparison method. For count data, in the case of a discrepancy between human observers, the observers went back to the recording and confirmed the event.

The variables assessed by human observers included prep-lag time (time from first touch to the attachment of the milking unit; min:s; unit of analysis: cow), unit-on time (average of time from first unit attachment to unit removal observed in the cows with the shortest and longest milking time in each turn; min:s; unit of analysis: turn), turn time (time to milk one side of the parlour from gate opening to cows exiting; min:s; unit of analysis: turn), turns per hour, and operator behaviours considered inconsistent with the milking protocol. Variables compared between methods were prep-lag time, unit-on time, turn time, and event count of protocol deviations including specific operator behaviours. No a priori sample size calculation was conducted before data collection. The sample size was based on the available data from the respective farms and milking shifts for which video footage was assessed by CVS. A total of 44 turns were available for evaluation of unit-on time, and 25 turns for turn time, and 214 cows for prep-lag time all including footage for both farms.

Statistical analysis

The CVS data were combined with human observer assessments in Microsoft Excel and analysed using R (version 4.4.1; R Core Team, 2026). Descriptive statistics were reported using the means ± SD of continuous variables, while count data were summarized as frequencies.

Interoperator agreement between human observers for mean unit-on time and agreement between human observers and CVS for all continuous variables were assessed using Lin’s concordance correlation coefficient (CCC) calculated with the epiR package (Stevenson and Sergeant, 2024). Values were classified as poor (<0.40), fair (0.40 to 0.59), good (0.60 to 0.74), or excellent (0.75-1.00) (Cicchetti, 1994). Bland-Altman plots were created to visually assess systematic bias and limits of agreement using the ggplot2 package (Wickham, 2016). Criteria for an acceptable level of discrepancy between the CVS and human observations were not established a priori. For categorical variables, the F-score was calculated to assess event-detection performance. Discordant observations between the CVS-generated reports and human observations were reviewed to establish whether the detected event had occurred. The F-score, defined as the harmonic mean of sensitivity and positive predictive value (Hripcsak and Rothschild, 2005), was then calculated for the CVS using the confirmed events as the reference.

On each farm, two one-hour segments of the milking sessions were evaluated. The mean unit-on time per turn was 5:06 ± 1:54 min on Farm A (164 cows assessed over the two milking sessions; 2 operators assessed) and 4:06 ± 1:37 min on Farm B (96 cows assessed; over two sessions; 1 operator assessed). Farm A completed 10.0 ± 0.15 turns per hour (turn length: 12:01 ± 1:22 min) and Farm B completed 10.4 ± 0.73 turns per hour (turn length: 11:36 ± 1:21 min). The prep-lag time for Farm A was 52 ± 3 seconds, Farm B was 143 ± 6 seconds. A total of 51 events outside the established protocol (Farm A = 28, Farm B = 23 events) were detected by human observers. Detections included cellphone usage during milking (n = 3), manual detachments (n =15), missed forestripping on Farm B (n = 1), and milking towel errors (n = 32), including multiple cows cleaned with a single towel or a missed cow. Six events related to towel misuse (namely, flipping used towels and using the clean side on the next cow) were flagged by the CVS and not detected by the human observers. Upon review of these events by the researchers, all events were confirmed to have been correctly identified by the CVS. Missed pre- or post-milking teat dipping and smoking events did not occur and therefore were not available for assessment. The CCC between human observers for mean unit-on time was (0.998; 95% CI: 0.997 to 0.998) demonstrating excellent interrater agreement. Agreement between the CVS and human observers was excellent for continuous outcomes, with a CCC of 0.999 (95% CI: 0.999-0.999) for mean unit-on time per turn (n = 44 turns), 0.993 (95% CI: 0.991–0.995) for prep-lag time (n = 214 cows), and 0.995 (95% CI: 0.994 to 0.996) for turn time (n = 25 turns). Based on human observers, mean milking time per cow was 0.5 seconds (95% CI: 0.1-1.2) shorter than the CVS estimates with limits of agreement ranging from −4 to +5 seconds (Figure 1a); prep-lag time was 3.8 seconds shorter (95% CI -4.2 to -3.4) than the CVS estimates, with limits of agreement ranging from -9.9 to 2.3 seconds (Figure 1b). The mean turn time was 1.7 seconds longer (95% CI: −0.8 to 4.2 seconds) for human observers compared to the CVS with limits of agreement ranging from -10.4 to 13.8 seconds (Figure 1c). The CVS detected all operator behaviour outside the milking protocol detected by the human observers (n = 51 events) and six additional events not detected by the human observers. The CVS showed a sensitivity of 100% and a positive predictive value of 100%, resulting in an F-score of 1.

In this validation study we compared the CVS against human observers for time variables and events considered inconsistent with the milking protocol. Our findings illustrate that the CVS provides a reliable assessment of prep-lag time, unit-on time, and turn time, as well as the occurrence of deviations from protocol including issues with towel use, manual detachment, and cellphone use.

Wieland and Heuwieser (2025) evaluated prep-lag time, detachments and re-attachments of the same CVS in a single parlour setting operated by two technicians. The CCC for prep-lag time in that study (0.99) was comparable to the one observed in our study. This is relevant as prep-lag time has been associated with shorter milking times and lower odds of short-term teat tissue changes (Singh et al., 2024). Those researchers reported that the CVS was accurate (accuracy = 0.99) in the detection of reattachments and manual detachment compared to human observation.

For prep-lag time, we observed a consistent negative bias, but although CVS overestimated prep-lag time by an average of 3.8 s relative to human observation, this difference represents only 3.1–4.2% of the recommended prep-lag time range for cows milked 3×/d (90–120 s; National Mastitis Council, 2013). The use of a correction factor may have little practical impact on milk production or milking time (Watters et al., 2012). The difference observed between mean unit-on time was less than one second (<1% of the observed unit-on time in previous studies (246–253 s; Singh et al., 2024), which is an accurate reflection of human observation and not operationally important. Similarly, the mean turn time difference was 1.72 seconds between methods. There was one outlier with a difference of 26 seconds between methods in mean turn difference due to a delay in the opening of the gate and the cows entering the parlour unidentified by the CVS. We did not observe substantial bias across the variables assessed, as differences between methods remained relatively consistent across the range of observed prep-lag times, unit-on times, and turn times. Although small differences between methods were observed, their magnitude appears unlikely to be biologically or practically meaningful when considered in the context of recommended on-farm targets. Beyond efficiency metrics, the ability of the CVS to detect adverse events and deviations from protocol provides valuable opportunities to improve milking procedures and staff training. Specific events involving towel misuse were detected by the CVS but not by the human observers. Observational assessments are inherently subject to factors such as attention, distraction, fatigue, and differences in awareness (Faber et al., 2012). In contrast, automated systems can continuously monitor predefined events without being affected by these observer-related factors. The value of this type of technology may not be limited to reproducing human observations but may also lie in providing a consistent and complementary source of information that can reduce the likelihood of events being overlooked by the human eye.

Beyond the scope of the present study, the ability of this technology to continuously and consistently monitor these measures may offer advantages over single-point, in-person audits by allowing deviations in milking practices to be identified over extended periods. This may be particularly relevant because consistency of the milking routine has been associated with greater milk yield (Rasmussen et al., 1990; Wieland et al., 2019), while the repetitive nature of milking may contribute to procedural drift over time (Douphrate et al., 2012). Continuous monitoring could therefore facilitate the identification and correction of deviations that might otherwise go unnoticed. Collectively, these features suggest that automated monitoring may provide opportunities to support improvements in milking consistency, with potential downstream benefits for milk quality, udder health, and employee performance. However, these outcomes were beyond the scope of the present validation study and warrant further evaluation.

Although the CVS substantially reduces the need for manual auditing, some aspects of milking routine assessment and udder health require close-up inspection that cannot be obtained with the current camera placement and field-of-view limitations. In particular, variables such as teat-end condition and cleanliness, as well as post-dipping coverage depend on detailed visualization of the teat, which is not reliably achievable from the available perspective. Others have evaluated the use of machine-learning approaches to automate some of these evaluations (Douphrate et al., 2019; Porter et al., 2021).

Several considerations should be considered when interpreting our findings. The study was conducted on two farms with different parlour configurations (parallel vs. herringbone) and milking protocols which provided an opportunity to evaluate the CVS under distinct commercial conditions. However, the limited number of farms restricts the ability to assess the generalizability of the findings across the broader range of parlour configurations and milking practices. We could not exclude the possibility of a Hawthorne effect, as observations were conducted over a limited number of milking sessions and operators had limited time to become accustomed to being recorded. Although unlikely to affect the validation of the CVS, this may have reduced the frequency of protocol deviations that might otherwise have occurred under routine conditions. The relatively low occurrence of protocol deviations in the study footage could have resulted in an overestimation of agreement between methods. In particular, several events that the CVS is designed to detect did not occur during the observation periods, limiting the opportunity to evaluate its performance across a broader range of protocol deviations. Furthermore, the limited observation time may restrict the generalizability of CVS performance across different days, shifts, and operating conditions. Validation under a larger number of farms, operators, parlour configurations and environmental conditions could help improve the generalizability of our findings.

Under the conditions tested, the evaluated technology demonstrated high agreement with human observations for prep-lag time, unit-on time, turn time, and milking-protocol deviations, while also identifying some events not detected by human observers. The assessed technology may complement human expertise by enabling more efficient and streamlined assessment of milking parlour performance.