AVAL.v2: Multi-Assessment Validation and Error Analysis of On-Device Mobility AI for Fall-Risk Screening
ForesightCares, Inc.Charlotte, NCPublished
20 pages · 17 figures ·11 tables · PDF 746 KB
The question · Report 2 of 3 in the AVA Evidence Program
“When do AVA's measurement errors change a clinical decision?”
Key findings
0.386 s
TUG mean absolute error
AVA's TUG times differed from the PT's by 0.386 s on average (95% CI 0.314–0.462; n = 71), with a concordance correlation of 0.989 and 95% limits of agreement of −1.11 to +0.67 s.
Source: Abstract; Table 4, p. 11
90.1%
Same TUG risk zone as PT
AVA placed 64 of 71 participants in the same TUG zone as the PT (95% CI 81.0–95.1; κ = 0.80); all seven disagreements had PT times within 0.6 s of a zone boundary.
Source: Section 5.2, p. 12; Table 7, p. 13
76 of 79
Chair-stand counts matched exactly
AVA's count matched the PT's for 76 of 79 participants (96.2%, 95% CI 89.4–98.7); the other three were each one stand lower, and 78 of 79 were placed in the same risk zone (κ = 0.98).
Source: Abstract; Section 5.3, p. 13
≥90.9%
Balance pass/fail agreement per stance
Pass/fail agreement was 100% for side-by-side, 98.6% for semi-tandem, 90.9% for full tandem and 92.0% for single-leg stance; most balance errors were early terminations that make the result more severe.
Source: Abstract; Table 9, p. 16
Abstract
Vision-based Artificial Intelligence (AI) can automate standardized fall-risk assessments. Before such systems are used for clinical decisions, however, they must be validated with the rigor expected of any measurement instrument. That validation needs to cover more than average error: it must show how errors are distributed, where they change clinical decisions, and which parts of the model cause them. We present AVAL.v2, an extended validation framework for the AVA Mobility on-device AI pipeline. It evaluates three assessments from the Centers for Disease Control and Prevention (CDC) Stopping Elderly Accidents, Deaths, and Injuries (STEADI) toolkit: the Timed Up and Go (TUG), the 30-second chair stand, and, new in this version, the 4-stage balance test.
AVAL.v2 evaluates model output at three levels. The first is continuous measurement agreement, using error metrics, Bland–Altman limits of agreement, concordance and intraclass correlation, and bootstrap confidence intervals. The second is tolerance coverage, the share of participants whose error falls below each threshold. The third is agreement on clinical risk zones, with each error classified by its direction (over-severity or under-severity). The framework also supports component-level ablation of pipeline settings.
We evaluated 80 older adults (72.5% female, aged 54–98), enrolled between February and September 2026, against physical-therapist (PT) scoring. On the TUG (n = 71), AVA achieved a mean absolute error of 0.386 s (95% CI 0.314–0.462), a mean relative error of 3.5%, a concordance correlation of 0.989, and 95% limits of agreement of −1.11 to +0.67 s, with no evidence of proportional bias (slope −0.001, 95% CI −0.033 to 0.031). On the chair stand (n = 79), AVA’s count matched the PT’s exactly for 76 participants and was one stand lower for the other three, and 78 of 79 participants were placed in the correct risk zone (κ = 0.98). On the 4-stage balance test, pass/fail agreement was 100% for side-by-side stance, 98.6% for semi-tandem, 90.9% for full tandem, and 92.0% for single-leg stance. Most balance errors were early terminations of the hold, which make the result more severe rather than less. An ablation of the single-leg stance-check configuration showed a trade-off between exact-duration accuracy and pass/fail agreement. All seven TUG zone disagreements occurred within 0.6 s of a zone boundary.
These results extend our earlier validation to a larger cohort and a new assessment. They also identify specific, measurable targets for improving the model.
Keywords: AI validation · error analysis · pose estimation · fall risk · Timed Up and Go · 30-second chair stand · 4-stage balance · STEADI · Bland-Altman · agreement
Why it matters
AVA Mobility uses a single iPad camera and on-device AI to time and score three standard fall-risk screening tests from the CDC’s STEADI toolkit: the Timed Up and Go, the 30-second chair stand, and the 4-stage balance test. This report compared AVA’s results with a physical therapist’s scoring for 80 older adults from independent-living communities, senior centers, and community activity groups in North Carolina. A typical TUG error was about a third of a second, chair-stand counts matched exactly for 76 of 79 participants, and balance pass/fail agreement ranged from 90.9% to 100% across the four stances.
Average accuracy is only part of the picture. In a screening workflow, what matters is whether an error changes the result someone acts on: the risk zone, or whether a balance stance is passed. The largest TUG error in the study (1.64 s) did not change the participant’s risk zone, while all seven TUG zone changes involved participants whose therapist-measured time was within 0.6 s of a cut-off. On the harder balance stances, most disagreements went in the cautious direction, with AVA ending a hold early and recording a fail the therapist had passed; AVA passed a hold the therapist had failed only twice, both on the single-leg stance.
Looking at errors this way shows where an automated screening result can be relied on and where it needs a second look. The report proposes flagging results that fall close to a zone boundary for clinician review, and sets out specific improvement targets for the next model version. The green, yellow, and red zones are study-defined categories used only to compare AVA’s and the therapist’s classifications; they are not used to predict falls.
Explore the results
Interactive views of numbers published in the report. Hover or focus a mark for details.
Error size vs. clinical impact on the TUG
All seven participants for whom AVA and the PT assigned different TUG risk zones. Each PT time lay within 0.6 s of a cut-off; the zones are study-defined categories used to compare classifications, not to predict falls.
- Physical therapist (reference)
- AVA
- Lower risk (<12 s)
- Intermediate (12–15 s)
- Higher risk (≥15 s)
Every zone disagreement in the study
US under-severity: AVA placed the participant in a lower-risk zone than the PT. OS over-severity: a higher-risk zone.
Try it: when does an error change the zone?
Move the reference time and the size of AVA’s error. The study’s mean absolute TUG error was 0.386 s.
View data tableHide data table
| Participant | PT (s) | AVA (s) | Error | Zone change | Distance to cut-off |
|---|---|---|---|---|---|
| P075 | 12.38 | 11.47 | −0.91 s | Intermediate → Lower risk (US) | 0.38 s |
| P029 | 12.37 | 11.70 | −0.67 s | Intermediate → Lower risk (US) | 0.37 s |
| P013 | 12.35 | 11.79 | −0.56 s | Intermediate → Lower risk (US) | 0.35 s |
| P048 | 12.33 | 11.83 | −0.50 s | Intermediate → Lower risk (US) | 0.33 s |
| P008 | 12.04 | 11.47 | −0.57 s | Intermediate → Lower risk (US) | 0.04 s |
| P019 | 15.60 | 14.75 | −0.85 s | Higher risk → Intermediate (US) | 0.60 s |
| P016 | 11.67 | 12.16 | +0.49 s | Lower risk → Intermediate (OS) | 0.33 s |
Pass/fail agreement by balance stance
Share of participants for whom AVA and the PT agreed on whether each stance was held for the full 10 s. Single-leg results use the 100 ms stance-check configuration.
- Estimate
- 95% Wilson CI
View data tableHide data table
| Item | Estimate | 95% Wilson CI | n | Notes |
|---|---|---|---|---|
| Side-by-side | 100.0% | 94.7%–100.0% | 69 | No disagreements |
| Semi-tandem | 98.6% | 92.5%–99.8% | 72 | 1 over-severity, 0 under-severity |
| Full tandem | 90.9% | 81.6%–95.8% | 66 | 6 over-severity, 0 under-severity |
| Single-leg | 92.0% | 83.6%–96.3% | 75 | 4 over-severity, 2 under-severity |
Selected figures
From the report. Select a figure to enlarge it, or jump to its page in the full report.
Full report
The PDF is the version of record. Search, follow internal links, or jump to any page.
Limitations
- Every result depends on a single reference standard, PT scoring; the PT's own measurement error was not estimated, and without two or more independent PT raters AVA's agreement cannot be compared with inter-rater agreement.
- Balance durations were recorded in whole seconds, which limits how finely duration agreement can be measured.
- The cohort is predominantly female (72.5%), some subgroups are small (nine TUG participants in the red zone; eight aged 90 and over), and sample sizes differ across assessments and stances because of missing data.
- The stance-check ablation was run on the same data used for evaluation, so the preferred configuration should be confirmed on an independent set.
- Secondary engine outputs (TUG phase times, step count, chair-stand cycle duration, sway area) are not yet validated, and the sample was not stratified by site, so performance across environments cannot yet be reported.
Ethics, funding & disclosures
The study was approved by the Advarra Institutional Review Board (protocol Pro00085739) and conducted in accordance with the Declaration of Helsinki. All participants gave informed consent before taking part, including consent to video recording.
This report was prepared by ForesightCares, Inc., which develops AVA.
Responsible use
AVA supports mobility awareness and screening. It does not diagnose conditions, replace clinical evaluation, or serve as a standalone clinical decision tool. Results describe the study population and settings reported here.
Cite this report
ForesightCares, Inc. (2026). AVAL.v2: Multi-Assessment Validation and Error Analysis of On-Device Mobility AI for Fall-Risk Screening (Technical Report No. FC-TR-2026-02). https://insights.foresightcares.com/papers/aval-v2/
@techreport{foresightcares2026avalv2,
title = {AVAL.v2: Multi-Assessment Validation and Error Analysis of On-Device Mobility AI for Fall-Risk Screening},
author = {{ForesightCares, Inc.}},
institution = {ForesightCares, Inc.},
address = {Charlotte, NC},
type = {Technical Report},
number = {FC-TR-2026-02},
year = {2026},
month = oct,
url = {https://insights.foresightcares.com/papers/aval-v2/},
keywords = {AI validation, error analysis, pose estimation, fall risk, Timed Up and Go, 30-second chair stand, 4-stage balance, STEADI, Bland-Altman, agreement}
}
TY - RPRT AU - ForesightCares, Inc. TI - AVAL.v2: Multi-Assessment Validation and Error Analysis of On-Device Mobility AI for Fall-Risk Screening PY - 2026 DA - 2026/10/16 PB - ForesightCares, Inc. CY - Charlotte, NC M1 - FC-TR-2026-02 M3 - Technical Report UR - https://insights.foresightcares.com/papers/aval-v2/ L1 - https://insights.foresightcares.com/papers/aval-v2/aval-v2.pdf AB - Vision-based Artificial Intelligence (AI) can automate standardized fall-risk assessments. Before such systems are used for clinical decisions, however, they must be validated with the rigor expected of any measurement instrument. That validation needs to cover more than average error: it must show how errors are distributed, where they change clinical decisions, and which parts of the model cause them. We present AVAL.v2, an extended validation framework for the AVA Mobility on-device AI pipeline. It evaluates three assessments from the Centers for Disease Control and Prevention (CDC) Stopping Elderly Accidents, Deaths, and Injuries (STEADI) toolkit: the Timed Up and Go (TUG), the 30-second chair stand, and, new in this version, the 4-stage balance test. AVAL.v2 evaluates model output at three levels. The first is continuous measurement agreement, using error metrics, Bland–Altman limits of agreement, concordance and intraclass correlation, and bootstrap confidence intervals. The second is tolerance coverage, the share of participants whose error falls below each threshold. The third is agreement on clinical risk zones, with each error classified by its direction (over-severity or under-severity). The framework also supports component-level ablation of pipeline settings. We evaluated 80 older adults (72.5% female, aged 54–98), enrolled between February and September 2026, against physical-therapist (PT) scoring. On the TUG (n = 71), AVA achieved a mean absolute error of 0.386 s (95% CI 0.314–0.462), a mean relative error of 3.5%, a concordance correlation of 0.989, and 95% limits of agreement of −1.11 to +0.67 s, with no evidence of proportional bias (slope −0.001, 95% CI −0.033 to 0.031). On the chair stand (n = 79), AVA’s count matched the PT’s exactly for 76 participants and was one stand lower for the other three, and 78 of 79 participants were placed in the correct risk zone (κ = 0.98). On the 4-stage balance test, pass/fail agreement was 100% for side-by-side stance, 98.6% for semi-tandem, 90.9% for full tandem, and 92.0% for single-leg stance. Most balance errors were early terminations of the hold, which make the result more severe rather than less. An ablation of the single-leg stance-check configuration showed a trade-off between exact-duration accuracy and pass/fail agreement. All seven TUG zone disagreements occurred within 0.6 s of a zone boundary. These results extend our earlier validation to a larger cohort and a new assessment. They also identify specific, measurable targets for improving the model. KW - AI validation KW - error analysis KW - pose estimation KW - fall risk KW - Timed Up and Go KW - 30-second chair stand KW - 4-stage balance KW - STEADI KW - Bland-Altman KW - agreement ER -
See AVA in your setting
Talk with our team about bringing camera-based mobility assessment to your clinic, community, or health plan, or get notified when we publish the next report.