A symptom checker that just returns a list of possible conditions feels like a feature. It is actually a decision-support system operating without a decision-support architecture, and that gap shows up fast once real patient volume hits it.
Override rates climb when clinicians can’t see why the system flagged something as urgent, and staff stop trusting outputs they can’t explain to a patient. Alert fatigue sets in when every borderline case gets flagged the same way as a genuine emergency. Not a feature gap. A design gap.
The fix isn’t a better chatbot. It’s a system that scores severity, explains its reasoning, and knows when to defer to a human.