Predicting Scholar Attrition from Family Surveys: What to Ask, When to Ask, and Who to Prioritize

This report evaluates which family survey questions predict scholar attrition. Families receive surveys asking how satisfied they are with their school and whether they intend to switch schools. Some parents also provide open-ended responses, which have been categorized into topics. Attrition is defined as a scholar who left within 90 days of the survey date. Surveys were collected from March 2024 through Dec 2025 across seven family survey waves, one engagement survey (Spring 2024), and three open-text family feedback surveys (SY25-26).

The switch intent question predicts attrition more accurately than the satisfaction question. For NPS, we cannot draw conclusions about its predictive accuracy relative to switch intent or satisfaction due to limited data. Coded open-text topics add signal beyond either closed-end predictor on its own. Among the individual topics, mentions of unmet special needs, disrespectful staff behavior, and overly strict discipline carry the strongest attrition signal. Combining switch intent, satisfaction, and open-text topic mentions further identifies which family profiles carry the highest attrition risk. Results suggest families who are considering switching, rate their experience negatively, and raise concerns about staff conduct or unmet student needs are among the highest-risk groups.

Research questions

Do families who respond to the survey have lower attrition compared to non-responders?

Overall responded vs non-responded

We compared scholar attrition rates between families who answered the family survey and those who did not; in 5 of the 6 waves, families who answered the survey had lower attrition than those who did not, and the only exception was the Aug 2024 (GrK–8) wave, where respondents had higher attrition than non-responders. Plausible explanation can be the pool of 447 respondents combined with August-specific selection effects; we treat this as noise.

By satisfaction response

Respondent subpopulation broken down by collapsed satisfaction: Positive (Very Positive + Positive), Neutral, and Negative (Negative + Very Negative). Aug 2024 (GrK–8) used a 3-level scale (great / okay / not happy).

By switch response

Respondent subpopulation broken down by switch intent: No, Unsure, and Yes. Mar 2025 (GrK–3) and May 2025 (GrK–3) use a 2-level scale (No / Yes). Sept 2025 (GrK–8) uses a 3-level scale (No / Unsure / Yes).

Among respondents, does satisfaction or switch better predict attrition?

To compare satisfaction and switch as predictors, we fit a logistic regression for each, then compared the two models by their AUC (area under the receiver operating characteristic curve; see the Appendix 1 for an explanation), which measures how well each single predictor discriminates between scholars who stayed and those who left. Results show that the switch question (AUC = 0.64) has stronger predictive power than the satisfaction question (AUC = 0.60).

AUC measures how well a predictor ranks scholars by attrition risk: 0.50 = random guessing, higher = better.

Cross-wave comparison: how do survey conditions collectively affect predictive power?

Timing in the school year, question wording, grade population, and response scale all shift across survey waves and often change together, so observed AUC differences cannot be attributed to a single cause. The wave-by-wave AUC tables below show how each survey version performed.

Satisfaction framing comparison

Survey	Question	Response levels	AUC
Aug 2024 (GrK–8)	How was your first week of school?	3-level (great / okay / not happy)	0.52
Jan 2025 (GrK–3)	How would you rate your experience at Success Academy?	5-level (Very Positive → Very Negative)	0.64
Mar 2025 (GrK–3)	How would you rate your experience at Success Academy? Select one.		0.60
May 2025 (GrK–3)	How would you describe your experience at Success Academy over the past two months? Select one.		0.65
Sept 2025 (GrK)	How do you feel about your first week at Success Academy?		0.63
Sept 2025 (Gr1–8)	Please rate your SA experience so far this school year:		0.64
Dec 2025 (GrK–8)	How would you rate your family's experience at your school this year?		0.59

Green survey names scored significantly higher AUC than the red one(s).

Switch framing comparison

Survey	Question	Response levels	AUC
Mar 2025 (GrK–3)	Are you thinking about withdrawing your scholar(s) from Success Academy? Select one.	Binary (Yes / No)	0.68
May 2025 (GrK–3)	Are you exploring other school options for SY25-26? Select one.	Binary (Yes / No)	0.70
Sept 2025 (GrK)	Are you feeling confident about continuing at Success Academy for the full year?	3-level (Yes / Mostly hesitations / No)	0.59
Sept 2025 (Gr1–8)	Do you feel confident SA is right for your scholar for the full year? Select one.	3-level (Yes / Mostly hesitations / No)	0.65

Green survey names scored significantly higher AUC than the red one(s).

Part I Closed-End Family Survey Summary

Families who answer the family survey have lower attrition than non-responders in 5 of the 6 waves.
Among respondents, the switch question is a stronger predictor of attrition than the satisfaction question.
Predictive accuracy varies across waves; survey timing, wording, response scale, and grade range all change together, so currently we are unable to attribute differences to a single cause (need A/B test to find out the cause).
Results suggest timing is a stronger driver of predictive power than wording. Jan 2025 and Mar 2025 used nearly identical wording yet differed in predictive power; the Sept 2025 GrK and Gr1–8 surveys used different wording but performed almost identically.

In the engagement survey, do NPS, satisfaction, or switch best predict attrition?

The Spring 2024 family engagement survey included NPS, satisfaction, and switch-intent items. All three predictors show attrition signal, but with only 37 attrition events in this sample, no single predictor clearly outperforms the others.

Note: this analysis uses a 180-day window for attrition. The engagement survey dataset has only 231 responses, and with the 90-day window used elsewhere in the report only 1 attrition event was observed, too few for an AUC estimation. Extending the window to 180 days yields 37 events.

Which topics signal attrition risk?

The table below lists the top 10 topics by attrition rate. For each topic it shows how many families mentioned it, how many of those families' scholars attrited, and the resulting attrition rate (families whose scholar attrited divided by families who mentioned the topic).

Topic	Families who mentioned	Scholars who attrited	Attrition rate
Challenges with ELL/language barriers	102	11	10.8%
Heavy backpacks burden students	29	2	6.9%
Scholars' special needs/accommodations not met	472	32	6.8%
Staff disrespectful and unprofessional toward families	334	22	6.6%
Difficult transition to new school environment	124	8	6.5%
Experience worsened	246	15	6.1%
Lack of SL accountability	189	11	5.8%
Breakfast/lunch/snack time insufficient	156	9	5.8%
Unfair discipline policies/application of policies	316	18	5.7%
Teachers disrespectful and unprofessional toward students	802	39	4.9%

Part III Open-End Text Summary

Topics most strongly associated with attrition include challenges with ELL/language barriers, scholars’ special needs/accommodations not met, and disrespectful staff toward families.
Topics most associated with retention center on gratitude, visible academic progress, and a structured learning environment.

Do open-text topics predict attrition as well as closed-end survey questions?

To compare the predictive value of open-text responses against closed-end survey items, we built a combined model for each side. On this comparison, the closed-end model (AUC 0.66) outperforms the open-end model (AUC 0.55). Two closed-end questions (satisfaction and switch intent) match or exceed the predictive power of coded open-text topics. The closed-end items are more efficient: they require less processing and deliver equal or better attrition signal.

What combinations of factors identify the highest-risk families?

The analyses above examine each predictor in isolation. Here we cross-tabulate three dimensions: switch intent, satisfaction rating, and whether a family mentioned each of the 10 highest-attrition-rate topics from Part III. Each row in the table below represents a group of families who share the same switch intent, satisfaction level, and mentioned a specific topic.

This table helps prioritize proactive outreach: families who appear in the highest-attrition rows share the same profile, for example, families who expressed switching intent, rated their experience negatively, and raised a specific concern in open-ended feedback.

Appendix 1: What “AUC” means

Switch intent	Satisfaction	Topic mentioned	N	Attritions	Attrition rate
Unsure about switching	Positive	Challenges with ELL/language barriers	10	4	40.0%
Considering switching	Negative	Staff disrespectful and unprofessional toward families	27	7	25.9%
Considering switching	Negative	Unfair discipline policies/application of policies	21	5	23.8%
Considering switching	Negative	Teachers disrespectful and unprofessional toward students	83	16	19.3%
Considering switching	Negative	Lack of SL accountability	26	5	19.2%
Considering switching	Negative	Scholars' special needs/accommodations not met	42	8	19.0%
Considering switching	Negative	Experience worsened	27	3	11.1%
Unsure about switching	Positive	Teachers disrespectful and unprofessional toward students	39	4	10.3%
Not considering switching	Negative	Scholars' special needs/accommodations not met	43	4	9.3%
Not considering switching	Positive	Challenges with ELL/language barriers	44	4	9.1%

AUC (area under the ROC curve) is a single number that answers: If the model is asked to sort scholars from highest to lowest risk of attrition, how often does it place people who actually leave ahead of people who stay? A higher AUC means better ranking; 0.50 is like random guessing. This report uses AUC to compare survey-based predictors (e.g. satisfaction vs. switch).

Imagine a tiny group: 11 scholars, with exactly 1 who truly attrits. Everyone else is retained. The model gives each scholar a risk score; we sort the table from highest to lowest. In this toy setup, where that one scholar shows up in the list drives the AUC. The three examples below use the same scholars and the same real outcome; only the model’s risk scores and ordering change, like comparing a weaker model to a stronger one.

Example: AUC = 0.50 (like random guessing)

Here the model places Scholar C in the middle of the list. The scholar who really left is surrounded by people who stayed, with no useful separation. This is what you would expect if the model were no better than chance.

Rank (risk: high to low)	Scholar	Attrited?
1	A	No
2	B	No
3	D	No
4	E	No
5	F	No
6	C	Yes
7	G	No
8	H	No
9	I	No
10	J	No
11	K	No

Example: AUC = 0.60 (weak but real signal)

Scholar C moves a bit up the risk list, but is still not near the very top. The model picks up a hint, but it is easy to miss this family in day-to-day work.

Rank (risk: high to low)	Scholar	Attrited?
1	A	No
2	B	No
3	D	No
4	E	No
5	C	Yes
6	F	No
7	G	No
8	H	No
9	I	No
10	J	No
11	K	No

Example: AUC = 0.80 (stronger signal)

Scholar C is now in the top few on predicted risk, while most who stayed sit lower. If you only had capacity to check in on the 3 riskiest cases, you would include the family that actually left.

Rank (risk: high to low)	Scholar	Attrited?
1	A	No
2	B	No
3	C	Yes
4	D	No
5	E	No
6	F	No
7	G	No
8	H	No
9	I	No
10	J	No
11	K	No

Why this matters in practice

Suppose you could only reach out to the top 3 highest-risk families. In the AUC = 0.80 example, Scholar C (the one who really left) is in that top 3, so a proactive call list would catch them. In the AUC = 0.60 example, Scholar C is not in the top 3, so the same list would miss them. Higher AUC means the risk ranking is more often useful for prioritization, though not a guarantee for every individual, but a better compass when resources are limited.

Note: The analysis in this report uses many thousands of scholars and many who attrit, not 11. This simplified story is for intuition only; real AUC is computed over all attrited–retained pairs in the data, not a single case.

Appendix 2: Full risk-combination table

All switch × satisfaction × topic combinations with at least one observation. Rows in grey are based on fewer than 10 families (†) and should be interpreted cautiously.