Skip to content

Invest1 publisher2 min readPublished

Education Department moves the edtech test from activation to evidence at renewal

New Education Department guidance asks districts to judge instructional software on demonstrated learning outcomes and to make vendors publish independent evaluations. Two numbers from a Tennessee tutor trial explain why.

The Investor · Invest desk

Photograph accompanying Education Department moves the edtech test from activation to evidence at renewal
Photo: govtech.com

What happened

  • MIT reported last year that 95% of corporate generative-AI pilots produced no measurable return, the figure Fortune uses to describe dashboards showing near-total activation while output stays flat.
  • The Department of Education's guidance last week separated recreational technology from instructional technology and said screen time is a poor measure of educational value.
  • The same guidance asks districts to judge products by demonstrated learning outcomes instead of usage, to build evidence into renewals as well as purchases, and to expect vendors to publish independent evaluations.
  • In a two-year randomized trial across 18 Tennessee middle schools, 96% of students tried the AI tutor at least once and then turned to it in 17% of the moments when they had gotten something wrong.

Compiled by The InvestorSomething wrong?How this is made

Why it matters

  • exposure A vendor whose renewal turns on published outcome evidence carries the cost of producing it, and the smallest sellers are the least able to fund a trial that would satisfy a district.
  • cost Whatever a district saves by adopting a free AI tutor comes back as a staffing and timetable bill, because the last wave of tools needed paid coaches and protected school-day time to reach students at all.
  • contradiction The 95% belongs to corporate pilots, so the claim that American schools repeated the failure nationally rests on Fortune's pattern argument, with no district spending or outcome figures in the piece to test it.

The complement of the Tennessee trial's 17% is 83%: in more than four of every five moments when a student had gotten something wrong, the tutor stayed closed [5]. A vendor's dashboard reports the trial rate. A renewal decided on demonstrated learning outcomes is decided on the other number [3].

There are AI tutors available now, some of them free, patient and competent at eleven at night [10]. Moving the 83% costs something else. In Fortune's account of the post-pandemic career-exploration wave, schools had to carve out time during the school day and pay career coaches to get students through the tools, and identifying a promising career meant little unless somebody had brought employers to the table to create an internship, apprenticeship or an actual job at the other end [9].

Responsible design, the Department wrote, is "the floor" [6]. Fortune reads the document as the beginning of edtech's accountability era and says Washington last week started asking schools for proof [12][11].

The renewal language is where this touches revenue. A purchase can be won with a pilot and a demo; a renewal judged on outcomes has to be won with data somebody outside the company gathered, and the guidance asks vendors to publish exactly that, along with a disclosure of what their products cannot do [3].

MIT's 95% covers corporate generative-AI pilots [1]. One pilot in twenty produced a measurable return [7]. Fortune's case is that American schools ran the same play at national scale, buying interactive whiteboards, laptop carts and pandemic-funded platforms and activating each without the training and instructional model built around it [8]. That case is an argument from pattern, and the piece does not report district spending totals or student outcome data for AI in US schools [14].

In my view the guidance moves a vendor's cost of sale before it moves a student's score. An independent evaluation is a fixed cost paid once per product; the training and the protected class period are recurring costs paid per district per year, and Fortune calls the constraint "everything nobody builds around it" [15]. Two things would show me wrong. Districts renewing on usage reports straight through the next budget cycle, or evaluations turning out cheap enough that publishing one stops separating vendors at all.

What to watch

  • Whether districts write outcome-evidence clauses into renewals in the next budget cycle or keep signing off on usage reports.
  • Whether the Department follows this guidance with anything binding on how schools buy technology.
  • Whether any AI tutor vendor publishes an independent evaluation with outcome data, and what running the trial cost.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories