Amazon Memo Calls Token Usage and Other AI Input Metrics "Deeply Flawed," Then Tells Managers to Keep Using Them "With Care and Judgement"
Amazon published on October 27 a memo from its engineering leadership telling software engineers and managers that input metrics such as token usage and AI tool adoption are "deeply flawed" and that "tokenmaxxing is at odds with frugality," while saying the metrics still correlate with results and can be used "with care and judgement." CNBC reports the memo first circulated internally in September and that Amazon says it has not capped employees' token usage.
Amazon's memo does not abandon AI usage metrics; it keeps them and warns against using them as targets. That is a narrower position than "token counts don't matter," and it leaves the numbers it calls flawed in use as engineers and managers think about their careers and their organizations' performance.
What the memo says#
Amazon published the memo on its website on October 27 under the byline of Terk Terkowitz, a vice president and distinguished engineer in its Worldwide Amazon Stores business. It is addressed to software development engineers and managers and accompanies a revised role guideline for the job.
The memo says Amazon has sometimes measured AI work with input metrics, "like token usage or adoption of AI tools," or throughput metrics such as code changes and reviews. "We are all aware of how deeply flawed these metrics are, primarily because they are disconnected from what matters," it says. It warns that using them as targets could raise costs, adding "(tokenmaxxing is at odds with frugality)."
It then says "these input metrics correlate well with the outputs that matter" and that using them "with care and judgement" is appropriate. It recommends paying special attention to the 10th and 90th percentiles of AI adoption: teams that are not using AI effectively, and teams that use it heavily, to check whether their delivery has risen to match.
"We believe that it is best to measure, and be measured by, long-term outcomes," the memo says. It also says: "If your work does not benefit from current AI tooling, that is a finding, not a failing."
Why Amazon is addressing tokenmaxxing#
CNBC reports that the memo circulated internally in September before Amazon posted it. An Amazon spokesperson told CNBC the company has not capped employees' token usage. CNBC also reports that Amazon shut down an internal leaderboard in May after finding that employees were tokenmaxxing, using AI as heavily as possible to climb the rankings.
Marc Brooker, a vice president and distinguished engineer at Amazon Web Services, told CNBC that Amazon has looked to measures such as "feature velocity," including how many launches and successful production changes teams make.
Limits#
- We quote the memo as Amazon posted it, which spells "judgement"; CNBC's article renders one phrase as "with care and judgment."
- We could not compare the posted memo with the version that circulated internally in September.
- The memo does not say how much weight token usage carries in performance reviews.