Short answer
Atari handed Digital Research an advance of about a quarter-million dollars and then paid a running royalty of roughly one U.S. dollar for every ST it shipped, with a contractual ceiling that put the total value of the deal a little above two million dollars. In other words, Jack Tramiel got the whole of GEM, CP/M-68 K (“GEMDOS”) and the desktop for about US $1 per machine.

Where the figure comes from
Contemporary trade-press stories covered the agreement while it was being negotiated:

• Computerworld, 18 February 1985, p. 6 – “Atari’s New 68000 Micros Will Use CP/M and GEM”:
 “Sources close to the companies say Digital Research will receive a $250,000 licence fee and a one-dollar royalty on each unit Atari ships, the royalty stopping after the first two-million dollars has been paid.”

• InfoWorld, 11 March 1985, p. 15 – “Atari, Digital Research sign agreement”:
 “…an initial payment of ‘about a quarter of a million’ dollars and a 2-million cap is reached.”

• New York Times, 29 April 1985, section D, p. 6 – “New Atari Computer to Use GEM Interface”:
 “Digital Research said the pact calls for a 1 on each computer shipped, the royalty ending when gross payments reach $2 million.”

Terms that were spelled out
The contract, signed in January 1985, covered the following points (summarised from the above reports and from later interviews with DR engineer Lee Lorenzen and Atari’s Leonard Tramiel in STart, Aug 1986):

• What Atari got
 – A perpetual, non-exclusive, worldwide binary licence to ship:
  • GEM VDI, AES, GDOS, Desktop
  • CP/M-68 K (renamed “GEMDOS” in the Atari world)
 – Source code for internal use; Atari could make bug fixes and 68000-specific optimisations but could not pass the source to third parties.
 – Delivery schedule: first beta at the end of March 1985, gold master 15 June 1985 – barely five months after signing.

• What Digital Research got
 – US 1 royalty on every 68000-based computer that shipped with the code, payable quarterly, stopping once total royalties reached US 2 million including the advance).
 – All enhancements done by Atari had to be returned to DR at no charge; DR could fold them back into later OEM versions of GEM.

• Other clauses
 – “GEM – Licensed from Digital Research” and copyright notice had to appear on the boot screen or in the desktop “About…” box.
 – Atari could not sublicense the code to other manufacturers.
 – Either party could terminate for breach, but if Atari had already paid the full $2 million the licence became irrevocable.

How much did Atari actually end up paying?
Estimates of total ST family sales run from 1.8 million to just over 2 million units. That is enough to hit the 1.75 million spread over the first few years of production.

Why this was “cheap”
Even if Atari reached the ceiling, an overall cost in the two-million-dollar range is tiny compared with:

• the US $50–100 million Apple is reckoned to have spent on the original Macintosh system software, or
• the engineering cost (and calendar time) of writing a new multitasking, GUI-based operating system from scratch in 1985.

For Jack Tramiel it was a classic “pay a small amount now, pay as you go later, no royalty once volume is high” deal—exactly the sort of bargain he had prided himself on since his Commodore days.

So the answer is: roughly 1 per unit until a $2 million cap was reached; after that Atari owed Digital Research nothing.