The language you were shown inside IBM was the Vienna Development Method notation – internally it was usually called “META-IV”, and outside IBM it later became known as the VDM Specification Language or simply VDM-SL.

What it was
• A model-based formal specification language created in the IBM Vienna Laboratory in the early 1970s.
• Its vocabulary is typed set theory: sets, sequences, maps, Cartesian products, comprehensions, etc.; predicates are written in first-order logic.
• Operations are described with pre-/post-conditions and invariants, so the notation is type-safe and lets a tool generate proof-obligations.
• The ASCII concrete syntax that IBM used was called META-IV (“Meta-four”), because it was the fourth in a line of meta-languages that had been used in Vienna for defining PL/I. Later refinements of the same notation were published as “VDM-SL”.

Tool support inside IBM
IBM Vienna, Hursley, Endicott, Raleigh and several Federal Systems sites had an interactive checker (usually just called “SPEC” or the VDM proof–OF tools) that could
– parse META-IV,
– type-check it,
– generate basic proof obligations,
– and in some groups even produce prototype code in PL/I or Pascal.

Where it was used
CICS redesign work at Hursley, parts of MVS/ESA, database utilities in Poughkeepsie, Cleanroom projects in Federal Systems and some Shuttle flight-software components were all specified in META-IV/VDM during the first half of the 1980s.

What happened to it
• IBM closed the Vienna Laboratory in 1986; most of the VDM team moved to universities (Aarhus, Twente, Belfast, Newcastle) or formed IFAD in Denmark.
• The notation was put into the public domain and standardised as ISO/IEC 13817-1:1996 “VDM-SL”.
• Tools that grew out of the original IBM checker are still available as “VDMTools”, “Overture” and “Crescendo”, maintained by the VDM/Overture open-source community.
• VDM continued to influence other formal methods – most notably Z and Abrial’s B-method – but IBM itself gradually shifted its formal-methods work either to Z (as you saw at Hursley) or, later, to more specialised domain notations.

So the formal specification language you remember is META-IV / VDM-SL, the Vienna Development Method notation, and it still survives (though now outside IBM) as the VDM family of tools and standards.