[Cryptography] How to find hidden/undocumented instructions

Bill Frantz frantz at pwpconsult.com
Mon Aug 28 16:33:02 EDT 2017


On 8/27/17 at 5:14 AM, leichter at lrw.com (Jerry Leichter) wrote:

>There were other hacks in that era based on the vulnerability 
>of accessing parameters to system calls directly from user 
>memory (e.g., modifying the value from some kind of parallel 
>thread after the OS had validated it but before it used it).  
>Eventually we learned to immediately copy parameters into 
>system space, and validate and use them from there.

The IBM 370 had what I think was a design bug involving data 
which spans pages. There was a quite useful instruction, TR 
(translate) which would take a string of up to 256 bytes and 
translate it in place to new values in the bytes. It used a 
table of 256 bytes which was indexed by the input value. The 
table entry was the output value. This instruction was left over 
from the 360, so it wasn't designed for paged memory.

If the table spanned a page boundary, the hardware did a "dry 
run" where it scanned the input string to ensure that all the 
table entries were mapped. If all the entries were mapped it 
proceeded to execute the instruction, otherwise it raised a page fault.

If some other functional unit (another CPU or an I/O channel) 
changed the input string to reference a table entry that wasn't 
mapped between the dry run and the execution, the hardware 
generated a machine check. It couldn't start over, because the 
input string had been partially modified. It couldn't continue 
because the needed table entry was not addressable. It was up a creek.

I never saw, or heard of this machine check occurring in real 
life, but you could, in theory, write a program to crash a 370 
in a way that was supposed to be impossible to do from a program.

Cheers - Bill

-----------------------------------------------------------------------
Bill Frantz        |The nice thing about standards| Periwinkle
(408)356-8506      |is there are so many to choose| 16345 
Englewood Ave
www.pwpconsult.com |from.   - Andrew Tanenbaum    | Los Gatos, 
CA 95032



More information about the cryptography mailing list