Previous studies have shown that reduced carbamoylmethylated lysozyme (RCAM-lysozyme, MW approximately 14.5K) is a substrate and inhibitor (Ki approximately 0.6 microM) of insulin receptor kinase (InsRK) autophosphorylation (Kohanski & Lane, 1986; Lane & Kohanski, 1986). In this study we have prepared a family of defined modified derivatives of RCAM-lysozyme and used them to probe the nature of the substrate and inhibitory sites of InsRK. All open-chain derivatives of lysozyme in which either the tryptophanyl, methionyl, cysteinyl, arginyl, or histidyl side chains were modified served as substrates and were potent inhibitors of InsRK autophosphorylation. This was true whether the substitutions were either hydrophilic or hydrophobic, although the hydrophilic derivatives had a higher inhibitory potency. Tryptic peptides derived from RCAM-lysozyme, however, were inactive as inhibitors, and a mixture of the three cyanogen bromide fragments (containing 12, 24, and 93 amino acids, respectively) was found to be less potent in inhibiting the receptor kinase. Derivatization of either tyrosyl or carboxyl side chains produced derivatives that were neither substrates nor capable of inhibiting receptor autophosphorylation. Derivatives with modified amino groups were substrates for InsRK but were not able to inhibit InsRK autophosphorylation. The present study suggests that (a) unphosphorylated InsRK has a large hydrophilic substrate binding domain and is effectively inhibited by long-chain polypeptides but not by short sequences, (b) some of the amino, carboxyl, and hydroxyphenyl side chains are essential to the inhibitory nature of these polypeptides, and (c) derivatives that fail to inhibit autophosphorylation can still be recognized and phosphorylated by active InsRK.(ABSTRACT TRUNCATED AT 250 WORDS)