lib: rsa: add support to other openssl engine types than pkcs11

There are multiple other openssl engines used by HSMs that can be used to
sign FIT images instead of forcing users to use pkcs11 type of service.

Relax engine selection so that other openssl engines can be specified and
use generic key id definition formula.

Signed-off-by: Vesa Jääskeläinen <vesa.jaaskelainen@vaisala.com>
Cc: Tom Rini <trini@konsulko.com>

doc/uImage.FIT/signature.txt

@@ -388,8 +388,8 @@ Test Verified Boot Run: signed config with bad hash: OK
 Test passed
 
 
-Hardware Signing with PKCS#11
------------------------------
+Hardware Signing with PKCS#11 or with HSM
+-----------------------------------------
 
 Securely managing private signing keys can challenging, especially when the
 keys are stored on the file system of a computer that is connected to the
@@ -402,14 +402,43 @@ them perform the signing. PKCS#11 is standard for interfacing with these crypto
 device.
 
 Requirements:
-Smartcard/USB token/HSM which can work with the pkcs11 engine
+Smartcard/USB token/HSM which can work with some openssl engine
 openssl
+
+For pkcs11 engine usage:
 libp11 (provides pkcs11 engine)
 p11-kit (recommended to simplify setup)
 opensc (for smartcards and smartcard like USB devices)
 gnutls (recommended for key generation, p11tool)
 
-The following examples use the Nitrokey Pro. Instructions for other devices may vary.
+For generic HSMs respective openssl engine must be installed and locateable by
+openssl. This may require setting up LD_LIBRARY_PATH if engine is not installed
+to openssl's default search paths.
+
+PKCS11 engine support forms "key id" based on "keydir" and with
+"key-name-hint". "key-name-hint" is used as "object" name and "keydir" if
+defined is used to define (prefix for) which PKCS11 source is being used for
+lookup up for the key.
+
+PKCS11 engine key ids:
+   "pkcs11:<keydir>;object=<key-name-hint>;type=<public|private>"
+or
+   "pkcs11:object=<key-name-hint>;type=<public|private>",
+
+Generic HSM engine support forms "key id" based on "keydir" and with
+"key-name-hint". If "keydir" is specified for mkimage it is used as a prefix in
+"key id" and is appended with "key-name-hint".
+
+Generic engine key ids:
+  "<keydir><key-name-hint>"
+or
+  "<key-name-hint>"
+
+As mkimage does not at this time support prompting for passwords HSM may need
+key preloading wrapper to be used when invoking mkimage.
+
+The following examples use the Nitrokey Pro using pkcs11 engine. Instructions
+for other devices may vary.
 
 Notes on pkcs11 engine setup:
 

lib/rsa/rsa-sign.c

@@ -141,6 +141,15 @@ static int rsa_engine_get_pub_key(const char *keydir, const char *name,
 			snprintf(key_id, sizeof(key_id),
 				 "pkcs11:object=%s;type=public",
 				 name);
+	} else if (engine_id) {
+		if (keydir)
+			snprintf(key_id, sizeof(key_id),
+				 "%s%s",
+				 keydir, name);
+		else
+			snprintf(key_id, sizeof(key_id),
+				 "%s",
+				 name);
 	} else {
 		fprintf(stderr, "Engine not supported\n");
 		return -ENOTSUP;
@@ -252,6 +261,15 @@ static int rsa_engine_get_priv_key(const char *keydir, const char *name,
 			snprintf(key_id, sizeof(key_id),
 				 "pkcs11:object=%s;type=private",
 				 name);
+	} else if (engine_id) {
+		if (keydir)
+			snprintf(key_id, sizeof(key_id),
+				 "%s%s",
+				 keydir, name);
+		else
+			snprintf(key_id, sizeof(key_id),
+				 "%s",
+				 name);
 	} else {
 		fprintf(stderr, "Engine not supported\n");
 		return -ENOTSUP;

tools/mkimage.c

@@ -105,7 +105,7 @@ static void usage(const char *msg)
 		"          -F => re-sign existing FIT image\n"
 		"          -p => place external data at a static position\n"
 		"          -r => mark keys used as 'required' in dtb\n"
-		"          -N => engine to use for signing (pkcs11)\n");
+		"          -N => openssl engine to use for signing\n");
 #else
 	fprintf(stderr,
 		"Signing / verified boot not supported (CONFIG_FIT_SIGNATURE undefined)\n");