PeterLi · February 8, 2023 20:53 · tianskylan · Sep 9, 2021 · bartosz-treeline · Sep 27, 2022
diff --git a/Encrypted Realm with secret in keychain in app supporting background fetch intermittently causes realm to fail to initialize and crash app at try! - SOLVED b/Encrypted Realm with secret in keychain in app supporting background fetch intermittently causes realm to fail to initialize and crash app at try! - SOLVED
 This is like a drop-in replacement for realms example swift project enabling encryption using the function getKey().

 https://github.com/realm/realm-cocoa/blob/master/examples/ios/swift/Encryption/ViewController.swift

 The following swift function is a drop in replacement, it documents the painful experience of an issue
 which has been unsolved for years till crashlytics revealed a very useful hint, which lead to the root cause
 and fix.

 Comments included for education on what the symptoms were, the cause and fix and other notes.

 private static func getEncryptionKey() -> NSData {
 		/*
 			PLI @2019-07-01
 			identified from crashlytics the "Realm file decryption failed" error...
 			https://github.com/realm/realm-cocoa/issues/5615

 			https://forums.developer.apple.com/thread/114159 - KeyChain SecItemAdd return -25308 when app is launched
 			https://forums.developer.apple.com/thread/78372 - Access Keychain When iPhone Locked
 		
 			in short, add kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock
 		
 			Disgnosis Explained:
 			====================
 			Symptoms: intermittent/unexplained crashes (force unwrap try! Realm()) when realm for some reason would
 		              not properly initialise, when it should initialise perfectly fine.
 		
 			Thoery of contributing factors:
 				- encrypted realm
 				- application supports background updates
 				- device is in locked state
 				- *** application is not running, but is launched by OS to initiate a background update ***
 					- This fact alone is VERY hard, if not impossible to replicate, as the simulator only allows
 					  simulate background fetch when the app is being debugged... but what we need is
 					  app not running, and device locked, and OS launches our app!
 					  Even if you did replicate it by not running app and leaving overnight or for a long duration
 					  one would not know when the OS calculated 'launch app to do fetch' will occur, as the algorithm
 					  of the OS is not published. So one could be waiting for ages...
 		
 			Theory of sequence of events to cause crash:
 				- when the device is locked, keychain requests if not configured correctly will likely
 				  return errSecInteractionNotAllowed (-25308), which indicates a UI cannot be presented to prompt to unlock
 				  and thus gain access. (unless kSecAttrAccessibleAfterFirstUnlock attribute set)
 				- as a result the code would fall through (as it didn't check for errSecInteractionNotAllowed error)
 				  and only checked for errSecSuccess, and run the code assuming the entry did not exist
 				  and thus create a new key and add to the keychain.
 				- the code would have asserted, as it would have got a duplicate entry error, however being production
 				  code asserts are not compiled in. And since this bug was very intermittent, we never saw it
 				  in the debug code version where asserts were compiled in.
 				- because in production code the asserts was compiled out, the result would be a new key was created
 				  and returned to the caller, which would of course be a different encryption key as that
 				  originally used to encrypt the realm DB, and thus realm would correctly thrown an
 				  exception "Realm file decryption failed".
 				- thus resulting in a Realm object not being instantiated, and hence forcing to resolve try! would crash.
 		
 			Solution:
 				- in order to resolve the problem we need to tag the entry with accessible attribute:
 					kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock
 					We dont want to limit to only current device, as user may backup and restore, and we
 					would still like the DB to work.
 					We dont want to have only accessible on unlock, as then it will not work when performing
 					background tasks.
 				- for compatibility, we need to delete/update the entry with the missing attribute to then include
 					the required attribute, to avoid crashes moving forwards and not require the user to delete
 					and re-install the app (since keychain persists per app items for different durations when
 					app deleted depending on iOS version)
 				- a query for item with or without kSecAttrAccessibleAfterFirstUnlock attribute will find item
 				  originally created with kSecAttrAccessibleAfterFirstUnlock attribute set. BUT a query will only
 				  find an item created without kSecAttrAccessibleAfterFirstUnlock if the query does not have
 				  the attribute kSecAttrAccessibleAfterFirstUnlock set.
 				  UPDATE: OS automatically adds attribute default of kSecAttrAccessibleWhenUnlocked if none
 					  aupplied by caller on adding to keychain. Thus if you query/fetch with attribute
 					  not supplied or as kSecAttrAccessibleWhenUnlocked, then a match will result.

 		
 			Notes:
 				- interestingly in realms keychain_helper.cpp there is a is a function build_search_dictionary()
 				  which conveniently does pass in the appropriate attributes:

 					CFDictionaryAddValue(d.get(), kSecAttrAccessible, kSecAttrAccessibleAlways);
 		
 				  We do NOT pass kSecAttrAccessibleAlways, as this is deprecated in iOS 13, and is probably considered
 				  less secure, so we use the next best thing, kSecAttrAccessibleAfterFirstUnlock
 		
 		*/
 		
 		// Identifier for our keychain entry - should be unique for your application
 		let keychainIdentifier = kKeychainIdentifier /* REPLACE THIS WITH YOUR OWN IDENTIFIER */
 		let keychainIdentifierData = keychainIdentifier.data(using: String.Encoding.utf8, allowLossyConversion: false)!
 		
 		// First check in the keychain for an existing key
 		var query: [NSString: AnyObject] = [
 			kSecClass: kSecClassKey,
 			kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
 			kSecAttrKeySizeInBits: 512 as AnyObject,
 			kSecReturnData: true as AnyObject,
 			kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock	/* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
 		]
 		
 		// To avoid Swift optimization bug, should use withUnsafeMutablePointer() function to retrieve the keychain item
 		// See also: http://stackoverflow.com/questions/24145838/querying-ios-keychain-using-swift/27721328#27721328
 		var dataTypeRef: AnyObject?
 		var status = withUnsafeMutablePointer(to: &dataTypeRef) { SecItemCopyMatching(query as CFDictionary, UnsafeMutablePointer($0)) }
 		if status == errSecSuccess {
 			return dataTypeRef as! NSData
 		}
 		
 		////////////////////////////////////////////////
 		////// BUG FIX COMPATIBILITY CODE MI v5.2 //////
 		////////////////////////////////////////////////
 		let queryWithCompatibility: [NSString: AnyObject] = [
 			kSecClass: kSecClassKey,
 			kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
 			kSecAttrKeySizeInBits: 512 as AnyObject,
 			kSecReturnData: true as AnyObject
 			/* kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock - OLD APP DIDNT HAVE THIS KEY */
 		]

 		status = withUnsafeMutablePointer(to: &dataTypeRef) { SecItemCopyMatching(queryWithCompatibility as CFDictionary, UnsafeMutablePointer($0)) }
 		if status == errSecSuccess {
 			// delete old entry
 			status = SecItemDelete(queryWithCompatibility as CFDictionary)
 			
 			// add newly updated entry... oddly can't use SecItemUpdate() as always getting -50 error... maybe did something wrong.
 			query = [
 				kSecClass: kSecClassKey,
 				kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
 				kSecAttrKeySizeInBits: 512 as AnyObject,
 				kSecValueData: dataTypeRef as! NSData,
 				kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock		/* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
 			]
 			
 			status = SecItemAdd(query as CFDictionary, nil)
 			assert(status == errSecSuccess, "Failed to insert/update the new key in the keychain")
 			
 			return dataTypeRef as! NSData
 		}

 		////////////////////////////////////////////////////
 		////// END BUG FIX COMPATIBILITY CODE MI v5.2 //////
 		////////////////////////////////////////////////////

 		// No pre-existing key from this application, so generate 64 bytes of random data to serve as the encryption key
 		let keyData = NSMutableData(length: 64)!
 		let result = SecRandomCopyBytes(kSecRandomDefault, 64, keyData.mutableBytes.bindMemory(to: UInt8.self, capacity: 64))
 		assert(result == 0, "Failed to get random bytes")
 		
 		// Store the key in the keychain
 		query = [
 			kSecClass: kSecClassKey,
 			kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
 			kSecAttrKeySizeInBits: 512 as AnyObject,
 			kSecValueData: keyData,
 			kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock		/* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
 		]
 		
 		status = SecItemAdd(query as CFDictionary, nil)
 		assert(status == errSecSuccess, "Failed to insert the new key in the keychain")
 		
 		return keyData
 	}
	This is like a drop-in replacement for realms example swift project enabling encryption using the function getKey().

	https://github.com/realm/realm-cocoa/blob/master/examples/ios/swift/Encryption/ViewController.swift

	The following swift function is a drop in replacement, it documents the painful experience of an issue
	which has been unsolved for years till crashlytics revealed a very useful hint, which lead to the root cause
	and fix.

	Comments included for education on what the symptoms were, the cause and fix and other notes.

	private static func getEncryptionKey() -> NSData {
	/*
	PLI @2019-07-01
	identified from crashlytics the "Realm file decryption failed" error...
	https://github.com/realm/realm-cocoa/issues/5615

	https://forums.developer.apple.com/thread/114159 - KeyChain SecItemAdd return -25308 when app is launched
	https://forums.developer.apple.com/thread/78372 - Access Keychain When iPhone Locked

	in short, add kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock

	Disgnosis Explained:
	====================
	Symptoms: intermittent/unexplained crashes (force unwrap try! Realm()) when realm for some reason would
	not properly initialise, when it should initialise perfectly fine.

	Thoery of contributing factors:
	- encrypted realm
	- application supports background updates
	- device is in locked state
	- * application is not running, but is launched by OS to initiate a background update *
	- This fact alone is VERY hard, if not impossible to replicate, as the simulator only allows
	simulate background fetch when the app is being debugged... but what we need is
	app not running, and device locked, and OS launches our app!
	Even if you did replicate it by not running app and leaving overnight or for a long duration
	one would not know when the OS calculated 'launch app to do fetch' will occur, as the algorithm
	of the OS is not published. So one could be waiting for ages...

	Theory of sequence of events to cause crash:
	- when the device is locked, keychain requests if not configured correctly will likely
	return errSecInteractionNotAllowed (-25308), which indicates a UI cannot be presented to prompt to unlock
	and thus gain access. (unless kSecAttrAccessibleAfterFirstUnlock attribute set)
	- as a result the code would fall through (as it didn't check for errSecInteractionNotAllowed error)
	and only checked for errSecSuccess, and run the code assuming the entry did not exist
	and thus create a new key and add to the keychain.
	- the code would have asserted, as it would have got a duplicate entry error, however being production
	code asserts are not compiled in. And since this bug was very intermittent, we never saw it
	in the debug code version where asserts were compiled in.
	- because in production code the asserts was compiled out, the result would be a new key was created
	and returned to the caller, which would of course be a different encryption key as that
	originally used to encrypt the realm DB, and thus realm would correctly thrown an
	exception "Realm file decryption failed".
	- thus resulting in a Realm object not being instantiated, and hence forcing to resolve try! would crash.

	Solution:
	- in order to resolve the problem we need to tag the entry with accessible attribute:
	kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock
	We dont want to limit to only current device, as user may backup and restore, and we
	would still like the DB to work.
	We dont want to have only accessible on unlock, as then it will not work when performing
	background tasks.
	- for compatibility, we need to delete/update the entry with the missing attribute to then include
	the required attribute, to avoid crashes moving forwards and not require the user to delete
	and re-install the app (since keychain persists per app items for different durations when
	app deleted depending on iOS version)
	- a query for item with or without kSecAttrAccessibleAfterFirstUnlock attribute will find item
	originally created with kSecAttrAccessibleAfterFirstUnlock attribute set. BUT a query will only
	find an item created without kSecAttrAccessibleAfterFirstUnlock if the query does not have
	the attribute kSecAttrAccessibleAfterFirstUnlock set.
	UPDATE: OS automatically adds attribute default of kSecAttrAccessibleWhenUnlocked if none
	aupplied by caller on adding to keychain. Thus if you query/fetch with attribute
	not supplied or as kSecAttrAccessibleWhenUnlocked, then a match will result.


	Notes:
	- interestingly in realms keychain_helper.cpp there is a is a function build_search_dictionary()
	which conveniently does pass in the appropriate attributes:

	CFDictionaryAddValue(d.get(), kSecAttrAccessible, kSecAttrAccessibleAlways);

	We do NOT pass kSecAttrAccessibleAlways, as this is deprecated in iOS 13, and is probably considered
	less secure, so we use the next best thing, kSecAttrAccessibleAfterFirstUnlock

	*/

	// Identifier for our keychain entry - should be unique for your application
	let keychainIdentifier = kKeychainIdentifier /* REPLACE THIS WITH YOUR OWN IDENTIFIER */
	let keychainIdentifierData = keychainIdentifier.data(using: String.Encoding.utf8, allowLossyConversion: false)!

	// First check in the keychain for an existing key
	var query: [NSString: AnyObject] = [
	kSecClass: kSecClassKey,
	kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
	kSecAttrKeySizeInBits: 512 as AnyObject,
	kSecReturnData: true as AnyObject,
	kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock /* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
	]

	// To avoid Swift optimization bug, should use withUnsafeMutablePointer() function to retrieve the keychain item
	// See also: http://stackoverflow.com/questions/24145838/querying-ios-keychain-using-swift/27721328#27721328
	var dataTypeRef: AnyObject?
	var status = withUnsafeMutablePointer(to: &dataTypeRef) { SecItemCopyMatching(query as CFDictionary, UnsafeMutablePointer($0)) }
	if status == errSecSuccess {
	return dataTypeRef as! NSData
	}

	////////////////////////////////////////////////
	////// BUG FIX COMPATIBILITY CODE MI v5.2 //////
	////////////////////////////////////////////////
	let queryWithCompatibility: [NSString: AnyObject] = [
	kSecClass: kSecClassKey,
	kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
	kSecAttrKeySizeInBits: 512 as AnyObject,
	kSecReturnData: true as AnyObject
	/* kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock - OLD APP DIDNT HAVE THIS KEY */
	]

	status = withUnsafeMutablePointer(to: &dataTypeRef) { SecItemCopyMatching(queryWithCompatibility as CFDictionary, UnsafeMutablePointer($0)) }
	if status == errSecSuccess {
	// delete old entry
	status = SecItemDelete(queryWithCompatibility as CFDictionary)

	// add newly updated entry... oddly can't use SecItemUpdate() as always getting -50 error... maybe did something wrong.
	query = [
	kSecClass: kSecClassKey,
	kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
	kSecAttrKeySizeInBits: 512 as AnyObject,
	kSecValueData: dataTypeRef as! NSData,
	kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock /* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
	]

	status = SecItemAdd(query as CFDictionary, nil)
	assert(status == errSecSuccess, "Failed to insert/update the new key in the keychain")

	return dataTypeRef as! NSData
	}

	////////////////////////////////////////////////////
	////// END BUG FIX COMPATIBILITY CODE MI v5.2 //////
	////////////////////////////////////////////////////

	// No pre-existing key from this application, so generate 64 bytes of random data to serve as the encryption key
	let keyData = NSMutableData(length: 64)!
	let result = SecRandomCopyBytes(kSecRandomDefault, 64, keyData.mutableBytes.bindMemory(to: UInt8.self, capacity: 64))
	assert(result == 0, "Failed to get random bytes")

	// Store the key in the keychain
	query = [
	kSecClass: kSecClassKey,
	kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
	kSecAttrKeySizeInBits: 512 as AnyObject,
	kSecValueData: keyData,
	kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock /* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
	]

	status = SecItemAdd(query as CFDictionary, nil)
	assert(status == errSecSuccess, "Failed to insert the new key in the keychain")

	return keyData
	}