Compiling OpenCV with contrib modules has been very problematic for me. Here are some solutions I've found to various compilation errors.

Problem:

CMake Error at cmake/OpenCVCompilerOptimizations.cmake:611 (message):
  Compiler doesn't support baseline optimization flags:
Call Stack (most recent call first):
  cmake/OpenCVCompilerOptions.cmake:327 (ocv_compiler_optimization_options)
  CMakeLists.txt:646 (include)

Solution: Only build architecture that your CPU supports. e.g. on M1 mac, only compile arm64. No arm7, no x86_64

python3 opencv/platforms/apple/build_xcframework.py \
--out ./opencv-build \
--contrib opencv_contrib \
--iphoneos_archs arm64 \
--iphonesimulator_archs arm64 \
--build_only_specified_archs

Problem:

~/opencv-build/iphoneos/build/build-arm64-iphoneos/OpenCV.xcodeproj: 
warning: The iOS deployment target 'IPHONEOS_DEPLOYMENT_TARGET' is set to 9.0, but the range of supported deployment target versions is 11.0 to 16.2.99. (in target 'zlib' from project 'OpenCV')

Solution: Change this line in build_framework.py to a supported deployment target e.g. 11.0

IPHONEOS_DEPLOYMENT_TARGET='9.0'

Problem:

PhaseScriptExecution Generate\ CMakeFiles/dephelper/gen_opencv_objc_source_ios /Users/shogun/Downloads/test/opencv-build/iphoneos/build/build-arm64-iphoneos/modules/objc_bindings_generator/OpenCV
.build/Release-iphoneos/gen_opencv_objc_source_ios.build/Script-6ABAC01C8CEE4839A86DE99D.sh (in target 'gen_opencv_objc_source_ios' from project 'OpenCV')

Solution: symlink python opencv/opencv#21926 (comment)

sudo ln -s /opt/homebrew/bin/python3	/opt/homebrew/bin/python

Other solutions: comment out the following in build_xcframework.py opencv/opencv#21571 (comment)

        if macos_archs:
            build_folder = get_or_create_build_folder(args.out, "macos")
            build_folders.append(build_folder)
            command = ["python3", osx_script_path, build_folder, "--macos_archs", macos_archs, "--framework_name", args.framework_name, "--build_only_specified_archs"] + unknown_args
            print_header("Building MacOS frameworks")
            execute(command, cwd=os.getcwd())
        if catalyst_archs:
            build_folder = get_or_create_build_folder(args.out, "catalyst")
            build_folders.append(build_folder)
            command = ["python3", osx_script_path, build_folder, "--catalyst_archs", catalyst_archs, "--framework_name", args.framework_name, "--build_only_specified_archs"] + unknown_args
            print_header("Building Catalyst frameworks")
            execute(command, cwd=os.getcwd())