Blu-ray API bug on Mac OS X
NOTE: The text below applies only to "OS access mode". Starting with version 1.6.4 MakeMKV introduces DASPI disc access mode that is faster and more stable than "OS access" mode. DASPI kernel extension is included with MakeMKV.
Mac OS X version 10.4 "Tiger" and below don't provide any Blu-ray API and MakeMKV has to use so-called "direct access" API that works only with write-capable drives and is significantly slower compared to regular read access. A separate rant on this matter can be found here.
Mac OS X version 10.5 "Leopard" and above support Blu-ray discs and feature API for Blu-ray specific AACS key exchange. So, at least in theory, any program can read data and key material from Blu-ray discs without obtaining exclusive disc access and from any capable drive. But there is a catch.
The API has a bug in it that prevents its usage without workarounds, and for small subset of drives there is no workaround at all. The bug prohibits release of a system resource so in a worst case you'll have to shut down and power on after opening every single Blu-ray disc. Apple has been notified about this bug but apparently ignored it.
What is the bug
Two functions -IOSCSIMultimediaCommandsDevice::REPORT_KEY_V2 and IOSCSIMultimediaCommandsDevice::SEND_KEY_V2 in IOSCSIArchitectureModelFamily / IOSCSIMultimediaCommands / SCSIMultimediaCommands.cpp have the same error that prevents sending INVALIDATE_AGID command to a Blu-ray drive. Per both SCSI MMC and AACS specifications this command shall have no data payload but OS X driver requires non-zero data buffer. The same command for DVD discs is implemented correctly.
What are the workarounds
When MakeMKV detects the bug it tries to use a workaround specified in preferences. In the best case a workaround will fix the problem; in a worst case a workaround will crash drive's firmware and reset would be required. In no case will any workaround permanently damage the drive.
Type 0 - invalidate DVD AGID
MakeMKV will invalidate DVD-specific AGID instead of Blu-ray one. This will work for drives that use the same AGID slots for Blu-ray and DVD. In the worst case workaround won't work.
Type 1 - data underrun
MakeMKV will supply a buffer for REPORT_KEY command despite specification requiring no buffer. Most drives will ignore the buffer and return zero data but some will hang or behave erratically. aacskeys uses this workaround.
Type 2 - data overrun
Similarly to type 1 workaround, MakeMKV will supply a buffer for SEND_KEY command despite specification requiring no buffer. Some drives will ignore the data, some will return an error and some will hang. Use it as a last resort.
Conclusion
If none of the workarounds works for your particular drive, your choice is to use slow direct access mode, change drive, or call Apple and make them fix the bug in their OS. Starting from OS X 10.4.4 Apple doesn't release source code for IOSCSIArchitectureModelFamily driver.
NOTE: Installing DASPI kernel extension will fix this problem once and forever.