After Android Marshmallow update in Samsung Note 4, the option to detach the OTG USB pen drive is deep buried into the storage settings of the phone. Follow the below steps to safely detach your pen drive from Note 4
On successful connection of the OTG pen drive a notification would be shown as below. Before Marshmallow, selecting the notification would provide an option to detach the drive now it opens the File Explorer and lists the files and folders in the attached USB drive
To dismount the drive, navigate to the settings and select “storage” in the available menu items
Select the up arrow symbol of the USB drive and wait for couple of seconds. This would un-mount the drive and now you can safely remove the USB drive from the mobile.
Note : The USB pen drive that I am using is 64 GB and it needs to be formatted in exFAT file format to work seamlessly in PC and Smartphone