Extract the index.xml
unzip Filename.iba index.xml
Edit your index.xml file as you see fit (like changing the GUID: http://www.reactuate.com/2013/03/21/ibooks-guid-problem-when-duplicating-ibooks-author-documents/)
Remove the original index.xml entry from the .iba file
zip -d Filename.iba index.xml
Re-add the modified one to the end, with the proper compression (assuming modified index.xml in same dir as Filename.iba)
zip Filename.iba index.xml
It seems that the reason why unzipping and rezipping normally doesn't work is that OS X Archive Utility creates ZIP 1.0 files and iBooks Author expects .iba files to be ZIP 2.0 files. You can also work around this by changing the extension of the .iba file to .zip, unzipping normally, changing the files however you like and then adding them back to the original .zip file with:
cd folder_with_changed_files
zip original_zip_file.zip *