What kind of hardware do you run firefly on?
If its a standard Mac or PC the used fs shouldn’t matter. Also, the performance gain/loss of that are doubtful. Seems you have a different issue going on there.
How do you rip your CDs, and is your CD drive also external?
If you are using a NSLU2 then hands off NTFS!