You need to put each music collection in a seperate folder per user.
You then need to run four instances of firefly, with each instance pointing to the base folder for that user, and on seperate ports.
So you create
musicFolder1, musicFolder2, musicFolder3 and musicFolder4.
Instance1 of firefly reads from musicFolder1 etc…
the itunes xml files for each user can then be maintained in the respective folders.