Add effects to playback

[hburgund]

Would be great if we could add reverb, delay etc to the playback loops, but not sure the best way to do this. I suppose it only makes sense to apply the effects to the entire playback mix, not each loop individually prior to combination; this should be fine.

Would be great to somehow pull out the settings so that they can easily be adjusted. I forget what was done with the effects applied while recording, but both of these should be pulled into the config or somewhere else easily accessible to adjust as we build the music.

[shreyas-jadhav]

I'm thinking if we could do the same optimisation here as well.
Pre rendering the audio with all the effects.
Assuming these levels won't be needed to adjust dynamically.

[hburgund]

I'm thinking if we could do the same optimisation here as well.
Pre rendering the audio with all the effects.
Assuming these levels won't be needed to adjust dynamically.

So we create copies of all speaker audio tracks with the effects pre-rendered and then use those in the playback? Or am I misunderstanding? Do you mean "just-in-time" pre-rendering? This would clearly be a more efficient approach and could be more stable, but as you say, it would only be possible if the effect setting were static. I think we won't need to adjust the effects during playback, but we will certainly need to try different effect settings and combinations to get things right and then may occasionally want to change those. LMK if you are thinking we store rendered copies of the audio or if you are thinking some kind of on-demand pre-rendering into a buffer...?

[shreyas-jadhav]

I think we will be doing this when the mixer is initialized. when the Play button is hit for the first time. that is when we can prerender the buffers. so you can adjust the config and then refresh the app for testing new values.

unrelated concern:
we need to make sure memory constraints of the browser as well. these AudioBufferSourceNode will take more memory than normal streaming. too many or large speakers, and processing/storing them at once might crash on some devices like mobiles not having much ram.

[hburgund]

I think we will be doing this when the mixer is initialized. when the Play button is hit for the first time. that is when we can prerender the buffers. so you can adjust the config and then refresh the app for testing new values.

OK, that sounds great if it is possible to do it this way without and significant delays or anything. I worry if there are hundreds of speakers that we may have resource issues as you mention, doing this, so we might want to consider doing this work in stages based on proximity to the listener? In other words, do all speakers within a certain distance, say 100m to start and then when more speakers enter this buffer zone, we do those?
Invisible Choir will likely have hundreds of speakers eventually. They will be short, but there will be a lot of them.

[hburgund]

unrelated concern:
we need to make sure memory constraints of the browser as well. these AudioBufferSourceNode will take more memory than normal streaming. too many or large speakers, and processing/storing them at once might crash on some devices like mobiles not having much ram.

This is a really great and important point. Especially, as you mention, because this will primarily be running on mobile devices which will have fewer resources. Not sure what we can do to minimize the chances of crashes other than to generally try to be as efficient as possible with everything?

[shreyas-jadhav]

work in stages based on proximity to the listener?

with this approach, i think we will be having issues on walking mode.
for any AudioBufferSourceNode, the start() function requires an user interaction. and we cannot already start() without a ready buffer.
current behaviour is all speakers have a ready buffer and started at first play hit, but at 0 volume.

[hburgund]

current behaviour is all speakers have a ready buffer and started at first play hit, but at 0 volume.

ah, ok...hmm...so if there are 300 speakers in the project, we would be asking the mobile browser to play 300 simultaneous pieces of audio? This seems like we are asking for trouble! What do you think? Shouldn't we have the ones not in range be paused at least to save resources? I understand if they are paused and then they are needed for playback, it would be hard to sync them(?), but I guess we could consider adding new speakers to the mix only at the loop point if that was easier?

Seems like there must be a way to make this more efficient or not require the user interaction for each and every different audio clip. Seems like mobile games work around this somehow with tons of sound effects?

[shreyas-jadhav]

sorry, actually disregard what i said about user interaction. i forgot that we are actually allowed to start() later too after one initial touch,
but the requirement to start() for all speakers at initial play button click was to ensure the synchronisation.
but i think there should be some way to achieve synchronisation too with this procedural loading.

[hburgund]

but the requirement to start() for all speakers at initial play button click was to ensure the synchronisation.
but i think there should be some way to achieve synchronisation too with this procedural loading.

Ah, this is great news @shreyas-jadhav! I'd love to hear your thoughts about how to achieve sync even without starting all at the same time. Seems to me that something like:

• set a physical buffer size for a project preloadSpeakerBufferDistance
• when initial play is pressed, load all speakers that overlap the circle defined by this range and the listener's location
• on each location update, re-evaluate which speakers are within this range, "load" any new ones (I assume this would simply mean to start them playing at 0 volume?) and "unload" any that are outside of the range
• each newly added speaker could start playing at the loop point to achieve proper sync. If we adjust the preloadSpeakerBufferDistance properly and don't have super long loops, in most cases, the listener would enter the new speaker zone well after it was loaded and started playing at the loop point, so this seems like it should work

This might be overly simplistic, but LMK what you think. This would be a great thing to work on now. I will create a different issue for this as it isn't related to effects really.