Speaker loading optimization on looping playback

[hburgund]

In order to use limited mobile browser resources efficiently - especially for projects with large numbers of speakers - we need to load speakers based on proximity rather than load and start playing all of them immediately (albeit out of range ones at 0 volume).

Below migrated from another unrelated ticket #81

we need to make sure memory constraints of the browser as well. these AudioBufferSourceNode will take more memory than normal streaming. too
many or large speakers, and processing/storing them at once might crash on some devices like mobiles not having much ram.

hburgund
I worry if there are hundreds of speakers that we may have resource issues as you mention, doing this, so we might want to consider doing this work in stages based on proximity to the listener? In other words, do all speakers within a certain distance, say 100m to start and then when more speakers enter this buffer zone, we do those?
Invisible Choir will likely have hundreds of speakers eventually. They will be short, but there will be a lot of them.

shreyas-jadhav
work in stages based on proximity to the listener?

with this approach, i think we will be having issues on walking mode.
for any AudioBufferSourceNode, the start() function requires an user interaction. and we cannot already start() without a ready buffer.
current behaviour is all speakers have a ready buffer and started at first play hit, but at 0 volume.

hburgund
ah, ok...hmm...so if there are 300 speakers in the project, we would be asking the mobile browser to play 300 simultaneous pieces of audio? This seems like we are asking for trouble! What do you think? Shouldn't we have the ones not in range be paused at least to save resources? I understand if they are paused and then they are needed for playback, it would be hard to sync them(?), but I guess we could consider adding new speakers to the mix only at the loop point if that was easier?

Seems like there must be a way to make this more efficient or not require the user interaction for each and every different audio clip. Seems like mobile games work around this somehow with tons of sound effects?

shreyas-jadhav
sorry, actually disregard what i said about user interaction. i forgot that we are actually allowed to start() later too after one initial touch,
but the requirement to start() for all speakers at initial play button click was to ensure the synchronisation.
but i think there should be some way to achieve synchronisation too with this procedural loading.

hburgund
Ah, this is great news @shreyas-jadhav! I'd love to hear your thoughts about how to achieve sync even without starting all at the same time. Seems to me that something like:

• set a physical buffer size for a project preloadSpeakerBufferDistance
• when initial play is pressed, load all speakers that overlap the circle defined by this range and the listener's location i.e. "in range"
• on each location update, re-evaluate which speakers are within this range, "load" any new ones (I assume this would simply mean to start them playing at 0 volume?) and "unload" any that are outside of the range
• each newly added speaker could start playing at the loop point to achieve proper sync. If we adjust the preloadSpeakerBufferDistance properly and don't have super long loops, in most cases, the listener would enter the new speaker zone well after it was loaded and started playing at the loop point, so this seems like it should work

This might be overly simplistic, but LMK what you think. This would be a great thing to work on now.

[shreyas-jadhav]

Should we also defer the "fetching" of speakers?
currently, the speakers are fetched all at once on the app load, and it would already be consuming memory.
but as you say there would be more of speakers this memory for storing fetched speakers can be too much as well.

I'm not sure if this will always break. But we can also defer the fetching part, for example have a greater range where fetching start, and another lower range where the processing and loading starts.

[hburgund]

Yes, this makes a lot of sense. I actually think I didn't fully consider how the system currently works with a distinction between "fetching" and "loading", but doing both of these progressively makes sense and with different buffer ranges, defined in the config. Thanks for bringing this up...excellent idea.

[hburgund]

I've been doing a lot of work on the speaker graphic representation and was wondering if it might help debugging and understanding what is going on with different speaker states - unfetched, fetched, loaded ? - if we changed their display based on this state? For example, only stroke for unfetched, add low opacity fill for fetched and higher opacity fill for loaded?

Don't want to make things more complicated, but if you think this might be useful and not too hard to implement, could be very helpful for debugging and understanding what is going on as things get more complex.

Relatedly, I'm going to make a new issue for changing speaker visual properties when a speaker is actively playing, hence my thinking about this as well...

[hburgund]

To be clear prefetchDistanceMeters is the distance between the listener and the edge (or center?) of a speaker, below which that speaker's audio is prefetched? It is set to 5 now and seems like it is bigger in my testing, so making sure.

Also, is there a second distance we use to determine when prefetched speaker audio should start playing at 0 volume so it is ready to be heard? I don't see an obvious config value for that, but might just be missing it.

[shreyas-jadhav]

It is the distance between the listener and the edge of a speaker shape.

seems like it is bigger in my testing

will test. turf might have been returning some different unit i guess.

determine when prefetched speaker audio should start playing at 0

maybe this is not necessary. we could start playing when it should actually be audible, and not need to play at 0 volume.
or is there any actual use case for this that i missed?

[hburgund]

maybe this is not necessary. we could start playing when it should actually be audible, and not need to play at 0 volume.
or is there any actual use case for this that i missed?

This is much better if we don't have to play any audio at zero volume. I somehow thought that this was required in order to maintain sync, but if not, I much prefer to only play audio that can actually be heard.

The only use case I wonder about is if a user is listening while walking from one speaker area into another in the middle of a loop. Does this new speaker audio fade in smoothly (assuming there is an available slot for a new track to play) as the user walks through the attenuation border of the speaker shape and remain synced? Or does the new speaker's audio not start playing until the loop point arrives and it is then added to an open slot?

If you are able to fade in speakers in a sync'd way, that would be great and would be more efficient by avoiding playing audio at 0 volume, but I just don't know if this is possible. LMK what you think and what is currently implemented so I can test more with this understanding.