PCM Dart Streams

Overview

Streams support on Flutter Sound is a very exciting feature. You can do two things :

Record to a Stream, listen to it and then do what you want with the live audio data. You can process them in dart or send them to a remote server.
Play from Stream allows you to play on your device things that are computed in dart (generator, sequencer, distorder, …) or things that come from a remote server,

Dart Streams support is actually being developped. Some Features are not completely supported:

On iOS things are completely finished. Everything is supposed to work

On Android things are completely finished. Everything is supposed to work

On Web, most everything is to be done. I am going to do that now.

Interleaving

Flutter Sound API supports the two main modes: interleaved, or not interleaved (planar mode). You can look to this guide for a discussion about PCM formats.

Interleaved

The data are sent or received as UInt8list. This are the Raw Data where each sample are coded with 2 or 4 unsigned bytes for each sample. This is convenient when you want to handle globally the data as raw buffers without accessing to the samples. For example when you want to send the raw data to a remote server.

Non interleaved (or Planar)

Non interleaved data are coded as <List<Float32List>> or <List<Int16List>> depending of the codec selected. The number of the element of the List is equal to the number of channels (1 for monophony, 2 for stereophony). This is convenient when you want to access the real audio data as Float32 or Int16.

You can specify toStreamFloat32 or toStreamInt16: even when you have just one channel. In this case the length of the list is 1.

Coding

Flutter Sound supports two codings: Float32 and Int16

Float32: the samples are coded as Floating Point numbers
Int16: the samples are coded as 16 bits Integers.

Record to a Stream

To record data to a live Stream, you use the regular verb startRecorder(), with specific paramaters. Then you listen to your stream. To record to a live PCM Stream, when calling the verb startRecorder(), you specify the parameter toStream:, toStreamFloat32: or toStreamInt16: with your Stream sink, instead of the parameter toFile:. This parameter is a Dart StreamSink that you can listen to, for processing the audio data.

The parameter toStream: is used when you want to record interleaved data to a <Uint8List> Stream Sink
The parameter toStreamFloat32: is used when you want to record non interleaved data (Planar mode) to a <List<Float32List>> Stream Sink as Float32 samples.
The parameter toStreamInt16: is used when you want to record non interleaved data (Planar mode) to a <List<Int16List>> Stream Sink as Int16 samples.

Parameters for `startRecorder()`:

The parameters used for the verb startRecorder() when you want to record to a Stream are :

codec: is mandatory. It can be either :
- Codec.pcm16 if you want to get your PCM samples with an Int16 coding.
- Codec.pcmFloat32 if you want to get your PCM samples with a Float32 coding.
sampleRate: specifies your Sample Rate. It can be anything. The default value is 16000. For example :
- 8000 is a very low Sample rate
- 44100 is the Sample Rate used by CD Audio
- 48000 is a very high quality sample rate
numChannels: specifies the number of channels you want. 1 for monophony, 2 for stereophony, … The default value is 1,
audioSource: specifies the source of your recording. The default value is defaultSource which is probably the mic and is probably what you want.
One (and only one) of the three following parameters to specify the Stream that you will listen to:
- toStream: is the StreamSink of your dart Stream when you want to get your data with the channels samples interleaved. You will receive the data as UInt8List, with the samples coded inside the unsigned bytes. This is convenient if you do not want to process the data in dart, and if you just want to send them to a remote server.
- toStreamInt16: is the StreamSink of your dart Stream when you want to get your data with the channel samples not interleaved not coded. You will receive the data as a List<Int16List>. The List corresponds to the data for each channels. The length of the list is 1 if you record monophony, 2 for stereo, … Each Int16 will be for each sample. Note : this parameter is convenient when you want to access the audio data in dart as integers
- toStreamFloat32: is the StreamSink of your dart Stream when you want to get your data with the channel samples not interleaved not coded. You will receive the data as a List<Float32List>. The List corresponds to the data for each channels. The length of the list is 1 if you record monophony, 2 for stereo, … Each Float32 will be for each sample. Note : this parameter is convenient when you want to access the audio data in dart as Float32 numbers.
bufferSize: is a not very interesting parameter and is for expert only. With this parameter you can specify the size of the internal buffers used by flutter Sound. I sugggest that you do not play with this parameter and keep its default value which is actually 8192. (This default value is probably too high, and I will try to downgrade it in the future),
enableVoiceProcessing: I cannot say anything about this parameter because I don’t know what it is for.

It is really important that you specify the Codec parameter, the sampleRate, the numChannels parameter and your stream. Do not relay on the default values. If you don’t, you will get bad values. You could even get Exceptions or crash.

Listen to your Stream

There is nothing special to listen to your stream. You will get the data in the Stream you specified in startRecorder(), coded either as an interleaved stream in toStream:, or as a list of not interleaved stream in toStreamFloat32: or toStreamInt16:.

Examples

You can look to

This simple example provided with Flutter Sound.
Or this other example

Example with interleaved data coded in UInt8List:

const int kSAMPLERATE = 16000;
const int kNUMBEROFCHANNELS = 2;

final FlutterSoundRecorder _mRecorder = FlutterSoundRecorder();
var recordingDataControllerUint8 = StreamController<Uint8List>();

      await _mRecorder.startRecorder(
        codec: Codec.pcmFloat32,
        sampleRate: kSAMPLERATE,
        numChannels: kNUMBEROFCHANNELS,
        audioSource: AudioSource.defaultSource,
        toStream: recordingDataControllerUint8.sink,
      );

      recordingDataControllerUint8.stream.listen((UInt8List data) {
          // Process my frame in data
          sendToMyServer(data);
      });

Example with data not interleaved (planar mode) given as Float32:

const int kSAMPLERATE = 16000;
const int kNUMBEROFCHANNELS = 2;

final FlutterSoundRecorder _mRecorder = FlutterSoundRecorder();
var recordingDataControllerF32 = StreamController<List<Float32List>>();

      await _mRecorder.startRecorder(
        codec: Codec.pcmFloat32,
        sampleRate: kSAMPLERATE,
        numChannels: kNUMBEROFCHANNELS,
        audioSource: AudioSource.defaultSource,
        toStream: recordingDataControllerF32.sink,
      );

      recordingDataControllerF32.stream.listen((data) {
        for (Float32List frame in data) {
          // Process my frame
          for (double sample in frame) {
            // process my sample
            // ...
          }
        }
      });

Play from Stream

To play live stream, you start playing with the verb startPlayerFromStream() instead of the regular startPlayer() verb.

The main parameters for the verb startPlayerFromStream() are :

codec: : The codec (Codec.pcm16 or Codec.pcmFloat32)
sampleRate: : The sample rate
numChannels: : The number of channels (1 for monophony, 2 for stereophony, or more …)
interleaved: : A boolean for specifying if the data played are interleaved. This parameter specifies if the data to be played are interleaved or not. When the data are interleaved, you will use the _mPlayer.uint8ListSink to play data. When the data are not interleaved, you will use _mPlayer.float32Sink or _mPlayer.int16Sink depending on the codec used. When the data are interleaved, the data provided by the app must be coded as UInt8List. This is convenient when you have raw data to be played from a remote server. When the data are not interleaved, they are provided as List<Int16List> or List<Float32List>, with an array of length equal to the number of channels.
_mPlayer.float32Sink is a Stream Sink used when the data are interleaved and when you have UInt8List buffers to be played
_mPlayer.int16Sink is a Stream Sink used when the data are not interleaved and when you have Float32 data to be played
_mPlayer.uint8ListSink is a Stream Sink used when the data are interleaved and when you have UInt8 data to be played

Example:

await myPlayer.startPlayerFromStream
(
    codec: Codec.pcmFloat32 
    numChannels: 2
    sampleRate: 48100
    interleaved: true,
);


await myPlayer.feedF32FromStream(aBuffer);

Parameters for `startPlayerFromStream()`:

codec: is mandatory. It can be either :
- Codec.pcm16 if you want to get your PCM samples with an Int16 coding
- Codec.pcmFloat32 if you want to get your PCM samples with a Float32 coding.
sampleRate: specifies your Sample Rate. It can be anything. The default value is 16000. For example :
- 8000 is a very low Sample rate
- 44100 is the Sample Rate used by CD Audio
- 48000 is a very high quality sample rate
numChannels: specifies the number of channels you want. 1 for monophony, 2 for stereophony, … The default value is 1.
interleaved: is a boolean which specifies if you will provide the data to be played as an interleaved stream of Int16, or if you will provide these data as Lists of Float32List (or Lists of Int16List).
bufferSize: is a not very interesting parameter and is for expert only. With this parameter you can specify the size of the internal buffers used by flutter Sound. I sugggest that you do not play with this parameter and keep its default value which is actually 8192. This default value is probably too high, and I will try to downgrade it in the future,

It is really important that you specify the Codec parameter, the sampleRate, the numChannels parameter and the interleaved boolean. Do not relay on the default values. If you don’t, you will get bad values. You could even get Exceptions or crash.

whenFinished:

This parameter cannot be used. After startPlayerFromStream() the player is always available until stopPlayer(). The app can provide audio data when it wants. Even after an elapsed time without any audio data.

Play your live data

After having starting your player, you can begin to play your live data to the output device.

You have two possibilities:

Play them without any flow control
Play them with flow control

Play data without flow control

The App does myPlayer.uint8ListSink.add(d) or _mPlayer.float32Sink(d) or mPlayer.int16Sink(d) each time it wants to play some data. No need to await, no need to verify if the previous buffers have finished to be played. All the data added to the Stream Sink are buffered, and are played sequentially. The App continues to work without knowing when the buffers are really played.

This means three things :

If the App is very fast adding buffers to the foodSink it can consume a lot of memory for the waiting buffers.
When the App has finished feeding the sink, it cannot just do myPlayer.stopPlayer(), because there are perhaps many buffers not yet played. If it does a stopPlayer(), all the waiting buffers will be flushed which is probably not what it wants.
The App cannot know when the audio data are really played.

Example:

UInt8List myBuffer;

await myPlayer.startPlayerFromStream(codec: Codec.pcm16, numChannels: 1, sampleRate: 48000, interleaved: true);
...
await myPlayer.uint8ListSink.add(myBuffer);
...
await myPlayer.stopPlayer();

Play data with flow control

Playing live data without flow control is very simple, because you don’t have to wait//handle Futures. But sometimes it can be interesting to manage a flow control :

When you have huge data generated and you cannot loop feeding your Stream Sink.
When you want to know when the data has been played for generating data on demand.
When you just want to know when your previous packet has been played

If the App wants to keep synchronization with what is played, it uses the verb feedUint8FromStream(), feedInt16FromStream() or feedF32FromStream() to play data.

It is really very important not to call another feedFromStream() before the completion of the previous future. When each Future is completed, the App can be sure that the provided data are correctely either played, or at least put in low level internal buffers, and it knows that it is safe to do another one.

Example:

UInt8List myBuffer;

await myPlayer.startPlayerFromStream(codec: Codec.pcm16, numChannels: 1, sampleRate: 48000, interleaved: true);
...
await myPlayer.feedUint8FromStream(myBuffer);
...
await myPlayer.stopPlayer();

You will await or use then() for each call to feedFromStream().

It is really important that you feed the correct stream, depending on the interleaved parameter and the Codec parameter. If you data are interleaved, you must feed your stream with myPlayer.uint8ListSink.add(d) or feedUint8FromStream(). If your data are not interleaved, use _mPlayer.float32Sink(d)/mPlayer.int16Sink(d); or feedInt16FromStream()/feedF32FromStream().

If your data are not interleaved, it is really important that all the lists with which you feed your stream have exactly the length of your number of channels. It is also very important that the data for each channels have exactly the same length.

Examples

You can look to the provided examples :

This example shows how to play live data, with Back Pressure.
This example shows how to play live data, without Back Pressure.
This example shows how to play Float32, or Int16, Interleaved or Planar.

PCM Dart Streams

Overview

Interleaving

Interleaved

Non interleaved (or Planar)

Coding

Record to a Stream

Parameters for startRecorder():

Listen to your Stream

Examples

Example with interleaved data coded in UInt8List:

Example with data not interleaved (planar mode) given as Float32:

Play from Stream

Parameters for startPlayerFromStream():

whenFinished:

Play your live data

Play data without flow control

Play data with flow control

Examples

Parameters for `startRecorder()`:

Parameters for `startPlayerFromStream()`: