关于openharmony:OpenHarmony-32-Beta-Audio音频渲染

5次阅读

共计 11057 个字符，预计需要花费 28 分钟才能阅读完成。

一、简介

Audio 是多媒体子系统中的一个重要模块，其波及的内容比拟多，有音频的渲染、音频的采集、音频的策略管理等。本文次要针对音频渲染性能进行具体地剖析，并通过源码中提供的例子，对音频渲染进行流程的梳理。

二、目录 foundation/multimedia/audio_framework

audio_framework
├── frameworks
│   ├── js                          #js 接口
│   │   └── napi
│   │       └── audio_renderer      #audio_renderer NAPI 接口
│   │           ├── include
│   │           │   ├── audio_renderer_callback_napi.h
│   │           │   ├── renderer_data_request_callback_napi.h
│   │           │   ├── renderer_period_position_callback_napi.h
│   │           │   └── renderer_position_callback_napi.h
│   │           └── src
│   │               ├── audio_renderer_callback_napi.cpp
│   │               ├── audio_renderer_napi.cpp
│   │               ├── renderer_data_request_callback_napi.cpp
│   │               ├── renderer_period_position_callback_napi.cpp
│   │               └── renderer_position_callback_napi.cpp
│   └── native                      #native 接口
│       └── audiorenderer
│           ├── BUILD.gn
│           ├── include
│           │   ├── audio_renderer_private.h
│           │   └── audio_renderer_proxy_obj.h
│           ├── src
│           │   ├── audio_renderer.cpp
│           │   └── audio_renderer_proxy_obj.cpp
│           └── test
│               └── example
│                   └── audio_renderer_test.cpp
├── interfaces
│   ├── inner_api                   #native 实现的接口
│   │   └── native
│   │       └── audiorenderer       #audio 渲染本地实现的接口定义
│   │           └── include
│   │               └── audio_renderer.h
│   └── kits                        #js 调用的接口
│       └── js
│           └── audio_renderer      #audio 渲染 NAPI 接口的定义
│               └── include
│                   └── audio_renderer_napi.h
└── services                        #服务端
    └── audio_service
        ├── BUILD.gn
        ├── client                  #IPC 调用中的 proxy 端
        │   ├── include
        │   │   ├── audio_manager_proxy.h
        │   │   ├── audio_service_client.h
        │   └── src
        │       ├── audio_manager_proxy.cpp
        │       ├── audio_service_client.cpp
        └── server                  #IPC 调用中的 server 端
            ├── include
            │   └── audio_server.h
            └── src
                ├── audio_manager_stub.cpp
                └── audio_server.cpp

三、音频渲染总体流程

四、Native 接口应用

在 OpenAtom OpenHarmony（以下简称“OpenHarmony”）零碎中，音频模块提供了性能测试代码，本文选取了其中的音频渲染例子作为切入点来进行介绍，例子采纳的是对 wav 格局的音频文件进行渲染。wav 格局的音频文件是 wav 头文件和音频的原始数据，不须要进行数据解码，所以音频渲染间接对原始数据进行操作，文件门路为：foundation/multimedia/audio_framework/frameworks/native/audiorenderer/test/example/audio_renderer_test.cpp

bool TestPlayback(int argc, char *argv[]) const
{FILE* wavFile = fopen(path, "rb");
        // 读取 wav 文件头信息
        size_t bytesRead = fread(&wavHeader, 1, headerSize, wavFile);

        // 设置 AudioRenderer 参数
        AudioRendererOptions rendererOptions = {};
        rendererOptions.streamInfo.encoding = AudioEncodingType::ENCODING_PCM;
        rendererOptions.streamInfo.samplingRate = static_cast<AudioSamplingRate>(wavHeader.SamplesPerSec);
        rendererOptions.streamInfo.format = GetSampleFormat(wavHeader.bitsPerSample);
        rendererOptions.streamInfo.channels = static_cast<AudioChannel>(wavHeader.NumOfChan);
        rendererOptions.rendererInfo.contentType = contentType;
        rendererOptions.rendererInfo.streamUsage = streamUsage;
        rendererOptions.rendererInfo.rendererFlags = 0;

        // 创立 AudioRender 实例
        unique_ptr<AudioRenderer> audioRenderer = AudioRenderer::Create(rendererOptions);

        shared_ptr<AudioRendererCallback> cb1 = make_shared<AudioRendererCallbackTestImpl>();
        // 设置音频渲染回调
        ret = audioRenderer->SetRendererCallback(cb1);

        //InitRender 办法次要调用了 audioRenderer 实例的 Start 办法，启动音频渲染
        if (!InitRender(audioRenderer)) {AUDIO_ERR_LOG("AudioRendererTest: Init render failed");
            fclose(wavFile);
            return false;
        }

        //StartRender 办法次要是读取 wavFile 文件的数据，而后通过调用 audioRenderer 实例的 Write 办法进行播放
        if (!StartRender(audioRenderer, wavFile)) {AUDIO_ERR_LOG("AudioRendererTest: Start render failed");
            fclose(wavFile);
            return false;
        }

        // 进行渲染
        if (!audioRenderer->Stop()) {AUDIO_ERR_LOG("AudioRendererTest: Stop failed");
        }

        // 开释渲染
        if (!audioRenderer->Release()) {AUDIO_ERR_LOG("AudioRendererTest: Release failed");
        }

        // 敞开 wavFile
        fclose(wavFile);
        return true;
    }

首先读取 wav 文件，通过读取到 wav 文件的头信息对 AudioRendererOptions 相干的参数进行设置，包含编码格局、采样率、采样格局、通道数等。依据 AudioRendererOptions 设置的参数来创立 AudioRenderer 实例（实际上是 AudioRendererPrivate），后续的音频渲染次要是通过 AudioRenderer 实例进行。创立实现后，调用 AudioRenderer 的 Start 办法，启动音频渲染。启动后，通过 AudioRenderer 实例的 Write 办法，将数据写入，音频数据会被播放。

五、调用流程

  1. 创立 AudioRenderer
std::unique_ptr<AudioRenderer> AudioRenderer::Create(const std::string cachePath,
    const AudioRendererOptions &rendererOptions, const AppInfo &appInfo)
{
    ContentType contentType = rendererOptions.rendererInfo.contentType;
    
    StreamUsage streamUsage = rendererOptions.rendererInfo.streamUsage;
   
    AudioStreamType audioStreamType = AudioStream::GetStreamType(contentType, streamUsage);
    auto audioRenderer = std::make_unique<AudioRendererPrivate>(audioStreamType, appInfo);
    if (!cachePath.empty()) {AUDIO_DEBUG_LOG("Set application cache path");
        audioRenderer->SetApplicationCachePath(cachePath);
    }

    audioRenderer->rendererInfo_.contentType = contentType;
    audioRenderer->rendererInfo_.streamUsage = streamUsage;
    audioRenderer->rendererInfo_.rendererFlags = rendererOptions.rendererInfo.rendererFlags;

    AudioRendererParams params;
    params.sampleFormat = rendererOptions.streamInfo.format;
    params.sampleRate = rendererOptions.streamInfo.samplingRate;
    params.channelCount = rendererOptions.streamInfo.channels;
    params.encodingType = rendererOptions.streamInfo.encoding;

    if (audioRenderer->SetParams(params) != SUCCESS) {AUDIO_ERR_LOG("SetParams failed in renderer");
        audioRenderer = nullptr;
        return nullptr;
    }

    return audioRenderer;
}

首先通过 AudioStream 的 GetStreamType 办法获取音频流的类型，依据音频流类型创立 AudioRendererPrivate 对象，AudioRendererPrivate 是 AudioRenderer 的子类。紧接着对 audioRenderer 进行参数设置，其中包含采样格局、采样率、通道数、编码格局。设置实现后返回创立的 AudioRendererPrivate 实例。

  1. 设置回调 
int32_t AudioRendererPrivate::SetRendererCallback(const std::shared_ptr<AudioRendererCallback> &callback)
{RendererState state = GetStatus();
    if (state == RENDERER_NEW || state == RENDERER_RELEASED) {return ERR_ILLEGAL_STATE;}
    if (callback == nullptr) {return ERR_INVALID_PARAM;}

    // Save reference for interrupt callback
    if (audioInterruptCallback_ == nullptr) {return ERROR;}
    std::shared_ptr<AudioInterruptCallbackImpl> cbInterrupt =
        std::static_pointer_cast<AudioInterruptCallbackImpl>(audioInterruptCallback_);
    cbInterrupt->SaveCallback(callback);

    // Save and Set reference for stream callback. Order is important here.
    if (audioStreamCallback_ == nullptr) {audioStreamCallback_ = std::make_shared<AudioStreamCallbackRenderer>();
        if (audioStreamCallback_ == nullptr) {return ERROR;}
    }
    std::shared_ptr<AudioStreamCallbackRenderer> cbStream =
std::static_pointer_cast<AudioStreamCallbackRenderer>(audioStreamCallback_);
    cbStream->SaveCallback(callback);
    (void)audioStream_->SetStreamCallback(audioStreamCallback_);

    return SUCCESS;
}

参数传入的回调次要波及到两个方面：一方面是 AudioInterruptCallbackImpl 中设置了咱们传入的渲染回调，另一方面是 AudioStreamCallbackRenderer 中也设置了渲染回调。

  1. 启动渲染 
bool AudioRendererPrivate::Start(StateChangeCmdType cmdType) const
{AUDIO_INFO_LOG("AudioRenderer::Start");
    RendererState state = GetStatus();

    AudioInterrupt audioInterrupt;
    switch (mode_) {
        case InterruptMode::SHARE_MODE:
            audioInterrupt = sharedInterrupt_;
            break;
        case InterruptMode::INDEPENDENT_MODE:
            audioInterrupt = audioInterrupt_;
            break;
        default:
            break;
    }
    AUDIO_INFO_LOG("AudioRenderer::Start::interruptMode: %{public}d, streamType: %{public}d, sessionID: %{public}d",
        mode_, audioInterrupt.streamType, audioInterrupt.sessionID);

    if (audioInterrupt.streamType == STREAM_DEFAULT || audioInterrupt.sessionID == INVALID_SESSION_ID) {return false;}

    int32_t ret = AudioPolicyManager::GetInstance().ActivateAudioInterrupt(audioInterrupt);
    if (ret != 0) {AUDIO_ERR_LOG("AudioRendererPrivate::ActivateAudioInterrupt Failed");
        return false;
    }

    return audioStream_->StartAudioStream(cmdType);
}

AudioPolicyManager::GetInstance().ActivateAudioInterrupt 这个操作次要是依据 AudioInterrupt 来进行音频中断的激活，这里波及了音频策略相干的内容，后续会专门出对于音频策略的文章进行剖析。这个办法的外围是通过调用 AudioStream 的 StartAudioStream 办法来启动音频流。

bool AudioStream::StartAudioStream(StateChangeCmdType cmdType)
{int32_t ret = StartStream(cmdType);

    resetTime_ = true;
    int32_t retCode = clock_gettime(CLOCK_MONOTONIC, &baseTimestamp_);

    if (renderMode_ == RENDER_MODE_CALLBACK) {
        isReadyToWrite_ = true;
        writeThread_ = std::make_unique<std::thread>(&AudioStream::WriteCbTheadLoop, this);
    } else if (captureMode_ == CAPTURE_MODE_CALLBACK) {
        isReadyToRead_ = true;
        readThread_ = std::make_unique<std::thread>(&AudioStream::ReadCbThreadLoop, this);
    }

    isFirstRead_ = true;
    isFirstWrite_ = true;
    state_ = RUNNING;
    AUDIO_INFO_LOG("StartAudioStream SUCCESS");

    if (audioStreamTracker_) {AUDIO_DEBUG_LOG("AudioStream:Calling Update tracker for Running");
        audioStreamTracker_->UpdateTracker(sessionId_, state_, rendererInfo_, capturerInfo_);
    }
    return true;
}

AudioStream 的 StartAudioStream 次要的工作是调用 StartStream 办法，StartStream 办法是 AudioServiceClient 类中的办法。AudioServiceClient 类是 AudioStream 的父类。接下来看一下 AudioServiceClient 的 StartStream 办法。

int32_t AudioServiceClient::StartStream(StateChangeCmdType cmdType)
{
    int error;
    lock_guard<mutex> lockdata(dataMutex);
    pa_operation *operation = nullptr;

    pa_threaded_mainloop_lock(mainLoop);

    pa_stream_state_t state = pa_stream_get_state(paStream);

    streamCmdStatus = 0;
    stateChangeCmdType_ = cmdType;
    operation = pa_stream_cork(paStream, 0, PAStreamStartSuccessCb, (void *)this);

    while (pa_operation_get_state(operation) == PA_OPERATION_RUNNING) {pa_threaded_mainloop_wait(mainLoop);
    }
    pa_operation_unref(operation);
    pa_threaded_mainloop_unlock(mainLoop);

    if (!streamCmdStatus) {AUDIO_ERR_LOG("Stream Start Failed");
        ResetPAAudioClient();
        return AUDIO_CLIENT_START_STREAM_ERR;
    } else {AUDIO_INFO_LOG("Stream Started Successfully");
        return AUDIO_CLIENT_SUCCESS;
    }
}

StartStream 办法中次要是调用了 pulseaudio 库的 pa_stream_cork 办法进行流启动，后续就调用到了 pulseaudio 库中了。pulseaudio 库咱们暂且不剖析。

  1. 写入数据 
int32_t AudioRendererPrivate::Write(uint8_t *buffer, size_t bufferSize)
{return audioStream_->Write(buffer, bufferSize);
}

通过调用 AudioStream 的 Write 形式实现性能，接下来看一下 AudioStream 的 Write 办法。

size_t AudioStream::Write(uint8_t *buffer, size_t buffer_size)
{
    int32_t writeError;
    StreamBuffer stream;
    stream.buffer = buffer;
    stream.bufferLen = buffer_size;
    isWriteInProgress_ = true;

    if (isFirstWrite_) {if (RenderPrebuf(stream.bufferLen)) {return ERR_WRITE_FAILED;}
        isFirstWrite_ = false;
    }

    size_t bytesWritten = WriteStream(stream, writeError);
    isWriteInProgress_ = false;
    if (writeError != 0) {AUDIO_ERR_LOG("WriteStream fail,writeError:%{public}d", writeError);
        return ERR_WRITE_FAILED;
    }
    return bytesWritten;
}

Write 办法中分成两个阶段，首次写数据，先调用 RenderPrebuf 办法，将 preBuf_的数据写入后再调用 WriteStream 进行音频数据的写入。

size_t AudioServiceClient::WriteStream(const StreamBuffer &stream, int32_t &pError)
{size_t cachedLen = WriteToAudioCache(stream);
    if (!acache.isFull) {
        pError = error;
        return cachedLen;
    }

    pa_threaded_mainloop_lock(mainLoop);


    const uint8_t *buffer = acache.buffer.get();
    size_t length = acache.totalCacheSize;

    error = PaWriteStream(buffer, length);
    acache.readIndex += acache.totalCacheSize;
    acache.isFull = false;

    if (!error && (length >= 0) && !acache.isFull) {uint8_t *cacheBuffer = acache.buffer.get();
        uint32_t offset = acache.readIndex;
        uint32_t size = (acache.writeIndex - acache.readIndex);
        if (size > 0) {if (memcpy_s(cacheBuffer, acache.totalCacheSize, cacheBuffer + offset, size)) {AUDIO_ERR_LOG("Update cache failed");
                pa_threaded_mainloop_unlock(mainLoop);
                pError = AUDIO_CLIENT_WRITE_STREAM_ERR;
                return cachedLen;
            }
            AUDIO_INFO_LOG("rearranging the audio cache");
        }
        acache.readIndex = 0;
        acache.writeIndex = 0;

        if (cachedLen < stream.bufferLen) {
            StreamBuffer str;
            str.buffer = stream.buffer + cachedLen;
            str.bufferLen = stream.bufferLen - cachedLen;
            AUDIO_DEBUG_LOG("writing pending data to audio cache: %{public}d", str.bufferLen);
            cachedLen += WriteToAudioCache(str);
        }
    }

    pa_threaded_mainloop_unlock(mainLoop);
    pError = error;
    return cachedLen;
}

WriteStream 办法不是间接调用 pulseaudio 库的写入办法，而是通过 WriteToAudioCache 办法将数据写入缓存中，如果缓存没有写满则间接返回，不会进入上面的流程，只有当缓存写满后，才会调用上面的 PaWriteStream 办法。该办法波及对 pulseaudio 库写入操作的调用，所以缓存的目标是防止对 pulseaudio 库频繁地做 IO 操作，进步了效率。

六、总结

本文次要对 OpenHarmony 3.2 Beta 多媒体子系统的音频渲染模块进行介绍，首先梳理了 Audio Render 的整体流程，而后对几个外围的办法进行代码的剖析。整体的流程次要通过 pulseaudio 库启动流，而后通过 pulseaudio 库的 pa_stream_write 办法进行数据的写入，最初播放出音频数据。

音频渲染次要分为以下几个档次：
（1）AudioRenderer 的创立，理论创立的是它的子类 AudioRendererPrivate 实例。
（2）通过 AudioRendererPrivate 设置渲染的回调。
（3）启动渲染，这一部分代码最终会调用到 pulseaudio 库中，相当于启动了 pulseaudio 的流。
（4）通过 pulseaudio 库的 pa_stream_write 办法将数据写入设施，进行播放。

对 OpenHarmony 3.2 Beta 多媒体系列开发感兴趣的读者，也能够浏览我之前写过几篇文章：
《OpenHarmony 3.2 Beta 多媒体系列——视频录制》
《OpenHarmony 3.2 Beta 源码剖析之 MediaLibrary》
《OpenHarmony 3.2 Beta 多媒体系列——音视频播放框架》
《OpenHarmony 3.2 Beta 多媒体系列——音视频播放 gstreamer》。

正文完
openharmony
发表至： openharmony
2023-03-02
 0
关于openharmony:课程汇总OpenHarmony-成长计划知识赋能第二期系列课程附链接
关于openharmony:OpenHarmony-Meetup常州站招募令
关于openharmony:OpenHarmony标准设备应用开发三分布式数据管理
关于openharmony:今晚战码先锋润和赛道第2期直播丨如何参与OpenHarmony代码贡献

站内搜索

-「」