背景
WebRTC我的项目中增加背景虚化性能，应用tensorflow.js和google现成的模型bodyPix实现，但理论应用中存在两个问题，一是帧率低(临时未解决)，二是切换到其余tab页后，背景虚化会很卡，简直停滞，查阅材料后发现Chrome浏览器会升高暗藏tab页的性能，解决方案是应用WebWorker。
应用一周工夫，一直地踩坑最初在webWorker中实现了优化。
踩坑
1.最后认为卡顿的起因是requestAnimationFrame办法，理论改成setTimeout或setInterval或间接持续虚化后发现仍旧卡顿，最初发现理论卡顿的中央是getSegmentPerson()的调用，在tab页切换后该办法调用须要几秒钟工夫。
2.造成卡顿的是getSegmentPerson()办法，就只能将该办法放到worker中进行，然而getSegmentPerson须要传入原始的video或者canvas,而dom不能在worker中应用。持续翻阅源码发现bodyPix还反对传入OffscreenCanvas和ImageData,OffscreenCanvas是专门在webWorker上应用的canvas，被叫做离屏canvas,ImageData是接口形容 <canvas> 元素的一个隐含像素数据的区域,能够间接在canvas.getContext('2d').getImageData()取得。最终我实现了两种，抉择了ImageData这种形式。
这是bodyPix getSegmentPerson的源码
segmentPerson(input: BodyPixInput, config?: PersonInferenceConfig): Promise<SemanticPersonSegmentation>;export declare type ImageType = HTMLImageElement | HTMLCanvasElement | HTMLVideoElement | OffscreenCanvas;export declare type BodyPixInput = ImageData | ImageType | tf.Tensor3D;3.无论是OffscreenCanvas还是ImageData都须要创立一个新的canvas来实时画video帧，新的canvas必须设置width,height与video的保持一致,否则获取的segmentation不准,我没设置时就始终看到的是全副虚化了包含我本人,钻研了很久发现是width和height的问题。
WebWorker
1.创立my.worker.ts
2.bodyPix.load()从主体代码迁徙到worker中，主体代码间接接管segmentation值
3.监听主体代码传过来的ImageData,调用net.segmentPerson()办法，获取的值再传回去
import * as tfjs from '@tensorflow/tfjs';import * as bodyPix from '@tensorflow-models/body-pix';import BodyPix from './service/BodyPix';const webWorker: Worker = self as any; let body = null;let offscreen = null;let context = null;webWorker.addEventListener('message', async (event) => { const { action, data } = event.data; switch(action) { case 'init': body = new BodyPix(); await body.loadAndPredict(); webWorker.postMessage({inited: true}); break; case 'imageData': body.net.segmentPerson(data.imageData, BodyPix.option.config).then((segmentation) => { requestAnimationFrame(() => { webWorker.postMessage({segmentation}); }) }) break; }});export default null as any;主体代码
截取局部代码
async blurBackground (canvas: HTMLCanvasElement, video: HTMLVideoElement) { //渲染的canvas和获取segmentation的canvas的widht,height必须和video保持一致，否则bodyPix返回的segmentation会不准 const [ width, height ] = [ video.videoWidth, video.videoHeight ]; video.width = width; canvas.width = width; video.height = height; canvas.height = height; this.workerCanvas = document.createElement('canvas'); this.workerCanvas.width = video.width; this.workerCanvas.height = video.height; this.bluring = true; this.blurInWorker(video, canvas);}async drawImageData (newCanvas: HTMLCanvasElement, video: HTMLVideoElement) { const ctx = newCanvas.getContext('2d'); ctx.drawImage(video, 0, 0, newCanvas.width, newCanvas.height); const imageData = ctx.getImageData(0, 0, newCanvas.width, newCanvas.height); this.worker.postMessage({ action: 'imageData', data: {imageData} });}async blurInWorker (video: HTMLVideoElement, canvas: HTMLCanvasElement) { this.worker = new myWorker(''); this.worker.addEventListener('message', (event) => { if(event.data.inited) { this.drawImageData(this.workerCanvas, video); } else if(event.data.segmentation) { bodyPix.drawBokehEffect( canvas, video, event.data.segmentation, BodyPix.option.backgroundBlurAmount, BodyPix.option.edgeBlurAmount, BodyPix.option.flipHorizontal); this.bluring && this.drawImageData(this.workerCanvas, video); } }) this.worker.postMessage({action: 'init', data: null});}async unBlurBackground (canvas: HTMLCanvasElement, video: HTMLVideoElement) { this.bluring = false; this.worker.terminate(); this.worker = null; canvas?.getContext('2d')?.clearRect(0, 0, canvas.width, canvas.height); this.workerCanvas?.getContext('2d')?.clearRect(0, 0, this.workerCanvas.width, this.workerCanvas.height); this.workerCanvas = null;}OffScreenCanvas实现
//worker中let offscreen = nulll;let context = null; case 'offscreen': offscreen = new OffscreenCanvas(data.width, data.height); context = offscreen.getContext('2d'); break; case 'imageBitmap': offscreen.getContext('2d').drawImage(event.data.imageBitmap, 0, 0); body.net.segmentPerson(offscreen, BodyPix.option.config).then((segmentation) => { requestAnimationFrame(() => { webWorker.postMessage({segmentation}); }) }); break;//主体中const [track] = video.srcObject.getVideoTracks();const imageCapture = new ImageCapture(track);imageCapture.grabFrame().then(imageBitmap => { this.worker.postMessage({ imageBitmap });});bodyPix帧率问题还在钻研中...
参考资料
bodyPix github: https://github.com/tensorflow...
背景虚化demo: https://segmentfault.com/a/11...
bodyPix其余应用: https://segmentfault.com/a/11...
webWorker优化bodyPix: https://segmentfault.com/a/11...
webWorker应用: https://www.ruanyifeng.com/bl...