.NET 摄像头采集

编程知识2152024-08-29评论

本文主要介绍摄像头（相机）如何采集数据，用于类似摄像头本地显示软件，以及流媒体数据传输场景如传屏、视讯会议等。

摄像头采集有多种方案，如AForge.NET、WPFMediaKit、OpenCvSharp、EmguCv、DirectShow.NET、MediaCaptre（UWP），网上一些文章以及github已经有很多介绍，这里总结、确认技术选型给大家一个参考

1. AForge.NET

AForge视频库是基于DirectShow技术开发的，提供了捕捉、处理和显示视频流接口，以及图像丰富的图像处理功能，如滤镜、特征提取和物体检测。详见官网开源仓库 andrewkirillov/AForge.NET(github.com)

我们下面看下AForge录制代码，安装Nuget包依赖：

1     <PackageReference Include="AForge.Video" Version="2.2.5" />
2     <PackageReference Include="AForge.Video.DirectShow" Version="2.2.5" />
3     <PackageReference Include="System.Drawing.Common" Version="8.0.8" />

摄像头显示：

1privatevoidStartButton_OnClick(object sender, RoutedEventArgs e)2{3//获取所有视频输入设备4var videoDevices = newFilterInfoCollection(FilterCategory.VideoInputDevice);5if (videoDevices.Count > 0)6{7//选择第一个视频输入设备8var videoSource = newVideoCaptureDevice(videoDevices[0].MonikerString);9//注册NewFrame事件处理程序10 videoSource.NewFrame += newNewFrameEventHandler(videoSource_NewFrame);11//开始摄像头视频源12videoSource.Start();13}14}1516privateasyncvoidvideoSource_NewFrame(object sender, NewFrameEventArgs eventArgs)17{18//获取当前的视频帧，显示19var image =ToBitmapImage(eventArgs.Frame);20await Dispatcher.InvokeAsync(() => { CaptureImage.Source = image; });21}

摄像头录制视频流：

1privateasyncvoidvideoSource_NewFrame1(object sender, NewFrameEventArgs eventArgs)2{3//获取当前的视频帧4//将Bitmap转换为byte[]，用于流媒体传输5byte[] byteArray = BitmapToByteArray(eventArgs.Frame, outintstride);6//将byte[]转换为BitmapImage，用于临时展示7 BitmapImage image = ByteArrayToBitmapImage(byteArray, eventArgs.Frame.Width, eventArgs.Frame.Height, stride, eventArgs.Frame.PixelFormat);8await Dispatcher.InvokeAsync(() => { CaptureImage.Source = image; });9}

其中的数据转换，这里要把分辨率stride同byte[]数据一同储存，不然后续数据是无法处理的：

1//将Bitmap转换为byte[]2publicbyte[] BitmapToByteArray(Bitmap bitmap, outintstride)3{4 Rectangle rect = newRectangle(0,0, bitmap.Width, bitmap.Height);5 BitmapData bitmapData = bitmap.LockBits(rect, ImageLockMode.ReadOnly, bitmap.PixelFormat);6//stride是分辨率水平值，如38407 stride =bitmapData.Stride;8int bytes = Math.Abs(bitmapData.Stride) *bitmap.Height;9byte[] rgbValues = newbyte[bytes];1011//复制位图数据到字节数组12 Marshal.Copy(bitmapData.Scan0, rgbValues, 0, bytes);1314bitmap.UnlockBits(bitmapData);15returnrgbValues;16}1718//将byte[]转换为BitmapImage19public BitmapImage ByteArrayToBitmapImage(byte[] byteArray, intwidth,intheight,int stride, PixelFormat pixelFormat)20{21var bitmapImage = newBitmapImage();22using(var memoryStream = newMemoryStream())23{24var bmp = new Bitmap(width, height, stride, pixelFormat, Marshal.UnsafeAddrOfPinnedArrayElement(byteArray, 0));25//保存到MemoryStream中26 bmp.Save(memoryStream, System.Drawing.Imaging.ImageFormat.Bmp);27memoryStream.Seek(0, SeekOrigin.Begin);28bitmapImage.BeginInit();29 bitmapImage.StreamSource =memoryStream;30 bitmapImage.CacheOption =BitmapCacheOption.OnLoad;31bitmapImage.EndInit();32bitmapImage.Freeze();33}34returnbitmapImage;35}

详见Demo代码 kybs00/AForgeNETDemo (github.com)

经验证，延迟较大，对比Windows系统相机不够清晰

2. WPFMediaKit

WPFMediaKit也是基于DirectShow的，它提供了一些封装便于在WPF应用中使用媒体功能。

使用 WPFMediaKit 要录制摄像头视频，需要结合 WPFMediaKit 提供的视频捕获功能和其他库（例如 AForge 或 FFmpeg）来实现录制功能。

这里引用WPFMediaKit、AForge.Video.FFMPEG俩个Nuget包，然后通过定时器捕获当前视频帧：

1var bitmap = new Bitmap(videoCaptureElement.Width, videoCaptureElement.Height, System.Drawing.Imaging.PixelFormat.Format24bppRgb);2var bitmapData = bitmap.LockBits(newRectangle(0,0, bitmap.Width, bitmap.Height),System.Drawing.Imaging.ImageLockMode.WriteOnly,bitmap.PixelFormat);3try4{5videoCaptureElement.VideoCaptureDevice.GetCurrentVideoFrame(out IntPtr frame);6System.Runtime.InteropServices.Marshal.Copy(frame,0, bitmapData.Scan0, videoCaptureElement.Width * videoCaptureElement.Height * 3);7}8finally9{10bitmap.UnlockBits(bitmapData);11}

这个定时器的实现比较low。延时较低、较流畅，但与Win系统相机对比也是不够清晰

3.MediaCapture（UWP）

MediaCapture是Windows 8及以上版本的WinRT API，专为捕获音频、视频和照片设计。

MediaCaptuer是UWP应用的API 使用 MediaCapture 捕获基本的照片、视频和音频 - UWP applications | Microsoft Learn，要在WPF内使用需要引入俩个Nuget包：

1 <PackageReference Include="Microsoft.Toolkit.Wpf.UI.XamlHost"Version="6.1.2"/>2 <PackageReference Include="Microsoft.Windows.SDK.Contracts"Version="10.0.26100.1"/>

初始化MediaCapture:

1var mediaCapture =newMediaCapture();2var videos = awaitDeviceInformation.FindAllAsync(DeviceClass.VideoCapture);3var settings = newMediaCaptureInitializationSettings()4{5 VideoDeviceId = videos[0].Id,6 StreamingCaptureMode =StreamingCaptureMode.Video,7};8awaitmediaCapture.InitializeAsync(settings);

分几个场景，分别输出Demo。录制本地文件，注释已经很详细了并不重复解释，直接看代码：

1private MediaCapture _mediaCapture;2private InMemoryRandomAccessStream _randomAccessStream;3privateasyncvoidStartButton_OnClick(object sender, RoutedEventArgs e)4{5// 1. 初始化 MediaCapture 对象6var mediaCapture = _mediaCapture = newMediaCapture();7var videos = awaitDeviceInformation.FindAllAsync(DeviceClass.VideoCapture);8var settings = newMediaCaptureInitializationSettings()9{10 VideoDeviceId = videos[0].Id,11 StreamingCaptureMode =StreamingCaptureMode.Video,12};13awaitmediaCapture.InitializeAsync(settings);1415// 2. 设置要录制的数据流16var randomAccessStream = _randomAccessStream = newInMemoryRandomAccessStream();17// 3. 配置录制的视频设置18var mediaEncodingProfile =MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Auto);19// 4. 开始录制20await mediaCapture.StartRecordToStreamAsync(mediaEncodingProfile, randomAccessStream);21}2223privateasyncvoidStopButton_OnClick(object sender, RoutedEventArgs e)24{25//停止录制26await_mediaCapture.StopRecordAsync();27//处理录制后的数据,保存至"C:\Users\XXX\Videos\RecordedVideo.mp4"28var storageFolder =Windows.Storage.KnownFolders.VideosLibrary;29var file = awaitstorageFolder.CreateFileAsync("RecordedVideo.mp4", Windows.Storage.CreationCollisionOption.GenerateUniqueName);30usingvar fileStream = awaitfile.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite);31awaitRandomAccessStream.CopyAndCloseAsync(_randomAccessStream.GetInputStreamAt(0), fileStream.GetOutputStreamAt(0));32_randomAccessStream.Dispose();33}

摄像头显示，通过UWP-WindowsXamlHost承载画面（置顶）：

1privateasyncvoidStartButton_OnClick(object sender, RoutedEventArgs e)2{3 _mediaCapture = newMediaCapture();4var videos = awaitDeviceInformation.FindAllAsync(DeviceClass.VideoCapture);5var settings = newMediaCaptureInitializationSettings()6{7 VideoDeviceId = videos[0].Id,8 StreamingCaptureMode =StreamingCaptureMode.Video,9};10await_mediaCapture.InitializeAsync(settings);11//显示WindowsXamlHost12 VideoViewHost.Visibility =Visibility.Visible;13//绑定画面源14 _captureElement.Source =_mediaCapture;15await_mediaCapture.StartPreviewAsync();16}

MediaCapture是UWP平台的实现方案，直接给CaptureElement赋值绑定画面源。直接用CaptureElement渲染速度很快，这个实现逻辑同windows系统相机是一样的

另外，使用MediaCapture也可以捕获画面帧事件，用于流媒体数据捕获收集：

1//配置视频帧读取器2var frameSource = mediaCapture.FrameSources.Values.FirstOrDefault(source => source.Info.MediaStreamType ==MediaStreamType.VideoRecord);3 _frameReader = await mediaCapture.CreateFrameReaderAsync(frameSource, MediaEncodingSubtypes.Argb32);4 _frameReader.FrameArrived +=FrameReader_FrameArrived;5await_frameReader.StartAsync();

如下方所示，监听FrameArrived，使用Windows.UI.Xaml.Media.Imaging.BitmapImage渲染展示（仅用于展示，延迟很高）：

1privateasyncvoid FrameReader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)2{3var frame =sender.TryAcquireLatestFrame();4if (frame != null)5{6var bitmap = frame.VideoMediaFrame?.SoftwareBitmap;7if (bitmap != null)8{9//在这里对每一帧进行处理10awaitDispatcher.InvokeAsync(async () =>11{12var bitmapImage = awaitConvertSoftwareBitmapToBitmapImageAsync(bitmap);13 _captureImage.Source =bitmapImage;14});15}16}17}

如需要将SoftwareBitmap转为buffer字节数据，可以按如下处理：

1publicasyncTask<byte[]> SoftwareBitmapToByteArrayAsync(SoftwareBitmap softwareBitmap)2{3//使用InMemoryRandomAccessStream来存储图像数据4usingvar stream = newInMemoryRandomAccessStream();5//创建位图编码器6var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, stream);7//转换为BGRA8格式，如果当前格式不同8var bitmap = SoftwareBitmap.Convert(softwareBitmap, BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied);9encoder.SetSoftwareBitmap(bitmap);10awaitencoder.FlushAsync();11bitmap.Dispose();1213//读取字节数据14usingvar reader = newDataReader(stream.GetInputStreamAt(0));15byte[] byteArray = newbyte[stream.Size];16awaitreader.LoadAsync((uint)stream.Size);17reader.ReadBytes(byteArray);1819returnbyteArray;20}