Real-time surveillance and remote monitoring are gaining large importance in a wide range of industries such as Oil and Gas, Power Grid, Industrial Automation, Smart Buildings, etc. The demand for high-quality real-time HD Video Streaming / 4K Video Streaming designs in surveillance applications is higher than ever and this calls for compute-intensive image processing designs and advanced encoding and decoding techniques. Advanced camera architectures include an onboard microprocessor or FPGA with visual analytics and image processing algorithms for pixel pre-processing and decision-making within the camera. Complemented by AI Algorithms, these advanced cameras facilitate real-time communication with the central command center and stream footages of significance to aid user-level decision making. The advent of digital technologies and rapid development in Sensor technologies and embedded electronics, particularly the arrival of compute-intensive SOCs, high-speed DSPs, high-speed wireless / wired interfaces, video streaming protocols and cloud technologies among others enabled cameras to go beyond traditional surveillance and facilitate advanced AI-based vision applications such as safe-zone monitoring, No-go zone, object detection, etc.

The design and implementation of a high-speed, real-time 4K video streaming camera requires expertise in a wide array of embedded technologies. This article is an attempt to dive a little deeper into the design of a high-end real-time HD video streaming designs / 4K video streaming designs for camera in surveillance applications. We explore the basics of camera architecture and video streaming along with several streaming protocols, selection of sensors and processing engines, system software requirements and cloud integration among others.

4K Video Streaming – Camera Design Considerations

– Key System Components

Image Sensors

The image sensor is the eye of a camera system which contains millions of discrete photodetector sites called pixels. The fundamental role of sensors is to convert light falling on the lens into electrical signals, which in turn will be converted to digital signals by the processor of the camera.

Image sensors make a big impact when it comes to the performance of the camera due to various factors such as size, weight, power and cost. Choosing the right sensor is key to the design of a high-quality camera for surveillance applications. Selection of the sensor is influenced by several factors such as frame rates, resolution, pixel size, power consumption, quantum efficiency and FoV among others.

Sensor Interface

The selection of sensor interface is another critical factor that impacts the performance of a real-time 4K video streaming camera. The MIPI CSI (1/2/3) is one of the most preferred sensor interfaces currently in use. Since MIPI CSI offers a lean integration, several leading Chip manufacturers such as NXP, Qualcomm and Intel among others are adopting this technology for industrial and surveillance markets. CMOS sensors with MIPI CSI interfaces enable quick data transfer from the sensor to the host SoC of the system, facilitating high-speed and system performance. Developers can also use high-speed interfaces such as GMSL and FPD Link in applications where the processing of signals happens at a different platform or system. These interfaces greatly impact the overall performance of the camera, which is key to real-time 4k video streaming applications.

Key Components of a 4K Video Streaming Camera

Camera Interfaces

Identifying the right camera interface with high-speed video streaming feature is equally critical. A developer can rely on high-speed interfaces such as USB 3.2, CoaXPress 2.0, Thunderbolt3, DCAM, CameraLink HS (Framegrabber) and Gigabit Ethernet, among others.

Signal Processing Engines

There are numerous SOMs and SBCs based on leading GPUs, SOCs and FPGAs in the market, which are ideal for a high-end video streaming camera design. Identifying the right processor from this large pool might be a tough task. The processing engine must support the ever-evolving video encoding, decoding and application processing requirements of the camera system. In addition, the processor should aid key camera parameter settings like autofocus, auto exposure, and white balance along with noise reduction, pixel correction and colour interpolation, among others. The advent of hardware accelerated video encoding (GPU) and 4K Video streaming is greatly enabling imaging applications, especially surveillance and machine vision applications by providing breakthrough performance.

GPUs offer more computing power over a CPU, which is indispensable to high-performance, real-time 4K video streaming designs. Processing platforms that support hardware accelerated media encoding and decoding would be an ideal choice for camera developers. Cameras meant for HD / 4K video streaming applications must be of high resolution, high frame rate, industry-leading image compression and streaming capabilities, and low power consumption.

Advanced low-power embedded processors with built-in video analytics are gaining high popularity among developers as they aid real-time video streaming and communication. HD video streaming cameras with edge computing capabilities are ideal for time-sensitive applications and help make intelligent decisions at the edge. Such systems are highly recommended for remote applications, where there is limited no connectivity to cloud or a central command center.

– System Software

Operating System

4K Video Streaming – Software Architecture

Despite the emergence of several operating systems, Linux rules the embedded world. Linux is ideal for HD Streaming Camera designs due to its stability and networking capabilities. As Linux is an open-source operating system, developers can easily and quickly make changes and redistribute it. One can meet the unique requirements of the project using Linux, while easily addressing critical factors including power consumption, data streaming, and any other challenges imposed by the hardware configuration or other software elements. Linux also allows easy customization and integration of I/Os and enables faster time to market.

Sensor Drivers

Sensor driver development and integration is another crucial part of camera design. Most of the sensor modules come with drivers, however developers can also write camera sensor driver as per the specific needs of the end product. Sensor drivers should address several key components such as clock frequencies, frame size, frame interval, signal transmission, stream controls, sensor power management and sensor control framework among others. In addition, developers have to address sensor interface design, protocol development and integration as well.

OpenCV

OpenCV, computer vision library has become the most popular image processing platform among the developers of imaging solutions in recent years. Developers can use OpenCV libraries to implement advanced image processing algorithms in their video streaming solutions. The platform offers more than 2000 algorithms related to image processing. Since OpenCV libraries are written in C++, developers can easily integrate these software stacks into their designs. OpenCV enables the developers to focus on video capture, image processing and analysis including features like face detection and object detection. OpenCV also offers storing, reading and writing images in an n-dimensional array and performs image processing techniques such as image conversion, blur, filtering, thresholding, rotation, scaling, histogram equalization, and many more.

Algorithms

Algorithms can be implemented at two layers in a high performance, real-time video streaming camera design. While the first layer specifically caters to image processing, the second layer of algorithms adds intelligence to the visuals captured. High-definition cameras capture images in high detail. However, the data captured may need further enhancements for effective analysis and accurate decision making. Implementation of visual algorithms, especially those aid in image correction and enhancement has become a minimal requirement for modern surveillance cameras. The commonly implemented algorithms include autofocus, auto exposure, histogram, color balancing, and focus bracketing among others. The second layer of algorithm is implemented on advanced surveillance cameras that leverage complex visual analytics powered by AI-algorithms to provide real-time situational awareness. These systems combine video capturing, processing, video analytics and real-time communication for effective situational awareness and decision making. These 4K Video Streaming cameras are finding increased applications in automated detection of fire, smoke, people counting, safe-zone monitoring, facial recognition, human pose estimation, scene recognition, etc.

Key Performance and Feature Considerations

HD / 4K Video Streaming

High-definition video streaming (4K Video Streaming) is getting more popular, however, in industrial and surveillance applications, real-time HD video streaming poses many challenges due to the enormous amount of data the system handles. The transmission of large volume of data to a remote system with minimal latency is a major challenge in HD / 4K video streaming. This can be addressed to a great extent by selecting appropriate processors, preferably FPGAs. FPGA has several benefits over conventional processors and the developers can easily integrate complex video analytics algorithms in the device. The availability of a large number of logic cells and embedded DSP blocks and flexible connectivity options make FPGA a powerhouse that can handle faster image processing and real-time 4K video streaming.

Video Compression

Raw video files are digitized, compressed and packetized by the processor for faster transmission to the remote monitoring system. Cameras use various video compression technologies and algorithms to compress a video, which can be further reconstructed by the host PC to its original resolution and format. Developer can use standards such as MJPEG, MPEG-4, H.263, H.264, H.265, and H.265+ for image compression for easy transfer.

Video Streaming Protocols

4K Video Streaming – Protocols

For real-time video streaming over a network, several video streaming protocols can be implemented in cameras. The commonly used protocols include Real-Time Streaming Protocol (RTSP), which acts as a control protocol between streaming severs (camera and the host PC) and facilitates efficient delivery of streamed multimedia, Real-time Transport Protocol (RTP), transports media stream over the network and HTTP, a protocol that provides a set of rules for transferring multimedia files including images, sound and video over IP networks. The RTSP in conjunction with RTP and RTCP, supports low-latency streaming, making it an ideal choice for high-speed streaming Camera designs.

Small Form-factor, Low-power Design

Today, embedded product developers are competing to reduce the size of their products while increasing the efficiency manifold. The advent of low power, small form-factor multi-core SoCs, graphic accelerators and memory technologies is aiding designers to build small footprint, power-efficient cameras. Industrial and surveillance cameras demand low-power designs due to the nature of their deployment. Such cameras designed for demanding environments must offer low energy dissipation while maximizing battery life. Developers can also consider PoE for wired camera designs. Surveillance systems heavily rely on either NVRs or cloud storage to save video footage for future references and analysis. Since a physical network connectivity is important for wired applications, PoE ensures that the camera is always powered and connected.

4K Video Streaming Designs require multiple disciplines

Real-time HD video streaming is a need of the hour. From Industrial robots to sophisticated drones, security and surveillance to medical applications, high-quality imaging is finding a crucial role in embedded world. Advanced cameras are capable of capturing and transferring a large amount of data, leveraging high-speed industrial interfaces and protocols. The idea of this Blog is to understand the fundamental components and explore key design considerations while developing a high performance, real-time video streaming camera for surveillance applications.

By using leading SOCs, latest sensors and integrating various image processing and streaming platforms, one can develop a state-of-the-art camera for real-time video streaming. Mechanical design including Sensor housing, meeting environmental and temperature standards are also critical to the successful design of a camera. As a developer one should have expertise in selection and integration of camera sensors, HD video streaming protocols, implementation of system software and sensor drivers, development and integration of camera interfaces, image signal processing, sensor tuning, image/video compression algorithms, image enhancement techniques, etc. to realize a high-end real-time video streaming design.

This blog is a concise version of the Article ‘Real-time video streaming for surveillance applications’ by Mistral Solutions, published in Electronic Specifier.