WHEN SURVEILLANCE MEETS INTELLIGENCE

scenario they may be centralized or de-centralized and therefore not co-located. These are the fundamental entities of a typical VMS system:

• Management
• Analytics, monitoring & event triggering
• Streaming
• Recording & Storage
• Display

It is important to note that while all the above entities have an important role in a complete VMS deployment, not all of them are mandatory for a viable system. In fact, apart from the management entity, any combination of the others is valid.

The deployment scheme of cameras is also quite diverse resulting in multiple optional configurations. Each of these configurations implies different settings and different KPIs. We will try to address some typical configuration types.

Some select deployment configurations are shown in the following figure.

Scalability is a fundamental property to be addressed in large scale video management system design. It is inspired by the growing demand to cover wider areas and more scenarios and events, as well as the growing traction of non-intrusive systems.
Scalability is translated into multiple parameters that eventually determine the system requirements and may in turn impact the deployment configuration. At the platform level, the number of channels (aka ‘streams’) is the primary factor. At the stream level, scalability can be attributed to stream properties. In the case of analytics, it is typically the number of operations per second per channel which is determined by:

• Resolution, measured in pixels-per-frame (spatial-domain scaling)
• Frame rate, measured in frames-per-second (time-domain scaling)
• Functionality scaling, measured in operation-per-pixel, which in AI processing case, is determined by the neural network model computation intensity.
• Pipeline scaling, measured by the number of tasks for the processing pipeline, representing its complexity. For instance, a typical processing flow may involve multi-stage processing of the source stream to reach the desired output. Complex pipelines are beyond the scope of this write-up but involve a combination of multiple tasks, scheduled to run in a sequence. These may run sequentially or in parallel and may involve dynamics that are dependent on the specific data observed in the video stream. For example, an Automatic License Plate Recognition (ALPR) pipeline, requires the identification of a vehicle, the identification of the vehicle’s license plate, and the recognition of the characters in the license plate.

The large and diverse nature of VMS deployment scenarios is derived from the different venues in which cameras are deployed.

• Management
• Analytics, monitoring & event triggering
• Streaming
• Recording & Storage
• Display

Due to the above properties, we can expect that larger scale deployments are more likely to be managed and that is the focus of the content from this point onwards.

The following tables summarize the entities as described above, and the different deployment configurations, along with their main functions, most important resource they require and the main KPIs to optimize to best serve their purpose.

• Management
• Analytics, monitoring & event triggering
• Streaming
• Recording & Storage
• Display

In any camera deployment, the source data is the raw video as acquired by the endpoints. This raw data, if not processed, imposes very demanding bandwidth requirements that need to be eventually processed, transferred, stored and retrieved.

The streaming entity serves as the data plane of the VMS, primarily purposed to manipulate the raw data, and ensure its delivery from one point to another along the deployment infrastructure.

The information that the raw data holds, however, is the actual purpose of deploying all these endpoints in the first place. With the application of artificial intelligence, the amount of information that a video stream carries can be delivered in much less bandwidth than the raw video; let alone insights drawn from that information and actions taken in response to specific events.

Furthermore, along with the meaningful data comes unnecessary data content which could be harmful in some cases. It may, unintentionally yet unnecessarily, include information that is not relevant for the subject matter and could potentially be an impediment to people’s privacy.

Equipping the streaming entity with the capability to identify the content of the stream both in the spatial, as well as the temporal domain, and determine in real-time what part of the raw data is unnecessary and what part should be delivered, when and how, allows potential reduction in the average bandwidth consumed and better control over personal identifiable information carried across the system.

In the context of the streaming entity, AI gives the opportunity to determine what content is being transferred and when.

Including AI in the streaming entity may allow few possible configurations and possible intelligence models that may be applied to them:

The key factors when introducing intelligence into the streaming entity are:

• No. of streams / channels [#]
• Per channel frame rate [fps]
• Encoding capacity [bps]
• Latency [s]
• Bandwidth [bps]

More specifically, a typical 4K30 stream, amounts to ~7.5 Gbps of raw data. If encoded without data analysis it may reach up to 45 MBps with H.264 (or ~22 Mbps with H.265). By introducing analysis of the stream, the number of streams that can fit into a given bandwidth can be increased, or alternatively, a gateway box with a given encoding capacity can handle more streams simultaneously.

For example, in a typical deployment, with proper video coverage, >90% of the video has no meaningful information, which means that only <10% of the video should be streamed. Furthermore, when important information does appear, not all the cameras will capture that important information simultaneously. Almost all areas are covered with 2-4 overlapping zones thereby reducing the simultaneous nature of events by at least 50%. An even more advanced solution, that applies AI also for spatial reasoning, may recover more bandwidth if delivering only the active region of the frame which may further reduce the input bandwidth by a factor of 2-4.

In such a case applying proper analytics can handle x10-20 more streams compared to a similar scenario in which full frame is uploaded. Such an approach may save the expense for upgrading the connectivity infrastructure due to the bandwidth demand, resulting in ~$20-30 saving per camera per month (50-70% saving of the monthly cost). In a 10,000 camera deployment, this amounts to >$2M saving per year.

The capacity of the analytics is obviously a function of the analytics deployed. It may vary widely depending on the resolution, frame rate and number of stages in the processing pipeline.

A high accuracy state-of-the-art object detector running at Full HD input resolution, and 30 fps may require 1-3 Tera Operations per Second (TOPS) per stream (~x10s Kops/pix/stream is a reasonable figure of merit). Typically, a multi-stage operation is applied, requiring higher peak capacity to address real-life use case.

Design Rules

• Input Bandwidth constraint:N ∙ W ∙ H ∙ R ≤ Camera connection bandwidth
• Output Bandwidth constraint:N ∙ σ ∙ W ∙ H ∙ R ≤ Downstream link bandwidth
• Video Encoding constraint:N ∙ W ∙ H ∙ R ≤ Encoder bandwidth
• AI capacity constraint :N ∙ R ∙ [W ∙ H ∙ C1 + NROI ∙ WROI ∙ C2 ] ≤ Available TOPS

N – number of streams

W – frame width (pixels)

H – frame height (pixels)

R – frame rate (fps)

σ - compression ratio (%)

C1, C2 – AI workload compute capacity of the 1st, 2nd stage in the pipeline respectively (ops / pixel)

NROI – Number of regions of interest (ROI)

WROI, HROI – Average width & height of an ROI box (pixels)

1. Refer to Hailo Model Explorer for further details about model compute requirements.
2. This equation assumes a typical 2-stage pipeline for analytics (it can easily be extended for more advanced pipelines with additional stages)

The analytics entity is the most recent entrant to traditional VMS solutions, emerging from the rapid evolution in this field. In terms of possible deployment configurations, the analytics entity can be included either as an add-on software component in one or more of the existing nodes, or as a separate hardware processing component.

In the former case, the analytics relies on the hosting hardware platform and its capabilities, while in the latter a new type of node is introduced. This node, which is sometimes called an AI-box or an AI-gateway, enables adding more compute power with the purpose of offloading the next stage in the pipeline thereby improving the overall system utilization.

In spite of the additional cost of another piece of hardware, In many cases the introduction of such a gateway improves the overall system capacity resulting in a better total cost of ownership (which can be quantified as the cost per stream for the overall system). In other cases, it is the most practical approach for upgrading an existing installed base without replacing or rewiring existing nodes.

The analytics entity is AI-centric and gives the opportunity to trigger events requiring further action only when needed, thereby lowering system load

The main factors that govern the properties of the analytics component are the following:

• Per channel frame rate (fps)
• Decoding capacity (pixels/sec)
• Event rate (events/sec)
• Latency (sec)
• Accuracy (% miss-rate; % false alarms)

Design Rules

• AI capacity constraint:Actual TOPS = N ∙ W ∙ H ∙ C ∙ R ≤ Available TOPS
• Number of streams constraint:

N – number of streams

W – frame width (pixels)

H – frame height (pixels)

C – AI workload compute intensity (ops/pixel)

R – frame rate (fps)

D – Decoder bandwidth (bits /sec)

B – Stream bandwidth (bits / sec)

For instance, an analytics box which addresses simultaneously 32 channels of FHD at 30 fps will require an actual AI capacity of ~50 TOPS (assuming 1.5 TOPS per channel for analytics).

This also implies a decoding capacity that can support 32 channels. An H.264 encoded stream will typically reach ~10 Mbps.

A platform supporting 320 Mbps decoding bandwidth is applicable in this case.

If we wish to consider 4K streaming, the analytics will scale nearly linearly while decoding ratio scaling is sub-linear and in the case of 4K the stream bandwidth will typically reach ~50Mbps.

In this case, the same platform with 320 Mbps will be able to handle only ~6 channels, even though the analytics alone can handle more than that.

(Another option, is to enjoy the benefit of a more advanced encoding such as H.265 which provides a x2 factor on the encoding/decoding performance)

Contrary to the analytics function, storage is a longstanding feature of any video deployment, facilitating the ability to record some or all of the streams over time. Its most important property is obviously the storage time and cost for a given capacity.

Other important parameters are the bandwidth into the system, storage access time, and finally the ability to search & retrieve required information.

Applying AI in a storage entity gives rise to multiple use cases such as the ability to dramatically improve the utilization of the available storage resources by making the storage content-aware. It also may be leveraged for on-the-fly indexing enabling metadata to be attached to the video for easier retrieval, and free text search to find events of relevance.

AI supported storage allows to determine what content is being stored, and when to suspend and resume

To summarize, the key factors of the storage entity include:

• No. of streams
• Per channel frame rate (fps)
• Storage duration (hours) Storage space (terabytes)
• Access (Read & Write) speed (sec)
• Search speed (sec)

Design Rules

• Actual Bandwidth ~ N ∙ σ(C) ∙ (W ∙ H ∙ R) ≤ Write speed
• Storage duration ~ N ∙

N – number of streams

W – frame width (pixels)

H – frame height (pixels)

C – AI workload compute intensity (ops/pixel)

R – frame rate (fps) σ - compression ratio (%)

In most VMS deployments, there is some sort of human interface. It can either be for the initial deployment phase where health checks are conducted by the solution provider, or after the initial deployment, for monitoring. Either as a complementary aid mainly used for debugging purposes or as an integral part of the overall system deployment in the form of display monitor wall in some facility management control room.

Beyond the obvious function that an AI-driven component can add to a display, such as local processing of the currently displayed view for extracting metadata, or tracking of objects, AI can be tasked to manipulate the display, to pop up relevant streams as soon as something meaningful is happening in them, or highlight specific areas in the image while dimming others based on activity.

AI supported display component can be used to determine what content is being displayed, and when it is displayed

What determines the performance of the display entity are the following factors:

• Number of displays
• Input stream resolution (pixels)
• Display resolution (pixels)
• Decoding speed (pixels/sec)

Design Rules

M – number of display monitors

W – Stream frame width (pixels)

H – Stream frame height (pixels)

MW – Display monitor width (pixels)

MH – Display monitor Height (pixels)

R – Refresh rate of display monitor (fps)

D – Decoding speed of video stream (pixels/sec)