Cloud storage technology in the era of AI big data

For the video cloud storage technology, the mainstream security manufacturers in the industry can be said to have done a good job. Security manufacturers combine the underlying technology of cloud storage with security-specific streaming media to form a security-independent cloud direct storage product. No need to pull the external device, the storage can directly receive the data transmitted from the front end. But nowadays, the security industry has undergone earth-shaking changes. The video stream is completely unable to represent the data characteristics of the security industry. The AI ​​era is coming.

At present, the AI ​​data content in the industry mainly includes face data and structured data, including motor vehicles, non-motor vehicles, and pedestrians. The data type includes a series of unstructured data such as pictures, snapshot records, alarm records, and image attribute information. This type of data is characterized by fragmentation and is different from the video stream data type. The video stream guarantees continuous writing and the file packing size is relatively uniform. However, fragmented files are unknown due to their size and number. Scattered writes consume a lot of CPU and hard disk resources. For the CPU, you need to handle many threads at the same time. For hard drives, the head requires constant lane change addressing, which greatly reduces the life of the hard drive.

For this particular type of data, traditional streaming services cannot be processed. At present, mainstream security vendors have specially developed software for pulling such data streams, and the storage functions can be realized by installing them in general storage hardware. Because it is an emerging market, the storage of a single device can be satisfied in most scenarios. However, with the popularity of AI, the amount of data will continue to increase. For a city, in order to grasp the traffic conditions in the city, it needs to be collected. The number of vehicles, the congestion information, and the traffic flow direction of each road and every intersection. Through the data after the algorithm, the operation status of the urban traffic can be simulated, so as to predict the trend of the next second, and make an early warning plan to realize the real era of big data. When the data scale is expanded to a certain extent, the underlying cloud storage mechanism will be the technical support that people have to consider. However, the problem arises. Traditional security cloud storage only has access to video and cannot actively acquire structured data. Therefore, in the short term, this AI data cloud storage is bound to become the mainstream of the storage application layer.

Although the integrated AI data cloud storage can be realized through the docking of the application layer and the underlying layer, how will cloud storage respond when data types are further evolved and new data structures emerge? It is not a long-term solution to do compatible development blindly, and it will waste manpower and resources. To make matters worse, if there are multiple data types in one site, it is necessary to deploy multiple sets of cloud storage to store different data, which is a waste of storage space, high capital cost and low feasibility.

In order to solve this dilemma, we need a new development model, that is, heterogeneous cloud storage, the storage application layer and the file management layer and resource allocation layer are independently developed and deployed, so that the cloud storage underlying and hardware are done. Vendors can concentrate on ensuring the stability of the storage mechanism, and application vendors can concentrate on the compatibility of different data types. As long as the underlying standardization is done, major security and storage vendors can form a stable ecological cooperation. One party provides physical resources, and one party provides upper-layer services, which is no longer limited to the product model of software and hardware. On this basis, some manufacturers that are limited by capital investment can even develop their own cloud services. The upper application software can even be stored in the cloud as a common resource for end users to develop their own professional storage services.

The future has come, I believe that in the near future, it is bound to see major security vendors, algorithm vendors, and storage vendors working together to build a unified AI storage ecosystem, providing powerful data support for AI's scenario and civilization. (Author: Jin Fu Bangpeng Laroche)