The AI Architecture for Semantic Discovery of Videos model with LLM
Enterprises deal with a surprising number of videos due to increasing usage. Remote work is one of the drivers that has replaced traditional documentation with videos. Afterall if images are 1000 words each second of video worth at least that much. Although videos do usually contain more than a few frames where each frame is an image. One of the most important factors is that videos also contain the audio track which carries the meat of content when combined with spoken words inside it.
The Challenges
Customers expect us to go above and beyond just searching by the title or description. At CircleHD we do support more than that. Speech to text, Channels metadata, Custom Fields, Taxonomy, tags and typo handling are to name a few. For advanced use cases we are able to deliver results in a faceted manner from those custom fields. Example, finding all videos uploaded from the headquarters or the video about the newest product. These work great when assisted by a human to fill those details while uploading the video or editing later to add more context to it. We use that information to discover similar videos. However these approaches are hard to scale and maintain consistency while keeping up with the growth. Therefore we bring in machine learning to the use. Large Language Models (LLM) have significant improvements that enable us to go beyond traditional discovery to provide customers with answers rather than just clips of videos in segments.
We respect customers’ privacy, therefore we needed to build a solution that prevents data leak by minimizing dependency on third party services. At the current landscape there are many services available with minimal to no respect for data privacy compliance we could not leverage that puts the solution at a bit disadvantageous situation. In this article we discuss many candidate solutions we considered that offer self hosting on AWS at the cost of speed of execution. This solution is only available to our multi tenant customers. We are working to bring the solution to enterprise deployments in the near future.
Solution
Most video context can be derived from the audio with help of speech to text if it’s meaningful. Like meeting recordings, talk shows and presentations. Sometimes the audio could be just background music or not included at all therefore a very little to use. The information presented in the video frames(Images) contains a lot of information such as text or objects. Extracting the semantic context from video could include a hybrid approach or object discovery and OCR. The architecture we employed is commonly known as Retrieval Augmented Generation (RAG). RAG is an application architecture that combines traditional search with generative artificial intelligence (GenAI) to discover information semantically rather than just word pattern matching.
Architecture
Closed Captioning
Some videos have closed captioning enabled. If it’s available it’s the best source of information. Youtube supports uploading closed captions that are produced alongside the video. And also been generating closed captions from the audio of the video using ASR as discussed below. When closed captions are available that is the best (richest) source of information in most cases even if it’s unstructured.
Audio
Audio to text works amazingly well with the current advancement of AI in that space. We can use any automatic speech recognition (ASR) solution such as self hosted OpenAI whisper or hosted solutions like Google Cloud Speech-to-Text, Amazon Transcribe, poly to name a few. We have tested most of them and the results have been more than desired for english. Some models started supporting international and regional languages, although the majority of business customers are happy with English alone.
Video
As said above this one is a bit tricky to get right. A video contains 25 frames per seconds on average. Each frame is a 1024×720 matrix of pixel data even at least at 720p HD resolution. We consider this as the baseline. To process each frame to OCR and object detection is a computationally expensive task. And a motion picture has frames interlaced that have a little to no change in most frames. It’s a technique that video codecs employ to compress and reduce the size of the videos even at a very high resolution. Therefore sampling would be more appropriate while computing becomes even more cheaper. For example only using a constant number of frames per second or every few seconds would work in most scenarios.
Example to produce one image per second.
Therefore this example will produce 600 images for a 10 minute long video. The next task is to derive the content from the images. Taken from here
OCR
Optical Character Recognition (OCR) is the commonly used technique for extracting textual information from images. Thankfully many OCR solutions are available with open source models. Tesseract and PaddleOCR are some popular ones. We had greater success with PaddleOCR for various reasons including support for international characters and unicode.
Classification
For advanced use cases such as object identification (such as packaging box dimension) or car model and color identification requires advanced computer vision techniques and custom trained vision models such as torch vision may be employed to further extract embedded information inside the frames. This is out of scope for this article.
Full Text Search (FTS)
Full text search allows searching documents (in our case the textual information extracted from the videos) from a prebuilt index. The index helps to keep the process fast and there are many self solutions such as Solr(Lucene) or ElasticSearch or Algolia (SAAS) that enable this functionality. Many database solutions like Postgres and MySQL offer full text search capabilities, however their capabilities are pretty limited in performance and are inferior to the prior solution. If faceting, paging or sub filtering is desired, there is a lot more involved than just the text matching itself.
Keeping the search external or dedicated to its own has many advantages. Full text search systems are lightning fast as they can hold the entire index in RAM and have dedicated algorithms for making use of it. Because there is no extra load on the database for computationally expensive string comparison it’s easier to scale.
The drawback is that the data needs to be synchronized and kept that way. The developers and operations need to maintain an additional system to perform this. But sooner or later this becomes a necessity to support the additional functionality necessary.
Embedding
Once the semi or unstructured data is available in form of text, it might be enough to search by the keywords, but not enough. Example if you would like to search for AI it would not match “Artificial Intelligence use case of LLM” videos that does not include any title or meta data with AI. Or Login while looking for a video with the title “Our newest authentication system”. LLM solves this by reducing the text or image or audio into a numeric representation called vectors. The process is called embedding. The way it works is that LLMs are neural networks that have been trained on a large volume of data. Given a new string it can return the string of numbers that represents the position of the content within its own model weights (think database) for simplicity. This article from Hugging face describes embedding in more detail.
Vector Store
Vector store is a specialized database that is capable of storing the vectors (string of float) and relevant procedures to use them by querying and matching. Many traditional databases such as Postgres can beautifully store the vectors in arrays alongside of the other structured data. The challenge however is putting the vectors to use and scaling them. SQL orintented databases wouldn’t have any use of the vectors in the tables in queries. Pgvector is a Postgres extension that enables vector queries inside postgres by supporting additional algorithms and optimization. (Also now obsolete pg_embedding) The vector operations are computationally expensive and could interfere and impact the regular database performance when employed together.
To solve this problem, an external vector store such as Faiss, PineCone, and other alternatives such as ChromaDB, Weviate, Milvus, QDrant may be preferable. This market is very hot, new vector database solutions are emerging. External vector stores may be a more preferable solution than using extensions for the same reason as full text search. To fill the void many Full text search solutions have been stepping up to support vectors and related operations. And have been offering a mix of vector search and full text search. We use this approach, which is also called hybrid search. More about that below. We use elastic search, therefore decided to use the embedded vector store to reduce maintenance complexity of another stack. Other dedicated vector stores provided better performance than both elastic vector stores but overall ROI wasn’t enough to justify that. We preferred to keep vector data in elastic over postgres because the elastic’s partitioning model was more scalable compared to the postgres for our multi-tenant implementation.
Limitation
Embedding is a lossy operation to derive the vectors from the content. The original document must be persisted separately. Current LLM models have a context size limit aka token limit. This notebook Massive Text Embedding Benchmark (MTEB) Leaderboard on hugging face keeps track of limits as well as benchmarks and token limits of most popular models.
Tokens are blocks of text (typically 4 characters). Which means they can only process a tiny bit of information at times. This is due to the limit of computing memory available. As GPUs become more powerful this limit could get less of concern in the future with the advancement of hardware. Therefore many architectures suggest use of chunking as a workaround to split large blocks of text into smaller blocks to get around this limitation. While this is an overhead, when deployed together with database queries and full text search, this has been proven as an useful mechanism to improve RAG performance due to the complexity of vector algorithms.
Legacy open source BERT-based models have a 512 token limit while popular OpenAI closed source ADA models have about 8000 token limit. At the time of writing, the top performing NV-Embed-v1 max token limit is 32768, Voyage-large-2-instruct 16000 Linq-Embed-Mistral 32768 on 30GB GPU memory instance hardware. This is increasing as we speak. If you are reading this in the future this limit might not be a concern for you however there will be a limit due to the nature of neural networks.
Querying
Vectors are beautiful numeric representations. The algorithms include linear algebraic functions to perform operations such as vector search and similarly. In our case we use it for searching and discovering similar videos. This approach is very popular for product recommendation in commercial applications. Cosine Similarity and Euclidean L2 normalize distance are 2 most common algorithms used to find distance between 2 vectors for similarity. We won’t go into details as they get more mathematically involved. Just note that most vector solutions have a variation of these algorithms made available for use. They are more computationally expensive operations on CPU. Some algorithms are optimized for running on GPU with assistance of acceleration such as CUDA from NVIDIA.
Caution
The query and embedding must use the same model. It can’t be mixed. The reason for that is every model has been trained differently with may be different data sets and different durations and especially due to initial randomization of the neural network parameters. When generating, embedding a different model would produce totally different vectors based on the weights and parameters of the model. And different vectors would not match. It’s like speaking in different languages.
Query Performance and Vector Indexing
Query can be computationally expensive the same way the database does, therefore the need of indexing is required to avoid table scan-like operation. Therefore a different indexing algorithm such as HNSW, IVFFLAT are popular to improve the performance for a traditional RAG system.
Result Serving and Ranking
Hybrid search can bring the best of both worlds, at least for our needs. It produces great results semantically. One disadvantage of this approach is that it can produce overwhelming results than previously possible. Ranking results are up to the needs and requires tuning of the queries according to the needs. Some customers like stale but accurate information while some expect most up to date information.
LLMs can also partially solve this by reducing the output by summarizing the results to a readable chunk. When matching results are passed to text summarizer (transformer) along with the original query, LLMs are smart enough to rank more accurate results with a text based summary to assist the parsing.of the results. A different and sophisticated model may be used for this purpose. We use a locally deployed LLAMA3 interference server over Ollama.
Conclusion
AI technologies such as LLM and computer vision are currently evolving at a faster speed than most of us can keep up. This article scratches the surface of what is currently available and stable and stuff we researched while building the solution at CircleHD. We are definitely keeping eyes on emerging technologies and research and overwhelmed ourselves with the available choices. We hope we did a good balance of cutting edge technologies while enabling our customer to do best without risking security. We would like to sincerely thank the researcher and developers for making the technology available to us. Looking forward to having a discussion with you if you have any questions or suggestions on this topic. Please feel free to reachout to me at [email protected]
