Recipe for searching similar videos

CHANGELOG.md

@@ -2,6 +2,158 @@
 
 
 
+## v0.8.0 (2024-09-12)
+
+### Chore
+
+* chore: updating vid vec rep resenet packages ([`9f212c2`](https://github.com/tattle-made/feluda/commit/9f212c2d21cb7ad33a41330f203dab3d41f04d69))
+
+* chore: updating audio vec embedding pacakges ([`a41f996`](https://github.com/tattle-made/feluda/commit/a41f996f40af1b5f9c70786ee376c6a5f2348b32))
+
+* chore: updating audio vec embedding pacakges ([`6e50e06`](https://github.com/tattle-made/feluda/commit/6e50e0683702cc9549773ab6bffa742a43105835))
+
+* chore: fix conflict in updating requirements packages ([`acf1a56`](https://github.com/tattle-made/feluda/commit/acf1a564d872c26575f0d23abe5a7a7b56df8c94))
+
+* chore: updating requirements packages ([`5390073`](https://github.com/tattle-made/feluda/commit/5390073e3a848e0984ef6049d1500991d27583aa))
+
+* chore: fix ruff lint warnings ([`5f73898`](https://github.com/tattle-made/feluda/commit/5f738980d18eb9572d61b26aaf7a6f23ac016aaa))
+
+* chore: add error handling to clustering and reduction operations ([`ba01287`](https://github.com/tattle-made/feluda/commit/ba01287cd7cbb175b516874fcc489e5b0261ebc7))
+
+* chore: hardcode perplexity value for t-SNE ([`60315b5`](https://github.com/tattle-made/feluda/commit/60315b5fe96e445831c8c84641b8eb95f2d0e701))
+
+* chore: change operator name to be more verbose
+
+Signed-off-by: Snehil Shah <[email protected]> ([`45bef72`](https://github.com/tattle-made/feluda/commit/45bef72d74617bff5915429e3802fc88b9ed595a))
+
+* chore: remove obsolete `print` statement
+
+Signed-off-by: Snehil Shah <[email protected]> ([`92d02e2`](https://github.com/tattle-made/feluda/commit/92d02e219ecb0844de261b35fa2801adca139f04))
+
+* chore: remove the file after the operation has ended
+
+Signed-off-by: Snehil Shah <[email protected]> ([`4a384f5`](https://github.com/tattle-made/feluda/commit/4a384f57b6f4daa41861162898415fa981dc73e0))
+
+* chore: removing docs gatsby website ([`03f5ac0`](https://github.com/tattle-made/feluda/commit/03f5ac078952851fdf17954484f34eccd066ca72))
+
+### Ci
+
+* ci: fix trivy github action ([`a5d4df8`](https://github.com/tattle-made/feluda/commit/a5d4df8d2e5f0b9b1654134557509df0a45c4583))
+
+* ci: update dependabot yml file ([`8a6b007`](https://github.com/tattle-made/feluda/commit/8a6b007235c957e76e369bc3d2346a45f50aa337))
+
+### Feature
+
+* feat: add config
+
+Signed-off-by: Snehil Shah <[email protected]> ([`73731ec`](https://github.com/tattle-made/feluda/commit/73731eceaad6e33c78052654012b1927e3682e5d))
+
+* feat: add dimensionality reduction operator ([`2caf4a4`](https://github.com/tattle-made/feluda/commit/2caf4a4398383ae95731ce167c3d7d5391a49284))
+
+* feat: improve exception handling and logic
+
+Co-authored-by: Chaithanya512 <[email protected]>
+Signed-off-by: Snehil Shah <[email protected]> ([`5756a59`](https://github.com/tattle-made/feluda/commit/5756a59d2a8f2e7395ff566f07213fbb6bb75bcd))
+
+* feat: add `cluster_embeddings` operator
+
+Signed-off-by: Snehil Shah <[email protected]> ([`4e08e35`](https://github.com/tattle-made/feluda/commit/4e08e35309c415b1bae0f65732d9df635da342a4))
+
+* feat: add audio embedding operator using CLAP model ([`233cfd4`](https://github.com/tattle-made/feluda/commit/233cfd4065f2e43d6cf533ade3002270f02f5eb2))
+
+* feat: add `classify_video` operator
+
+Signed-off-by: Snehil Shah <[email protected]> ([`2001dab`](https://github.com/tattle-made/feluda/commit/2001dab2f38b0892b31cb269d85a118d16c0210f))
+
+* feat: add `vid_vec_rep_clip` operator
+
+Signed-off-by: Snehil Shah <[email protected]> ([`8874ce5`](https://github.com/tattle-made/feluda/commit/8874ce58d5ee77df7844c6b49ae5180736fee00e))
+
+### Fix
+
+* fix: es_vec test ([`08b3085`](https://github.com/tattle-made/feluda/commit/08b308587029507136cdc4fe68719a1357b84917))
+
+* fix: enable worker support for dimension reductionn ([`89df059`](https://github.com/tattle-made/feluda/commit/89df059b251cd2ed7cddb46e39d60fe2d8b45e48))
+
+* fix: update logic as per clustering spec
+
+Signed-off-by: Snehil Shah <[email protected]> ([`0bea260`](https://github.com/tattle-made/feluda/commit/0bea260224c49cb6c2e0f0a47832789ff8b6591b))
+
+* fix: correct indentation logic ([`d208d17`](https://github.com/tattle-made/feluda/commit/d208d17ba7f2c2914ca3666f1b25295d50524eab))
+
+* fix: correct naming and add dimension reduction requirements ([`0334a4b`](https://github.com/tattle-made/feluda/commit/0334a4b20a5555c8278ba8fc99c55327c0c27b8b))
+
+### Refactor
+
+* refactor: migrate CLAP operator to Hugging Face Transformers ([`9e5195a`](https://github.com/tattle-made/feluda/commit/9e5195aa8ddf2c86d54333ee6e9c01a5576b32f4))
+
+* refactor: align dimension reduction with feluda interface ([`0acfdb7`](https://github.com/tattle-made/feluda/commit/0acfdb74befae505f20ebf112bb3a58b087ca980))
+
+* refactor: change modality to `video`
+
+Signed-off-by: Snehil Shah <[email protected]> ([`a1506ba`](https://github.com/tattle-made/feluda/commit/a1506ba879444458f09f286d34baec94b38c3744))
+
+### Style
+
+* style: prepend newline for clearer output in logs
+
+Signed-off-by: Snehil Shah <[email protected]> ([`bacafdd`](https://github.com/tattle-made/feluda/commit/bacafdd95142078626e1b996258859f13bbf8d97))
+
+### Test
+
+* test: add payload writer for the worker
+
+Signed-off-by: Snehil Shah <[email protected]> ([`57803e0`](https://github.com/tattle-made/feluda/commit/57803e07451e8263544cc8d239f2e95a99c303e0))
+
+### Unknown
+
+* Merge pull request #381 from tattle-made/development
+
+merge dev to main ([`48bfc87`](https://github.com/tattle-made/feluda/commit/48bfc87b813f5a34ea59c7bc9e7d7316000ce1ee))
+
+* Merge pull request #380 from aatmanvaidya/update-fel
+
+Update Packages and Fix minor Issues ([`cf64f4d`](https://github.com/tattle-made/feluda/commit/cf64f4d5ead811883d5012656222a3459cae4f09))
+
+* Merge pull request #379 from Snehil-Shah/worker
+
+worker for clustering media ([`4abae4d`](https://github.com/tattle-made/feluda/commit/4abae4dff4cd0d50c20eeb152c5a7220d2385923))
+
+* nit: add logs
+
+Signed-off-by: Snehil Shah <[email protected]> ([`1ba6de1`](https://github.com/tattle-made/feluda/commit/1ba6de164563dd086ab4797586a44c62770d8dd1))
+
+* draft: implement initial clustering worker ([`7129ac3`](https://github.com/tattle-made/feluda/commit/7129ac3ee76993bd97eb10ac2df6ff703da86dcb))
+
+* init: add Dockerfile for worker
+
+Signed-off-by: Snehil Shah <[email protected]> ([`777994b`](https://github.com/tattle-made/feluda/commit/777994b597b1554a7bbfb77dd3eed8967a3757a5))
+
+* Merge pull request #375 from Snehil-Shah/clustering
+
+[81] - add operator to cluster embeddings ([`1da1c27`](https://github.com/tattle-made/feluda/commit/1da1c275fe840327c88623ef8d6b968338ebfc89))
+
+* Merge pull request #376 from Chaithanya512/dim-reduction-operator
+
+feat: add dimensionality reduction operator ([`cb76d85`](https://github.com/tattle-made/feluda/commit/cb76d85c0cb1e32295b0cf1e39d765bf4b1efb0b))
+
+* Merge pull request #372 from Chaithanya512/audio_vec_emb_clap
+
+feat: audio embedding operator using CLAP model ([`faa9727`](https://github.com/tattle-made/feluda/commit/faa9727eec8c4639ea61615c69c006384dbb5665))
+
+* Merge pull request #370 from Snehil-Shah/classify-video
+
+[81] - add operator to classify videos using a zero-shot approach ([`91f604b`](https://github.com/tattle-made/feluda/commit/91f604b5e04e109645c0e55f1c691ddd7d9fb681))
+
+* Merge pull request #369 from Snehil-Shah/vid_vec_rep_clip
+
+[81] - add operator to encode videos into vector embeddings using `CLIP-ViT-B-32` ([`c5f09de`](https://github.com/tattle-made/feluda/commit/c5f09de7dba47eed1ef8efe4eae1cf5d7c16def5))
+
+* Merge pull request #367 from aatmanvaidya/remove-docs
+
+chore: removing docs gatsby website ([`2b9275a`](https://github.com/tattle-made/feluda/commit/2b9275a134677a1338dfbd5f7c9fcd9922c7b447))
+
+
 ## v0.7.2 (2024-05-28)
 
 ### Fix

docker-compose.yml

@@ -57,6 +57,7 @@ services:
     ports:
       - 7000:7000
       - 5678:5678
+      - 8888:8888
     command: tail -f /dev/null
     depends_on:
       store:
@@ -105,3 +106,4 @@ volumes:
 #  default:
 #    external: true
 #    name: kosh-v2_tattle
+

src/Dockerfile

@@ -38,7 +38,9 @@ RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-reco
 COPY --chown=python:python requirements.txt /usr/app/requirements.txt
 RUN pip install --no-cache-dir --require-hashes --no-deps -r requirements.txt
 COPY --chown=python:python . /usr/app
+
 EXPOSE 7000
+EXPOSE 8888
 
 # RUN apt-get update \
 #     && apt-get -y upgrade \

src/core/feluda.py

@@ -20,6 +20,7 @@ def __init__(self, configPath):
             from core.operators import Operator
 
             self.operators = Operator(self.config.operators)
+
         if self.config.store:
             from core import store
 

src/core/operators/vid_vec_rep_clip_requirements.in

@@ -1,3 +1,4 @@
 torch==2.4.0
 pillow==10.4.0
-transformers==4.44.0
+transformers==4.44.0
+torchvision==0.19.0

src/core/operators/vid_vec_rep_clip_requirements.txt

@@ -1,5 +1,5 @@
 #
-# This file is autogenerated by pip-compile with Python 3.11
+# This file is autogenerated by pip-compile with Python 3.10
 # by the following command:
 #
 #    pip-compile --allow-unsafe --generate-hashes vid_vec_rep_clip_requirements.in
@@ -244,7 +244,9 @@ numpy==2.0.1 \
     --hash=sha256:ec87f5f8aca726117a1c9b7083e7656a9d0d606eec7299cc067bb83d26f16e0c \
     --hash=sha256:f1659887361a7151f89e79b276ed8dff3d75877df906328f14d8bb40bb4f5101 \
     --hash=sha256:f9cf5ea551aec449206954b075db819f52adc1638d46a6738253a712d553c7b4
-    # via transformers
+    # via
+    #   torchvision
+    #   transformers
 nvidia-cublas-cu12==12.1.3.1 \
     --hash=sha256:2b964d60e8cf11b5e1073d179d85fa340c120e99b3067558f3cf98dd69d02906 \
     --hash=sha256:ee53ccca76a6fc08fb9701aa95b6ceb242cdaab118c3bb152af4e579af792728
@@ -388,7 +390,9 @@ pillow==10.4.0 \
     --hash=sha256:f7baece4ce06bade126fb84b8af1c33439a76d8a6fd818970215e0560ca28c27 \
     --hash=sha256:ff25afb18123cea58a591ea0244b92eb1e61a1fd497bf6d6384f09bc3262ec3e \
     --hash=sha256:ff337c552345e95702c5fde3158acb0625111017d0e5f24bf3acdb9cc16b90d1
-    # via -r vid_vec_rep_clip_requirements.in
+    # via
+    #   -r vid_vec_rep_clip_requirements.in
+    #   torchvision
 pyyaml==6.0.2 \
     --hash=sha256:01179a4a8559ab5de078078f37e5c1a30d76bb88519906844fd7bdea1b7729ff \
     --hash=sha256:0833f8694549e586547b576dcfaba4a6b55b9e96098b36cdc7ebefe667dfed48 \
@@ -772,6 +776,35 @@ torch==2.4.0 \
     --hash=sha256:e743adadd8c8152bb8373543964551a7cb7cc20ba898dc8f9c0cdbe47c283de0 \
     --hash=sha256:ed765d232d23566052ba83632ec73a4fccde00b4c94ad45d63b471b09d63b7a7 \
     --hash=sha256:f169b4ea6dc93b3a33319611fcc47dc1406e4dd539844dcbd2dec4c1b96e166d
+    # via
+    #   -r vid_vec_rep_clip_requirements.in
+    #   torchvision
+torchvision==0.19.0 \
+    --hash=sha256:02f1dd5cfc897957535b41b0258ec452d30de044e20c2de2c75869f7708e7656 \
+    --hash=sha256:079a696e0b2cb52e4be30afa8e9b3d7d280f02a2b5ffedd7e821fa1efd1a5a8d \
+    --hash=sha256:106842b1e475b14d9a04ee0d6f5477d43100e3bb78e9d31e37422384d0d84179 \
+    --hash=sha256:13aee7a46e049c8c1e7d35a0394b0587a7e62ff3d1a822cd2bbbacb675ac4a09 \
+    --hash=sha256:2acc436d043d4f81b3bc6929cbfa4ef1cdae4d8a0b04ec72ec30a497e9a38179 \
+    --hash=sha256:4d54b5e19b7ebebca7d0b08497b4c6335264cad04c94c05fa35988d9e9eed0c4 \
+    --hash=sha256:4e6aa4fa3f0bc3599fa071c149e651a3e6bdd67c9161794478f9f91471c406a2 \
+    --hash=sha256:5f9a598dcf82bdfc8e4436ce74763b3877dabec3b33f94613b94ede13e3e4dee \
+    --hash=sha256:6b1bce2e4c003d890a18f14ff289528707d918e38539ff890ef02aa31dae1b56 \
+    --hash=sha256:6ed066aae5c50465d7c4761357aefe5dbd2eb7075a33ab8c14b352fc2353ad4c \
+    --hash=sha256:854e967a16a9409e941b5bbe5aa357b23f7158bccb9de35ae20fd4945f05ecd1 \
+    --hash=sha256:a6ba7756f75c80212e51d3576f85ea204589e0c16efdb9b835dd677bc8929a67 \
+    --hash=sha256:aaa338ff3a55a8c0f94e0e64eff6fe2af1fc933a95fd43812760e72ea66e986b \
+    --hash=sha256:ac5525d5cc09e425b5cf5752ecf66eefbbbd8c8cd945198ce35eb01a694e6069 \
+    --hash=sha256:b5f70f5a8bd9c8b00a076bf466b39b5cd679ef62587c47cc048adb04d9c5f155 \
+    --hash=sha256:be0f27a28b8e9f2ae98a31af34a4bdd2a5bf154d92bd73a5797c8d2156fb3ab6 \
+    --hash=sha256:c09ef8ed184fa877f6251b620226e74f682b8f1d6b341456428d4955b8d9c670 \
+    --hash=sha256:d467d434005fd05a227a2ba7af4c591bb67e6d4a97bbd06eda8da83f43e9fd07 \
+    --hash=sha256:d9afb8a3c3ce99a161a64c2a3b91cb545632a72118053cbfb84e87a02a8dcd02 \
+    --hash=sha256:dbf3aa71a3899244fc884303ed3c4604a160824fefac77e82317a5463efc1d9b \
+    --hash=sha256:dd1279571d4b68d5a53d9b7a35aedf91c4cb1e0b08099f6a1effa7b25b8c95e7 \
+    --hash=sha256:ec1281c10402234d470bfd4d53663d81f4364f293b2f8fe24d4a7a1adc78c90c \
+    --hash=sha256:ec4162dc71d9db7f0b51d0f92491929c1419605ff436e1305e50de13504a1c30 \
+    --hash=sha256:ec874ef85dcb24c69e600f6e276af892c80cde3ffdaeb7275efda463242bc2a8 \
+    --hash=sha256:f77ac31f7337d0f6f4b58e65582c6c93b9d9eeec7dfd7478896b5cdc19a2d60d
     # via -r vid_vec_rep_clip_requirements.in
 tqdm==4.66.5 \
     --hash=sha256:90279a3770753eafc9194a0364852159802111925aa30eb3f9d85b0e805ac7cd \

src/notebooks/config.yml

@@ -0,0 +1,17 @@
+store:
+  entities:
+    - label: "Data Store"
+      type: "es_vec"
+      parameters:
+        host_name: "es"
+        image_index_name: "image"
+        text_index_name: "text"
+        video_index_name: "video"
+        audio_index_name: "audio"
+
+operators:
+  label: "Operators"
+  parameters:
+    - name: "Video Vector Representation"
+      type: "vid_vec_rep_clip"
+      parameters: { index_name: "video" }