CLIP-based Image Search

-- 2024 May Personal Project

Introduction

This personal project develops a straightforward image search application using CLIP, which matches images in a database to user-provided descriptions. The application is deployed through a simple Next.js-based website interface.

demo.mp4

Setup Instructions

Preparation

1. Download Data: Acquire the dataset from the open-source Flickr 8k Dataset available at Flickr 8k Dataset.
2. Rename Database: Change the folder name from "archive" to "data" and copy it into the ./backend and /frontend/public directories.
3. Folder Structure: Within ./backend, create the directory structure index/image. Ensure that image is an empty folder at ./backend/index.
4. Install Necessary Tools: Ensure npm and pipenv are installed on your local machine.

Installation

1. Python Environment: Navigate to ./backend and initiate a Python virtual environment by running pipenv shell.
2. Install Dependencies: Execute pipenv install to install all necessary Python dependencies.
3. Data Conversion: Run the setup.py Python script to convert the dataset into .pt files. Note that this process might take over 30 minutes.

Execution

• Frontend: Proceed to ./frontend and execute npm run dev to start the frontend.
• Backend: In ./backend, run python3 main.py to launch the backend.