🔍 X-Ray Image Search System

A production-ready prototype for searching medical X-ray images using both text descriptions and visual similarity. Built with state-of-the-art deep learning models for medical imaging applications.

🚀 Features

✅ Core Functionality

• Text-Based Search: Find X-rays using natural language (e.g., "chest X-ray with pneumonia")
• Image-Based Search: Upload an X-ray to find visually similar images
• Dual Model Architecture:
  • CLIP (ViT-B/32) for text-to-image search
  • ResNet50 for image-to-image similarity
• GPU Acceleration: Automatic CUDA detection and usage
• Smart Caching: Embeddings cached for fast subsequent searches
• Category Filtering: Filter results by anatomical categories
• Similarity Thresholding: Control result quality with adjustable thresholds

🎁 Bonus Features

• Ranking scores displayed for all results
• Category filters (chest, spine, dental, etc.)
• Real-time similarity metrics
• Responsive grid layout
• Dataset statistics dashboard

📁 Project Structure

xray-search-system/
├── data_processor.py      # Dataset handling & embedding generation
├── search_engine.py       # Search logic (text & image)
├── app.py                 # Streamlit UI
├── requirements.txt       # Python dependencies
├── README.md             # This file
├── dataset/              # Dataset directory (you create this)
│   ├── images/           # X-ray images (.png)
│   ├── metadata.csv      # Auto-generated metadata
│   ├── resnet_embeddings.pkl   # Cached ResNet embeddings
│   └── clip_embeddings.pkl     # Cached CLIP embeddings
└── screenshots/          # Demo screenshots (optional)

🚀 Quick Start

1️⃣ Installation

# Clone or download this project
cd xray-search-system

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\\Scripts\\activate

# Install dependencies
pip install -r requirements.txt

2️⃣ Dataset Setup

Option A: NIH Chest X-ray Dataset (Recommended)

Visit: https://nihcc.app.box.com/v/ChestXray-NIHCC Download images_001.tar.gz (~7GB, contains ~4,600 images) Extract to dataset/images/

Option B: Custom Dataset

Create this structure:

dataset/
├── images/
│   ├── image1.png
│   ├── image2.png
│   └── ...
└── metadata.csv (optional)

3️⃣ Run the Application

streamlit run app.py

💻 Usage

Text Search

1. Select "Text Search" mode
2. Enter a query like:
   • "chest x-ray with pneumonia"
   • "normal chest radiograph"
   • "lung nodule"
3. Adjust filters (number of results, similarity threshold, category)
4. Click "Search"

Image Search

1. Select "Image Search" mode
2. Upload an X-ray image (PNG, JPG, JPEG)
3. Adjust filters
4. Click "Find Similar Images"

🧠 Technical Details

Architecture

Text Search Pipeline:

Text Query → CLIP Text Encoder → Text Embedding (512-dim)
                                        ↓
                              Cosine Similarity
                                        ↓
              Database CLIP Image Embeddings (N x 512)
                                        ↓
                              Top-K Results

Image Search Pipeline:

Query Image → ResNet50 → Feature Embedding (2048-dim)
                                ↓
                      Cosine Similarity
                                ↓
         Database ResNet Embeddings (N x 2048)
                                ↓
                        Top-K Results

🐛 Troubleshooting

Issue: "Dataset not found"

Solution: Ensure dataset/images/ directory exists with PNG/JPG files

Issue: "CUDA out of memory"

Solution: System will auto-fallback to CPU

Issue: "No module named 'clip'"

Solution: Install CLIP from GitHub:

pip install git+https://github.com/openai/CLIP.git

📝 License & Attribution

Dataset: NIH Chest X-ray Dataset

• License: Public Domain
• Source: NIH Clinical Center

Code: MIT License

🤝 Contributing

For improvements or bug fixes:

1. Test on subset (100 images) first
2. Verify GPU and CPU modes both work
3. Check embeddings cache correctly
4. Ensure UI remains responsive

📧 Support

For issues:

1. Check "Troubleshooting" section above
2. Verify dataset setup is correct
3. Ensure all dependencies installed
4. Check GPU drivers if using CUDA

🎉 Acknowledgments

• OpenAI for CLIP model
• NIH Clinical Center for chest X-ray dataset
• PyTorch and Streamlit teams

Built with ❤️ for medical image retrieval