new redis migration

Signed-off-by: ale <ale@manalejandro.com>

REDIS_QUICKSTART.md

@@ -0,0 +1,222 @@
+# Redis Migration - Quick Reference
+
+## 🚀 Quick Start
+
+### 1. Install Redis
+```bash
+# Ubuntu/Debian
+sudo apt-get install redis-server
+
+# macOS
+brew install redis
+
+# Start Redis
+redis-server
+# or
+sudo systemctl start redis-server
+```
+
+### 2. Configure Environment (Optional)
+```bash
+# Create .env.local
+cat > .env.local << EOF
+REDIS_HOST=localhost
+REDIS_PORT=6379
+REDIS_PASSWORD=  # Leave empty if no password
+REDIS_DB=0
+EOF
+```
+
+### 3. Start Application
+```bash
+yarn dev
+```
+
+## 🔍 Testing the Migration
+
+### Test Health Endpoint
+```bash
+curl http://localhost:3000/api/health
+```
+
+Expected response:
+```json
+{
+  "status": "ok",
+  "redis": {
+    "version": "7.x",
+    "memory": "1.5M",
+    "dbSize": 0
+  },
+  "stats": {
+    "count": 0,
+    "size": 0
+  }
+}
+```
+
+### Test Search API
+```bash
+# Generate hashes
+curl -X POST http://localhost:3000/api/search \
+  -H "Content-Type: application/json" \
+  -d '{"query":"password"}'
+
+# Search for hash
+curl -X POST http://localhost:3000/api/search \
+  -H "Content-Type: application/json" \
+  -d '{"query":"5f4dcc3b5aa765d61d8327deb882cf99"}'
+```
+
+## 📊 Redis Commands
+
+### Check Connection
+```bash
+redis-cli ping
+# Should return: PONG
+```
+
+### View Data
+```bash
+# Count all keys
+redis-cli DBSIZE
+
+# List all documents
+redis-cli KEYS "hash:plaintext:*"
+
+# Get a specific document
+redis-cli GET "hash:plaintext:password"
+
+# Get statistics
+redis-cli HGETALL hash:stats
+
+# Search by hash
+redis-cli GET "hash:index:md5:5f4dcc3b5aa765d61d8327deb882cf99"
+```
+
+### Clear Data (if needed)
+```bash
+# WARNING: Deletes ALL data in current database
+redis-cli FLUSHDB
+```
+
+## 🔄 Bulk Indexing
+
+### Basic Usage
+```bash
+yarn index-file sample-wordlist.txt
+```
+
+### Advanced Options
+```bash
+# Custom batch size
+yarn index-file wordlist.txt -- --batch-size 500
+
+# Skip duplicate checking (faster)
+yarn index-file wordlist.txt -- --no-check
+
+# Resume from previous state
+yarn index-file wordlist.txt -- --resume
+
+# Custom state file
+yarn index-file wordlist.txt -- --state-file .my-state.json
+```
+
+## 🐛 Troubleshooting
+
+### Cannot connect to Redis
+```bash
+# Check if Redis is running
+redis-cli ping
+
+# Check Redis status
+sudo systemctl status redis-server
+
+# View Redis logs
+sudo journalctl -u redis-server -f
+```
+
+### Application shows Redis errors
+1. Verify Redis is running: `redis-cli ping`
+2. Check environment variables in `.env.local`
+3. Check firewall rules if Redis is on another machine
+4. Verify Redis password if authentication is enabled
+
+### Clear stale state files
+```bash
+rm -f .indexer-state-*.json
+```
+
+## 📈 Monitoring
+
+### Redis Memory Usage
+```bash
+redis-cli INFO memory
+```
+
+### Redis Stats
+```bash
+redis-cli INFO stats
+```
+
+### Application Stats
+```bash
+curl http://localhost:3000/api/health | jq .
+```
+
+## 🔒 Security (Production)
+
+### Enable Redis Authentication
+```bash
+# Edit redis.conf
+sudo nano /etc/redis/redis.conf
+
+# Add/uncomment:
+requirepass your-strong-password
+
+# Restart Redis
+sudo systemctl restart redis-server
+```
+
+### Update .env.local
+```env
+REDIS_PASSWORD=your-strong-password
+```
+
+## 📚 Key Differences from Elasticsearch
+
+| Feature | Elasticsearch | Redis |
+|---------|--------------|-------|
+| Data Model | Document-based | Key-value |
+| Search Complexity | O(log n) | O(1) |
+| Setup | Complex cluster | Single instance |
+| Memory | Higher | Lower |
+| Latency | ~50ms | <10ms |
+| Scaling | Shards/Replicas | Cluster/Sentinel |
+
+## ✅ Verification Checklist
+
+- [ ] Redis is installed and running
+- [ ] Application builds without errors (`yarn build`)
+- [ ] Health endpoint returns OK status
+- [ ] Can generate hashes from plaintext
+- [ ] Can search for generated hashes
+- [ ] Statistics display on homepage
+- [ ] Bulk indexing script works
+- [ ] Data persists after application restart
+
+## 📞 Support
+
+- Redis Documentation: https://redis.io/docs/
+- ioredis Documentation: https://github.com/redis/ioredis
+- Project README: [README.md](README.md)
+
+---
+
+**Quick Test Command:**
+```bash
+# One-liner to test everything
+redis-cli ping && yarn build && curl -s http://localhost:3000/api/health | jq .status
+```
+
+If all commands succeed, the migration is working correctly! ✅