redis migration

Signed-off-by: ale <ale@manalejandro.com>
Este commit está contenido en:
ale
2025-12-15 17:43:08 +01:00
padre da89037125
commit 4d9545d0ec
Se han modificado 18 ficheros con 1218 adiciones y 1065 borrados

Ver fichero

@@ -9,7 +9,7 @@ This guide will help you quickly set up and test the Hasher application.
Ensure you have:
- ✅ Node.js 18.x or higher (`node --version`)
- ✅ npm (`npm --version`)
-Elasticsearch running on `localhost:9200`
-Redis 7.x or higher running on `localhost:6379`
### 2. Installation
@@ -20,13 +20,16 @@ cd hasher
# Install dependencies
npm install
# Start Redis (if not running)
redis-server
# Start the development server
npm run dev
```
The application will be available at: **http://localhost:3000**
### 3. Verify Elasticsearch Connection
### 3. Verify Redis Connection
```bash
# Check health endpoint
@@ -37,7 +40,11 @@ Expected response:
```json
{
"status": "ok",
"elasticsearch": { ... }
"redis": {
"version": "7.2.4",
"connected": true,
"memoryUsed": "1.5M"
}
}
```
@@ -84,22 +91,19 @@ npm run index-file sample-wordlist.txt
**Expected Output**:
```
📚 Hasher Indexer
📚 Hasher Indexer - Redis Edition
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Elasticsearch: http://localhost:9200
Index: hasher
Redis: localhost:6379
File: sample-wordlist.txt
Batch size: 100
🔗 Connecting to Elasticsearch...
🔗 Connecting to Redis...
✅ Connected successfully
📖 Reading file...
✅ Found 20 words/phrases to process
⏳ Progress: 20/20 (100.0%) - Indexed: 20, Errors: 0
🔄 Refreshing index...
⏳ Progress: 20/20 (100.0%) - Indexed: 20, Skipped: 0, Errors: 0
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Indexing complete!
@@ -114,6 +118,16 @@ After running the bulk indexer, search for:
All should return their plaintext values.
### Test 6: Remove Duplicates
```bash
# Dry run to preview duplicates
npm run remove-duplicates -- --dry-run --field md5
# Execute removal
npm run remove-duplicates -- --execute --field md5
```
---
## 🔍 API Testing
@@ -185,13 +199,13 @@ fetch('/api/search', {
- [ ] Results display correctly
### Data Persistence
- [ ] New plaintext is saved to Elasticsearch
- [ ] New plaintext is saved to Redis
- [ ] Saved hashes can be found in subsequent searches
- [ ] Bulk indexing saves all entries
- [ ] Index is created automatically if missing
- [ ] Duplicate detection works correctly
### Error Handling
- [ ] Elasticsearch connection errors are handled
- [ ] Redis connection errors are handled
- [ ] Empty search queries are prevented
- [ ] Invalid input is handled gracefully
- [ ] Network errors show user-friendly messages
@@ -200,15 +214,20 @@ fetch('/api/search', {
## 🐛 Common Issues & Solutions
### Issue: Cannot connect to Elasticsearch
### Issue: Cannot connect to Redis
**Solution**:
```bash
# Check if Elasticsearch is running
curl http://localhost:9200
# Check if Redis is running
redis-cli ping
# Should respond: PONG
# If not accessible, update the environment variable
export ELASTICSEARCH_NODE=http://your-elasticsearch-host:9200
# If not running, start Redis
redis-server
# If using custom host/port, update environment variables
export REDIS_HOST=localhost
export REDIS_PORT=6379
npm run dev
```
@@ -242,33 +261,48 @@ npm run index-file -- "$(pwd)/sample-wordlist.txt"
---
## 📊 Verify Data in Elasticsearch
## 📊 Verify Data in Redis
### Check Index Stats
### Check Redis Connection
```bash
curl http://localhost:9200/hasher/_stats?pretty
redis-cli ping
```
### Count Documents
### Count Keys
```bash
curl http://localhost:9200/hasher/_count?pretty
redis-cli DBSIZE
```
### View Sample Documents
```bash
curl http://localhost:9200/hasher/_search?pretty&size=5
# List hash document keys
redis-cli --scan --pattern "hash:plaintext:*" | head -5
# Get a specific document
redis-cli GET "hash:plaintext:password"
```
### Check Statistics
```bash
redis-cli HGETALL hash:stats
```
### Search Specific Hash
```bash
curl http://localhost:9200/hasher/_search?pretty -H 'Content-Type: application/json' -d'
{
"query": {
"term": {
"md5": "5f4dcc3b5aa765d61d8327deb882cf99"
}
}
}'
# Find plaintext for an MD5 hash
redis-cli GET "hash:index:md5:5f4dcc3b5aa765d61d8327deb882cf99"
# Get the full document
redis-cli GET "hash:plaintext:password"
```
### Monitor Redis Activity
```bash
# Watch commands in real-time
redis-cli MONITOR
# Check memory usage
redis-cli INFO memory
```
---
@@ -310,9 +344,18 @@ Create `search.json`:
```
### Expected Performance
- Search latency: < 100ms
- Bulk indexing: 1000+ docs/sec
- Concurrent requests: 50+
- Search latency: < 5ms
- Bulk indexing: 5000-15000 docs/sec
- Concurrent requests: 100+
### Redis Performance Testing
```bash
# Benchmark Redis operations
redis-benchmark -t set,get -n 100000 -q
# Test with pipeline
redis-benchmark -t set,get -n 100000 -q -P 16
```
---
@@ -329,7 +372,13 @@ Create `search.json`:
- [ ] CORS configuration
- [ ] Rate limiting (if implemented)
- [ ] Error message information disclosure
- [ ] Elasticsearch authentication (if enabled)
- [ ] Redis authentication (if enabled)
### Redis Security Checklist
- [ ] Redis password configured (REDIS_PASSWORD)
- [ ] Redis not exposed to internet
- [ ] Firewall rules configured
- [ ] TLS/SSL enabled (if needed)
---
@@ -339,7 +388,8 @@ Before deploying to production:
- [ ] All tests passing
- [ ] Environment variables configured
- [ ] Elasticsearch secured and backed up
- [ ] Redis secured with password
- [ ] Redis persistence configured (RDB/AOF)
- [ ] SSL/TLS certificates installed
- [ ] Error logging configured
- [ ] Monitoring set up
@@ -347,6 +397,7 @@ Before deploying to production:
- [ ] Security review done
- [ ] Documentation reviewed
- [ ] Backup strategy in place
- [ ] Redis memory limits configured
---
@@ -357,7 +408,7 @@ Before deploying to production:
## Environment
- Node.js version:
- Elasticsearch version:
- Redis version:
- Browser(s) tested:
## Test Results
@@ -367,6 +418,7 @@ Before deploying to production:
- [ ] Hash search: PASS/FAIL
- [ ] Bulk indexing: PASS/FAIL
- [ ] API endpoints: PASS/FAIL
- [ ] Duplicate removal: PASS/FAIL
### Issues Found
1. [Description]
@@ -379,6 +431,7 @@ Before deploying to production:
- Average search time:
- Bulk index rate:
- Concurrent users tested:
- Redis memory usage:
## Conclusion
[Summary of testing]
@@ -394,7 +447,8 @@ After successful testing:
2. ✅ Fix any issues found
3. ✅ Perform load testing
4. ✅ Review security
5.Prepare for deployment
5.Configure Redis persistence
6. ✅ Prepare for deployment
See [DEPLOYMENT.md](DEPLOYMENT.md) for deployment instructions.