new redis migration

Signed-off-by: ale <ale@manalejandro.com>
2025-12-15 16:35:35 +01:00
commit 3ce64eeb8e
--- a/README.md
+++ b/README.md
@@ -1,20 +1,21 @@
 # Hasher 🔐

-A modern, high-performance hash search and generation tool powered by Elasticsearch and Next.js. Search for hash values to find their plaintext origins or generate hashes from any text input.
+A modern, high-performance hash search and generation tool powered by Redis and Next.js. Search for hash values to find their plaintext origins or generate hashes from any text input.

 ![Hasher Banner](https://img.shields.io/badge/Next.js-16.0-black?style=for-the-badge&logo=next.js)
-![Elasticsearch](https://img.shields.io/badge/Elasticsearch-8.x-005571?style=for-the-badge&logo=elasticsearch)
+![Redis](https://img.shields.io/badge/Redis-7.x-DC382D?style=for-the-badge&logo=redis)
 ![TypeScript](https://img.shields.io/badge/TypeScript-5.x-3178C6?style=for-the-badge&logo=typescript)

 ## ✨ Features

- 🔍 **Hash Lookup**: Search for MD5, SHA1, SHA256, SHA512, and Bcrypt hashes
+- 🔍 **Hash Lookup**: Search for MD5, SHA1, SHA256, and SHA512 hashes
 - 🔑 **Hash Generation**: Generate multiple hash types from plaintext
 - 💾 **Auto-Indexing**: Automatically stores searched plaintext and hashes
- 📊 **Elasticsearch Backend**: Scalable storage with 10 shards for performance
- 🚀 **Bulk Indexing**: Import wordlists via command-line script
+- 📊 **Redis Backend**: Ultra-fast in-memory storage with persistence
+- 🚀 **Bulk Indexing**: Import wordlists via command-line script with resume capability
 - 🎨 **Modern UI**: Beautiful, responsive interface with real-time feedback
 - 📋 **Copy to Clipboard**: One-click copying of any hash value
+- ⚡ **High Performance**: Lightning-fast searches with Redis indexing

 ## 🏗️ Architecture

@@ -32,8 +33,9 @@ A modern, high-performance hash search and generation tool powered by Elasticsea
       │
       ↓
 ┌─────────────┐
-│Elasticsearch│ ← Distributed storage
-│ 10 Shards   │   (localhost:9200)
+│    Redis    │ ← In-memory storage
+│  (Key-Value │   (localhost:6379)
+│   + Hashes) │
 └─────────────┘
 ```

@@ -42,7 +44,7 @@ A modern, high-performance hash search and generation tool powered by Elasticsea
 ### Prerequisites

 - Node.js 18.x or higher
- Elasticsearch 8.x running on `localhost:9200`
+- Redis 6.x or higher running on `localhost:6379`
 - npm or yarn

 ### Installation
@@ -58,20 +60,33 @@ A modern, high-performance hash search and generation tool powered by Elasticsea
   npm install
   ```

-3. **Configure Elasticsearch** (optional)
-   
-   By default, the app connects to `http://localhost:9200`. To change this:
+3. **Start Redis** (if not already running)
   
   ```bash
-   export ELASTICSEARCH_NODE=http://your-elasticsearch-host:9200
+   # Using Docker
+   docker run -d --name redis -p 6379:6379 redis:latest
+   
+   # Or using system package manager
+   sudo systemctl start redis
   ```

-4. **Run the development server**
+4. **Configure Redis** (optional)
+   
+   By default, the app connects to `localhost:6379`. To change this:
+   
+   ```bash
+   export REDIS_HOST=your-redis-host
+   export REDIS_PORT=6379
+   export REDIS_PASSWORD=your-password  # if authentication is enabled
+   export REDIS_DB=0                    # database number
+   ```
+
+5. **Run the development server**
   ```bash
   npm run dev
   ```

-5. **Open your browser**
+6. **Open your browser**
   
   Navigate to [http://localhost:3000](http://localhost:3000)

@@ -100,6 +115,12 @@ npm run index-file wordlist.txt
 # With custom batch size
 npm run index-file wordlist.txt -- --batch-size 500

+# Skip duplicate checking (faster)
+npm run index-file wordlist.txt -- --no-check
+
+# Resume interrupted indexing
+npm run index-file wordlist.txt -- --resume
+
 # Show help
 npm run index-file -- --help
 ```
@@ -114,10 +135,11 @@ qwerty

 **Script features**:
 - ✅ Bulk indexing with configurable batch size
- ✅ Progress indicator with percentage
+- ✅ Progress indicator and real-time stats
+- ✅ State persistence with resume capability
+- ✅ Optional duplicate checking
 - ✅ Error handling and reporting
 - ✅ Performance metrics (docs/sec)
- ✅ Automatic index refresh

 ## 🔌 API Reference

@@ -171,15 +193,17 @@ Search for a hash or generate hashes from plaintext.

 **GET** `/api/health`

-Check Elasticsearch connection and index status.
+Check Redis connection and index status.

 **Response**:
 ```json
 {
  "status": "ok",
-  "elasticsearch": {
-    "cluster": "elasticsearch",
-    "status": "green"
+  "redis": {
+    "connected": true,
+    "version": "7.0.15",
+    "usedMemory": 2097152,
+    "dbSize": 1542
  },
  "index": {
    "exists": true,
@@ -192,30 +216,33 @@ Check Elasticsearch connection and index status.
 }
 ```

-## 🗄️ Elasticsearch Index
+## 🗄️ Redis Data Structure

-### Index Configuration
+### Key Structure

- **Name**: `hasher`
- **Shards**: 10 (for horizontal scaling)
- **Replicas**: 1 (for redundancy)
+**Main Documents**: `hash:plaintext:{plaintext}`
+- Stores complete hash document as JSON string
+- Contains all hash algorithms and metadata

-### Mapping Schema
+**Hash Indexes**: `hash:index:{algorithm}:{hash}`
+- Reverse lookup from hash to plaintext
+- One key per algorithm (md5, sha1, sha256, sha512)
+- Value is the plaintext string

-```json
+**Statistics**: `hash:stats` (Redis Hash)
+- `count`: Total number of unique plaintexts
+- `size`: Approximate total size in bytes
+
+### Document Schema
+
+```typescript
 {
-  "plaintext": {
-    "type": "text",
-    "analyzer": "lowercase_analyzer",
-    "fields": {
-      "keyword": { "type": "keyword" }
-    }
-  },
-  "md5": { "type": "keyword" },
-  "sha1": { "type": "keyword" },
-  "sha256": { "type": "keyword" },
-  "sha512": { "type": "keyword" },
-  "created_at": { "type": "date" }
+  "plaintext": string,
+  "md5": string,
+  "sha1": string,
+  "sha256": string,
+  "sha512": string,
+  "created_at": string (ISO 8601)
 }
 ```

@@ -233,10 +260,11 @@ hasher/
 │   ├── page.tsx                # Main UI component
 │   └── globals.css             # Global styles
 ├── lib/
-│   ├── elasticsearch.ts        # ES client & index config
+│   ├── redis.ts                # Redis client & data layer
 │   └── hash.ts                 # Hash utilities
 ├── scripts/
-│   └── index-file.ts           # Bulk indexing script
+│   ├── index-file.ts           # Bulk indexing script
+│   └── remove-duplicates.ts    # Duplicate removal utility
 ├── package.json
 ├── tsconfig.json
 ├── next.config.ts
@@ -257,7 +285,10 @@ npm run start
 Create a `.env.local` file:

 ```env
-ELASTICSEARCH_NODE=http://localhost:9200
+REDIS_HOST=localhost
+REDIS_PORT=6379
+REDIS_PASSWORD=your-password
+REDIS_DB=0
 ```

 ### Linting
@@ -274,7 +305,6 @@ npm run lint
 | SHA1      | 40           | `^[a-f0-9]{40}$` |
 | SHA256    | 64           | `^[a-f0-9]{64}$` |
 | SHA512    | 128          | `^[a-f0-9]{128}$` |
-| Bcrypt    | 60           | `^\$2[abxy]\$` |

 ## 🚀 Performance

@@ -300,7 +330,7 @@ This project is open source and available under the [MIT License](LICENSE).
 ## 🙏 Acknowledgments

 - Built with [Next.js](https://nextjs.org/)
- Powered by [Elasticsearch](https://www.elastic.co/)
+- Powered by [Redis](https://redis.io/)
 - Icons by [Lucide](https://lucide.dev/)
 - Styled with [Tailwind CSS](https://tailwindcss.com/)