netpcap/README.md

# Network Packet Capture & Elasticsearch Indexer

A Node.js-based network packet capture tool that captures packets from network interfaces and indexes them into Elasticsearch for analysis and monitoring.

## Features

- 🔍 **Multi-interface capture**: Capture from one or multiple network interfaces simultaneously
- 🎯 **Flexible filtering**: Filter by protocol (TCP/UDP/ICMP), ports, and port ranges
- 🔒 **Promiscuous mode support**: Optionally capture all packets on the network segment
- 📊 **Elasticsearch integration**: Automatic indexing with optimized mapping
- <20> **Failover cache**: In-memory cache for packets when Elasticsearch is unavailable- 🌍 **GeoIP enrichment**: Automatic geolocation data for remote IP addresses- <20>📝 **Content extraction**: Captures and indexes readable (ASCII) packet content
- 🚀 **Smart content handling**: Automatically skips large binary content while preserving packet metadata
- 📈 **Real-time statistics**: Track capture performance and statistics
- ⚙️ **Highly configurable**: Environment variables and config file support

## Prerequisites

- Node.js >= 14.0.0
- Elasticsearch 7.x or 8.x
- Root/Administrator privileges (required for packet capture)
- Linux: libpcap-dev (`apt-get install libpcap-dev`)
- macOS: XCode Command Line Tools

### Installing System Dependencies

**Ubuntu/Debian:**
```bash
sudo apt-get update
sudo apt-get install libpcap-dev build-essential
```

**CentOS/RHEL:**
```bash
sudo yum install libpcap-devel gcc-c++ make
```

**macOS:**
```bash
xcode-select --install
```

## Installation

1. Clone or navigate to the project directory:
```bash
cd /path/to/netpcap
```

2. Install Node.js dependencies:
```bash
npm install
```

**Note**: The `geoip-lite` package will download a GeoIP database on first install. This may take a few moments.

3. Copy the example environment file and configure:
```bash
cp .env.example .env
# Edit .env with your configuration
```

## Configuration

Configuration can be done via environment variables or by editing the `config.js` file directly.

### Elasticsearch Configuration

```bash
ES_NODE=http://localhost:9200
ES_USERNAME=elastic
ES_PASSWORD=your_password
ES_INDEX=network-packets
```

### Capture Settings

**Interfaces:**
```bash
# Capture from specific interfaces
CAPTURE_INTERFACES=eth0,wlan0

# Leave empty to capture from all available interfaces
CAPTURE_INTERFACES=
```

**Promiscuous Mode:**
```bash
# Enable to capture all packets on the network segment
PROMISCUOUS_MODE=true
```

### Filtering

**Protocol Filtering:**
```bash
# Only capture specific protocols
FILTER_PROTOCOLS=tcp,udp

# Capture all protocols (leave empty)
FILTER_PROTOCOLS=
```

**Port Filtering:**
```bash
# Exclude specific ports (e.g., SSH, HTTP, HTTPS)
EXCLUDE_PORTS=22,80,443

# Exclude port ranges
EXCLUDE_PORT_RANGES=[[8000,9000],[3000,3100]]

# Only capture specific ports (takes precedence)
INCLUDE_PORTS=3306,5432
```

**Custom BPF Filter:**
```bash
# Use custom Berkeley Packet Filter syntax
CAPTURE_FILTER="tcp and not port 22"
```

### Content Indexing

```bash
# Maximum content size to index (1MB default)
MAX_CONTENT_SIZE=1048576

# Enable/disable content indexing
INDEX_READABLE_CONTENT=true
```

### Cache System (Elasticsearch Failover)

The application includes an in-memory cache system to handle Elasticsearch outages:

```bash
# Maximum documents to cache in memory (default: 10000)
CACHE_MAX_SIZE=10000

# ES availability check interval in milliseconds (default: 5000)
CACHE_CHECK_INTERVAL=5000
```

**How it works:**
- When Elasticsearch is unavailable, packets are stored in memory cache
- The system periodically checks ES availability (every 5 seconds by default)
- When ES comes back online, cached documents are automatically flushed
- If cache reaches maximum size, oldest documents are removed (FIFO)
- On graceful shutdown (SIGINT/SIGTERM), the system attempts to flush all cached documents

## GeoIP Enrichment

The application automatically enriches captured packets with geolocation data for public IP addresses:

**Features:**
- Automatically detects the server's public IP address and its geolocation
- Identifies private/local IP addresses (RFC 1918, link-local, etc.)
- Enriches all public IP addresses (including the server's own IP)
- Uses local GeoIP database (no external API calls during capture)
- Adds geolocation data for both source and destination IPs when they are public

**GeoIP Data Included:**
- Country code and name
- Region/state
- City
- Timezone
- Geographic coordinates (latitude/longitude)
- Geo-point data for Elasticsearch geo queries

**How it works:**
1. On startup, the application detects its public IP using an external service (ipify.org)
2. The public IP's geolocation is fetched and cached for the session
3. For each packet:
   - Source IP: If public (including server's IP), GeoIP data is added to `geoip_src`
   - Destination IP: If public (including server's IP), GeoIP data is added to `geoip_dst`
   - Private IPs, loopback addresses are excluded from GeoIP enrichment
4. The server's own IP is included in geolocation to provide complete visibility of traffic origin and destination

**Traffic visibility:**
- **Outbound traffic**: `geoip_src` = server location, `geoip_dst` = remote destination location
- **Inbound traffic**: `geoip_src` = remote source location, `geoip_dst` = server location
- **External to external**: Both IPs geolocated (if packet is captured in transit)

**Example GeoIP document structure:**
```json
{
  "geoip_src": {
    "country": "US",
    "region": "CA",
    "city": "San Francisco",
    "timezone": "America/Los_Angeles",
    "location": {
      "lat": 37.7749,
      "lon": -122.4194
    }
  },
  "geoip_dst": {
    "country": "DE",
    "city": "Frankfurt",
    ...
  }
}
```

## Usage

### Basic Usage

Run with default configuration:
```bash
sudo npm start
```

Or directly:
```bash
sudo node index.js
```

### Capture from Specific Interface

```bash
sudo CAPTURE_INTERFACES=eth0 node index.js
```

### Capture Only HTTP/HTTPS Traffic

```bash
sudo INCLUDE_PORTS=80,443 FILTER_PROTOCOLS=tcp node index.js
```

### Exclude SSH and High Ports

```bash
sudo EXCLUDE_PORTS=22 EXCLUDE_PORT_RANGES=[[8000,65535]] node index.js
```

### Enable Promiscuous Mode

```bash
sudo PROMISCUOUS_MODE=true node index.js
```

### Debug Mode

```bash
sudo LOG_LEVEL=debug node index.js
```

## Elasticsearch Index Structure

The tool creates an index with the following document structure:

```json
{
  "@timestamp": "2026-02-11T10:30:00.000Z",
  "interface": {
    "name": "eth0",
    "ip": "192.168.1.100",
    "mac": "aa:bb:cc:dd:ee:ff"
  },
  "ethernet": {
    "src": "aa:bb:cc:dd:ee:ff",
    "dst": "11:22:33:44:55:66",
    "type": 2048
  },
  "ip": {
    "version": 4,
    "src": "192.168.1.100",
    "dst": "8.8.8.8",
    "protocol": 6,
    "ttl": 64,
    "length": 60
  },
  "tcp": {
    "src_port": 54321,
    "dst_port": 443,
    "flags": {
      "syn": true,
      "ack": false,
      "fin": false,
      "rst": false,
      "psh": false
    },
    "seq": 123456789,
    "ack_seq": 0,
    "window": 65535
  },
  "geoip_dst": {
    "country": "US",
    "country_name": "US",
    "region": "CA",
    "city": "Mountain View",
    "timezone": "America/Los_Angeles",
    "location": {
      "lat": 37.4056,
      "lon": -122.0775
    },
    "coordinates": [-122.0775, 37.4056]
  },
  "content": "GET / HTTP/1.1\r\nHost: example.com\r\n",
  "content_length": 1024,
  "content_type": "binary"
}
```

## Querying Captured Data

### Example Elasticsearch Queries

**Find all packets from a specific IP:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "term": {
      "ip.src": "192.168.1.100"
    }
  }
}
'
```

**Find all SYN packets (connection attempts):**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "term": { "tcp.flags.syn": true } },
        { "term": { "tcp.flags.ack": false } }
      ]
    }
  }
}
'
```

**Find packets with readable content:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "exists": {
      "field": "content"
    }
  }
}
'
```

**Find packets from a specific country:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "term": {
      "geoip_src.country": "US"
    }
  }
}
'
```

**Find packets to a specific country:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "term": {
      "geoip_dst.country": "CN"
    }
  }
}
'
```

**Find all outbound international traffic:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "exists": { "field": "geoip_src" } },
        { "exists": { "field": "geoip_dst" } }
      ],
      "must_not": [
        { "term": { "geoip_src.country": { "value": "geoip_dst.country" } } }
      ]
    }
  }
}
'
```

**Find packets to/from a specific geographic area:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "geo_distance": {
      "distance": "100km",
      "geoip_dst.location": {
        "lat": 37.7749,
        "lon": -122.4194
      }
    }
  }
}
'
```

**Aggregate packets by destination country:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "aggs": {
    "countries": {
      "terms": {
        "field": "geoip_dst.country",
        "size": 20
      }
    }
  }
}
'
```

**Find traffic between two specific countries:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "term": { "geoip_src.country": "US" } },
        { "term": { "geoip_dst.country": "GB" } }
      ]
    }
  }
}
'
```

**Map visualization - Get traffic with coordinates:**
```bash
curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "exists": { "field": "geoip_src.location" } },
        { "exists": { "field": "geoip_dst.location" } }
      ]
    }
  },
  "_source": ["ip.src", "ip.dst", "geoip_src.location", "geoip_dst.location", "geoip_src.city", "geoip_dst.city"]
}
'
```

## Performance Considerations

- **Promiscuous mode** can generate high packet volumes on busy networks
- **Content indexing** increases storage requirements significantly
- Use **port filters** to reduce captured packet volume
- Adjust `MAX_CONTENT_SIZE` based on your storage capacity
- Monitor Elasticsearch cluster health when capturing high-volume traffic
- **Cache system** protects against data loss during ES outages but consumes memory
- Adjust `CACHE_MAX_SIZE` based on available RAM (each packet ~1-5KB in memory)

## Troubleshooting

### Permission Denied Errors

Packet capture requires root privileges:
```bash
sudo node index.js
```

### Interface Not Found

List available interfaces:
```bash
ip link show  # Linux
ifconfig      # macOS/Unix
```

### Elasticsearch Connection Failed

Verify Elasticsearch is running:
```bash
curl -X GET "localhost:9200"
```

### No Packets Being Captured

1. Check if the interface is up and receiving traffic
2. Verify filter configuration isn't too restrictive
3. Try running without filters first
4. Check system firewall settings

## Security Considerations

⚠️ **Important Security Notes:**

- This tool captures network traffic and may contain sensitive information
- GeoIP data reveals geographic locations of communication endpoints
- Store Elasticsearch credentials securely
- Restrict access to the Elasticsearch index
- Be aware of privacy and legal implications when capturing network traffic
- Use encryption for Elasticsearch connections in production
- The public IP detection makes an external HTTPS call to ipify.org on startup
- Comply with applicable laws and regulations regarding network monitoring and data retention

## License

MIT

## Author

ale

## Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.