netpcap/README.md

Network Packet Capture & Elasticsearch Indexer

A Node.js-based network packet capture tool that captures packets from network interfaces and indexes them into Elasticsearch for analysis and monitoring.

Features

• 🔍 Multi-interface capture: Capture from one or multiple network interfaces simultaneously
• 🎯 Flexible filtering: Filter by protocol (TCP/UDP/ICMP), ports, and port ranges
• 🔒 Promiscuous mode support: Optionally capture all packets on the network segment
• 📊 Elasticsearch integration: Automatic indexing with optimized mapping
• <EFBFBD> Failover cache: In-memory cache for packets when Elasticsearch is unavailable- 🌍 GeoIP enrichment: Automatic geolocation data for remote IP addresses- <20>📝 Content extraction: Captures and indexes readable (ASCII) packet content
• 🚀 Smart content handling: Automatically skips large binary content while preserving packet metadata
• 📈 Real-time statistics: Track capture performance and statistics
• ⚙️ Highly configurable: Environment variables and config file support

Prerequisites

• Node.js >= 14.0.0
• Elasticsearch 7.x or 8.x
• Root/Administrator privileges (required for packet capture)
• Linux: libpcap-dev (apt-get install libpcap-dev)
• macOS: XCode Command Line Tools

Installing System Dependencies

Ubuntu/Debian:

sudo apt-get update
sudo apt-get install libpcap-dev build-essential

CentOS/RHEL:

sudo yum install libpcap-devel gcc-c++ make

macOS:

xcode-select --install

Installation

1. Clone or navigate to the project directory:

cd /path/to/netpcap

2. Install Node.js dependencies:

npm install

Note: The geoip-lite package will download a GeoIP database on first install. This may take a few moments.

3. Copy the example environment file and configure:

cp .env.example .env
# Edit .env with your configuration

Configuration

Configuration can be done via environment variables or by editing the config.js file directly.

Elasticsearch Configuration

ES_NODE=http://localhost:9200
ES_USERNAME=elastic
ES_PASSWORD=your_password
ES_INDEX=network-packets

Capture Settings

Interfaces:

# Capture from specific interfaces
CAPTURE_INTERFACES=eth0,wlan0

# Leave empty to capture from all available interfaces
CAPTURE_INTERFACES=

Promiscuous Mode:

# Enable to capture all packets on the network segment
PROMISCUOUS_MODE=true

Filtering

Protocol Filtering:

# Only capture specific protocols
FILTER_PROTOCOLS=tcp,udp

# Capture all protocols (leave empty)
FILTER_PROTOCOLS=

Port Filtering:

# Exclude specific ports (e.g., SSH, HTTP, HTTPS)
EXCLUDE_PORTS=22,80,443

# Exclude port ranges
EXCLUDE_PORT_RANGES=[[8000,9000],[3000,3100]]

# Only capture specific ports (takes precedence)
INCLUDE_PORTS=3306,5432

Custom BPF Filter:

# Use custom Berkeley Packet Filter syntax
CAPTURE_FILTER="tcp and not port 22"

Content Indexing

# Maximum content size to index (1MB default)
MAX_CONTENT_SIZE=1048576

# Enable/disable content indexing
INDEX_READABLE_CONTENT=true

Cache System (Elasticsearch Failover)

The application includes an in-memory cache system to handle Elasticsearch outages:

# Maximum documents to cache in memory (default: 10000)
CACHE_MAX_SIZE=10000

# ES availability check interval in milliseconds (default: 5000)
CACHE_CHECK_INTERVAL=5000

How it works:

• When Elasticsearch is unavailable, packets are stored in memory cache
• The system periodically checks ES availability (every 5 seconds by default)
• When ES comes back online, cached documents are automatically flushed
• If cache reaches maximum size, oldest documents are removed (FIFO)
• On graceful shutdown (SIGINT/SIGTERM), the system attempts to flush all cached documents

GeoIP Enrichment

The application automatically enriches captured packets with geolocation data for remote IP addresses:

Features:

• Automatically detects the server's public IP address
• Identifies private/local IP addresses (RFC 1918, link-local, etc.)
• Only enriches remote public IP addresses
• Uses local GeoIP database (no external API calls during capture)
• Adds geolocation data for both source and destination IPs

GeoIP Data Included:

• Country code and name
• Region/state
• City
• Timezone
• Geographic coordinates (latitude/longitude)
• Geo-point data for Elasticsearch geo queries

How it works:

1. On startup, the application detects its public IP using an external service (ipify.org)
2. For each packet, it identifies if source/destination IPs are remote
3. Remote IPs are enriched with GeoIP data from the local database
4. Private IPs, loopback, and the server's own IP are excluded

Example GeoIP document structure:

{
  "geoip_src": {
    "country": "US",
    "region": "CA",
    "city": "San Francisco",
    "timezone": "America/Los_Angeles",
    "location": {
      "lat": 37.7749,
      "lon": -122.4194
    }
  },
  "geoip_dst": {
    "country": "DE",
    "city": "Frankfurt",
    ...
  }
}

Usage

Basic Usage

Run with default configuration:

sudo npm start

Or directly:

sudo node index.js

Capture from Specific Interface

sudo CAPTURE_INTERFACES=eth0 node index.js

Capture Only HTTP/HTTPS Traffic

sudo INCLUDE_PORTS=80,443 FILTER_PROTOCOLS=tcp node index.js

Exclude SSH and High Ports

sudo EXCLUDE_PORTS=22 EXCLUDE_PORT_RANGES=[[8000,65535]] node index.js

Enable Promiscuous Mode

sudo PROMISCUOUS_MODE=true node index.js

Debug Mode

sudo LOG_LEVEL=debug node index.js

Elasticsearch Index Structure

The tool creates an index with the following document structure:

{
  "@timestamp": "2026-02-11T10:30:00.000Z",
  "interface": {
    "name": "eth0",
    "ip": "192.168.1.100",
    "mac": "aa:bb:cc:dd:ee:ff"
  },
  "ethernet": {
    "src": "aa:bb:cc:dd:ee:ff",
    "dst": "11:22:33:44:55:66",
    "type": 2048
  },
  "ip": {
    "version": 4,
    "src": "192.168.1.100",
    "dst": "8.8.8.8",
    "protocol": 6,
    "ttl": 64,
    "length": 60
  },
  "tcp": {
    "src_port": 54321,
    "dst_port": 443,
    "flags": {
      "syn": true,
      "ack": false,
      "fin": false,
      "rst": false,
      "psh": false
    },
    "seq": 123456789,
    "ack_seq": 0,
    "window": 65535
  },
  "geoip_dst": {
    "country": "US",
    "country_name": "US",
    "region": "CA",
    "city": "Mountain View",
    "timezone": "America/Los_Angeles",
    "location": {
      "lat": 37.4056,
      "lon": -122.0775
    },
    "coordinates": [-122.0775, 37.4056]
  },
  "content": "GET / HTTP/1.1\r\nHost: example.com\r\n",
  "content_length": 1024,
  "content_type": "binary"
}

Querying Captured Data

Example Elasticsearch Queries

Find all packets from a specific IP:

curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "term": {
      "ip.src": "192.168.1.100"
    }
  }
}
'

Find all SYN packets (connection attempts):

curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "term": { "tcp.flags.syn": true } },
        { "term": { "tcp.flags.ack": false } }
      ]
    }
  }
}
'

Find packets with readable content:

curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "exists": {
      "field": "content"
    }
  }
}
'

Find packets from a specific country:

curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "term": {
      "geoip_src.country": "US"
    }
  }
}
'

Find packets to/from a specific geographic area:

curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "geo_distance": {
      "distance": "100km",
      "geoip_dst.location": {
        "lat": 37.7749,
        "lon": -122.4194
      }
    }
  }
}
'

Aggregate packets by destination country:

curl -X GET "localhost:9200/network-packets/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "aggs": {
    "countries": {
      "terms": {
        "field": "geoip_dst.country"
      }
    }
  }
}
'

Performance Considerations

• Promiscuous mode can generate high packet volumes on busy networks
• Content indexing increases storage requirements significantly
• Use port filters to reduce captured packet volume
• Adjust MAX_CONTENT_SIZE based on your storage capacity
• Monitor Elasticsearch cluster health when capturing high-volume traffic
• Cache system protects against data loss during ES outages but consumes memory
• Adjust CACHE_MAX_SIZE based on available RAM (each packet ~1-5KB in memory)

Troubleshooting

Permission Denied Errors

Packet capture requires root privileges:

sudo node index.js

Interface Not Found

List available interfaces:

ip link show  # Linux
ifconfig      # macOS/Unix

Elasticsearch Connection Failed

Verify Elasticsearch is running:

curl -X GET "localhost:9200"

No Packets Being Captured

1. Check if the interface is up and receiving traffic
2. Verify filter configuration isn't too restrictive
3. Try running without filters first
4. Check system firewall settings

Security Considerations

⚠️ Important Security Notes:

• This tool captures network traffic and may contain sensitive information
• GeoIP data reveals geographic locations of communication endpoints
• Store Elasticsearch credentials securely
• Restrict access to the Elasticsearch index
• Be aware of privacy and legal implications when capturing network traffic
• Use encryption for Elasticsearch connections in production
• The public IP detection makes an external HTTPS call to ipify.org on startup
• Comply with applicable laws and regulations regarding network monitoring and data retention

License

MIT

Author

ale

Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.