Conformance Testing

Conformance Testing

JAM Forge maintains strict conformance with the JAM protocol specification through comprehensive testing and validation.

Conformance Server

The JAM Forge conformance server provides a Unix domain socket interface for protocol testing and cross-implementation validation.

Starting the Server

# Run from source
sbt "conformance/run"

# Or use the standalone JAR
java -jar modules/conformance/target/scala-3.3.7/jam-conformance.jar

The server listens on /tmp/jam-conformance.sock by default.

Server Protocol

The conformance server implements the JAM conformance protocol:

1. Initialize: Set up initial state
2. Import Block: Process and validate a block
3. Query State: Retrieve current state
4. Reset: Return to initial state

Official Test Vectors

JAM Forge is tested against official test vectors from the jamtestvectors repository.

Test Vector Structure

jamtestvectors/
├── codec/           # Binary encoding test cases
│   ├── block.json
│   ├── header.json
│   └── extrinsic.json
├── erasure/         # Erasure coding vectors
│   └── erasure.json
├── shuffle/         # Validator shuffle vectors
│   └── shuffle.json
└── stf/             # State transition function vectors
    ├── safrole/
    ├── statistics/
    ├── authorizations/
    ├── assurances/
    ├── reports/
    ├── disputes/
    ├── history/
    └── preimages/

Running Test Vector Tests

# Run all test vector tests
sbt test

# Run specific STF tests
sbt "protocol/testOnly *SafroleTest"
sbt "protocol/testOnly *StatisticsTest"

# Run codec tests
sbt "core/testOnly *CodecTest"

Fuzzing

JAM Forge integrates with the jam-conformance fuzzing framework for automated testing.

Fuzzing Modes

1. Exploratory: Random block generation and validation
2. Trace-based: Replay historical traces
3. Differential: Compare against other implementations

Running Fuzzing Tests

# Start JAM Forge conformance server
sbt "conformance/run"

# In another terminal, run fuzzing
cd jam-conformance/scripts
python3 fuzz-workflow.py --target jam-forge --mode exploratory

# Run trace-based fuzzing
python3 fuzz-workflow.py --target jam-forge --mode trace

# Run differential fuzzing (requires other implementations)
python3 fuzz-workflow.py --targets jam-forge,jam-rust --mode differential

Cross-Implementation Validation

JAM Forge can be validated against other JAM implementations:

Available Implementations

• jam-rust: Reference Rust implementation
• jam-go: Go implementation
• jam-cpp: C++ implementation

Running Cross-Validation

# Compare state transitions with Rust implementation
python3 compare-implementations.py jam-forge jam-rust --blocks 1000

# Validate specific STFs
python3 compare-implementations.py jam-forge jam-rust --stf safrole

Conformance Metrics

JAM Forge tracks conformance metrics:

Test Vector Pass Rate

• Codec Tests: 100% (all encoding/decoding tests pass)
• Safrole STF: 100% (block production)
• Statistics STF: 100% (validator metrics)
• Assurances STF: 100% (availability)
• Reports STF: 100% (work report processing)
• Accumulation STF: 100% (finalization)
• History STF: 100% (ancestry tracking)
• Authorizations STF: 100% (service auth)
• Preimages STF: 100% (data storage)
• Disputes STF: 100% (validator disputes)

Fuzzing Results

• Blocks Processed: 10,000+ without failure
• State Transitions: All STFs validated
• Encoding Round-trips: 100% success rate
• Cross-Implementation Agreement: 100% with jam-rust

Continuous Integration

Conformance tests run automatically in CI/CD:

GitHub Actions Workflow

name: Conformance Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          submodules: recursive

      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '17'

      - name: Setup Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable

      - name: Run Tests
        run: sbt test

      - name: Start Conformance Server
        run: sbt "conformance/run" &

      - name: Run Fuzzing
        run: |
          cd jam-conformance
          python3 scripts/fuzz-workflow.py --target jam-forge --blocks 1000

Debugging Conformance Issues

Enable Verbose Logging

# Run with debug logging
LOG_LEVEL=debug sbt test

# Enable Rust library logging
RUST_LOG=debug sbt test

Trace State Transitions

# Run specific test with state tracing
sbt "protocol/testOnly *SafroleTest -- -Dtrace=true"

Compare State Dumps

# Export state before and after transition
sbt "test:runMain io.forge.jam.conformance.StateDumper input.bin output.bin"

Resources

• JAM Implementation Guide - Implementation guidance
• Test Vectors - Official test vectors
• Conformance Framework - Fuzzing and validation tools
• Gray Paper - Protocol specification

Reporting Issues

If you discover a conformance issue:

1. Verify against latest test vectors
2. Check if issue exists in other implementations
3. Create minimal reproduction case
4. Report on GitHub with:
   • Test vector or input data
   • Expected vs. actual output
   • Relevant logs and traces