Trigo Documentation

W3C Test Suites

Trigo includes infrastructure for running both the official W3C SPARQL 1.1 test suite and W3C RDF 1.1/1.2 test suites to validate compliance with SPARQL and RDF standards.

Setup

The W3C test suite is included as a git submodule:

# Clone with submodules
git clone --recursive https://github.com/aleksaelezovic/trigo.git

# Or initialize submodules after cloning
git submodule update --init --recursive

Test Suite Structure

The test suite is located in testdata/rdf-tests/ and includes:

Test Categories

Tests are organized by feature:

sparql11/
├── syntax-query/        # Query syntax (positive/negative)
├── syntax-update/       # Update syntax tests
├── aggregates/          # COUNT, SUM, AVG, etc.
├── bind/                # BIND clause
├── bindings/            # VALUES clause
├── construct/           # CONSTRUCT queries
├── exists/              # EXISTS and NOT EXISTS
├── functions/           # Built-in functions
├── grouping/            # GROUP BY
├── negation/            # MINUS, FILTER NOT EXISTS
├── property-path/       # Property paths
├── subquery/            # Subqueries
└── ...

Running Tests

Build the Test Runner

go build -o test-runner ./cmd/test-runner

Run Specific Test Suite

# Run SPARQL syntax tests
./test-runner testdata/rdf-tests/sparql/sparql11/syntax-query

# Run a specific SPARQL manifest
./test-runner testdata/rdf-tests/sparql/sparql11/syntax-query/manifest.ttl

# Run RDF parser tests
./test-runner testdata/rdf-tests/rdf/rdf11/rdf-turtle      # Turtle parser tests
./test-runner testdata/rdf-tests/rdf/rdf11/rdf-n-triples   # N-Triples parser tests
./test-runner testdata/rdf-tests/rdf/rdf11/rdf-n-quads     # N-Quads parser tests
./test-runner testdata/rdf-tests/rdf/rdf11/rdf-trig        # TriG parser tests
./test-runner testdata/rdf-tests/rdf/rdf11/rdf-xml         # RDF/XML parser tests

# Run RDF 1.2 parser tests
./test-runner testdata/rdf-tests/rdf/rdf12/rdf-turtle      # RDF 1.2 Turtle tests
./test-runner testdata/rdf-tests/rdf/rdf12/rdf-n-triples   # RDF 1.2 N-Triples tests
./test-runner testdata/rdf-tests/rdf/rdf12/rdf-trig        # RDF 1.2 TriG tests

Example Output

📋 Running manifest: testdata/rdf-tests/sparql/sparql11/syntax-query/manifest.ttl
   Found 94 tests

  ✅ PASS: syntax-construct-where-02.rq
  ❌ FAIL: syntax-aggregate-01.rq
  ⏭️  SKIP: syntax-update-01.ru

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📊 TEST SUMMARY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total:   94
Passed:  29 (30.9%)
Failed:  64
Skipped: 1
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Test Types

SPARQL Test Types

Positive Syntax Tests

Verify that valid SPARQL queries parse successfully:

:test1 rdf:type mf:PositiveSyntaxTest11 ;
    mf:name "Basic SELECT" ;
    mf:action <query.rq> .

Negative Syntax Tests

Verify that invalid queries are rejected:

:test2 rdf:type mf:NegativeSyntaxTest11 ;
    mf:name "Invalid WHERE" ;
    mf:action <bad-query.rq> .

Query Evaluation Tests

Execute queries against data and compare results:

:test3 rdf:type mf:QueryEvaluationTest ;
    mf:name "Basic BGP" ;
    mf:action [
        qt:query <query.rq> ;
        qt:data <data.ttl>
    ] ;
    mf:result <result.srx> .

RDF Test Types

Positive Syntax Tests

Verify that RDF documents parse successfully:

:test1 rdf:type rdft:TestTurtlePositiveSyntax ;
    mf:name "Basic Turtle" ;
    mf:action <data.ttl> .

Negative Syntax Tests

Verify that invalid RDF documents are rejected:

:test2 rdf:type rdft:TestTurtleNegativeSyntax ;
    mf:name "Invalid Turtle" ;
    mf:action <bad-data.ttl> .

Evaluation Tests

Parse RDF documents and compare with expected triples:

:test3 rdf:type rdft:TestTurtleEval ;
    mf:name "Complex Turtle" ;
    mf:action <input.ttl> ;
    mf:result <expected.nt> .

Current Test Support

RDF Parser Tests (RDF 1.1 & 1.2)

Implemented Formats

Test Results

Recent Improvements (Phase 1):

Recent Improvements (Phase 3 - Graph Isomorphism & SPARQL Results):

Recent Improvements (Phase 4 - RDF 1.2 Enhancements):

Recent Improvements (Phase 2 - TriG & RDF/XML):

Syntax Tests (Parser Validation)

SPARQL Execution Tests (End-to-End Validation)

Implemented and Validated

Parsed But Not Executed

Not Yet Implemented

Implementation Details

Test Runner Architecture

Test Runner
    ↓
Manifest Parser (.ttl files)
    ↓
Test Evaluator
    ↓
├─ SPARQL Syntax Tests → SPARQL Parser
├─ SPARQL Evaluation Tests → Full Pipeline:
│   ├─ Turtle Parser (load test data)
│   ├─ SPARQL Parser (parse query)
│   ├─ Optimizer (build execution plan)
│   ├─ Executor (run query)
│   ├─ SPARQL XML Parser (parse expected results)
│   └─ Result Comparator (validate correctness)
├─ RDF Positive Syntax Tests → RDF Parser (validate parse success)
├─ RDF Negative Syntax Tests → RDF Parser (validate parse failure)
├─ RDF Evaluation Tests → RDF Parser + Triple Comparator
│   ├─ Parse input RDF (Turtle/TriG/RDF-XML/JSON-LD)
│   ├─ Parse expected N-Triples output
│   └─ Compare triples (order-independent, blank node handling)
└─ SPARQL Update Tests → (TODO)

Key Components

Turtle Parser (pkg/rdf/turtle.go):

SPARQL XML Parser (pkg/server/results/xml.go):

Query Evaluation (internal/testsuite/runner.go):

  1. Clear store between tests
  2. Load test data using Turtle parser
  3. Parse SPARQL query
  4. Optimize query plan
  5. Execute query with full pipeline
  6. Parse expected results
  7. Compare actual vs expected (order-independent)

Manifest Format

Manifests are Turtle files describing tests:

@prefix mf: <http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#> .
@prefix qt: <http://www.w3.org/2001/sw/DataAccess/tests/test-query#> .

<> rdf:type mf:Manifest ;
    mf:entries (
        :test1
        :test2
    ) .

:test1 a mf:PositiveSyntaxTest11 ;
    mf:name "Test Name" ;
    mf:action <query-file.rq> .

Adding Test Support

To add support for a new test type:

  1. Update Manifest Parser (internal/testsuite/manifest.go)
    • Add new TestType constant
    • Update parser to recognize new test type
  2. Implement Test Evaluator (internal/testsuite/runner.go)
    • Add case in runTest() switch
    • Implement evaluation function
  3. Add Required Features (if needed)
    • Extend parser for new syntax
    • Add optimizer support
    • Implement executor operators

Testing Philosophy

The W3C SPARQL test suite serves multiple purposes:

  1. Compliance Validation - Ensure standard conformance
  2. Regression Testing - Catch breaking changes
  3. Feature Tracking - Identify gaps in implementation
  4. Documentation - Tests demonstrate expected behavior

Integration with CI/CD

GitHub Actions Example

name: SPARQL Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          submodules: recursive

      - uses: actions/setup-go@v4
        with:
          go-version: '1.21'

      - name: Build Test Runner
        run: go build -o test-runner ./cmd/test-runner

      - name: Run Syntax Tests
        run: ./test-runner testdata/rdf-tests/sparql/sparql11/syntax-query

References

Future Improvements

Contributing

When adding new SPARQL features:

  1. Run relevant test suite section
  2. Document pass rate in commit message
  3. Create issues for failing tests
  4. Update this document with current status