SBOM Quality Assurance Test

1. Scope

The scope of this project is to evaluate and benchmark open source Software Composition Analyzis (SCA) or SBOM generation tools in a structured and standardized manner.

Specifically, the project covers:

• Open source software repositories as test targets, spanning multiple programming languages and ecosystems.
• Generation and analysis of SBOMs in the SPDX format.
• Use of reference SBOMs exported from original repository dependency graphs to serve as a consistent baseline.
• Alignment with the OpenChain Telco SBOM Guide to ensure standardized practices for SBOM structure, content, and validation.
• Evaluation of tool performance, accuracy, and coverage to deliver reliable and actionable benchmarking results.

This project is licensed under the BSD 3-Clause License.

2. Terms and Definitions

SBOM

A Software Bill of Materials (SBOM) is a structured inventory of all software components, libraries, and dependencies within an application, including key metadata such as version, license, and origin. It enhances transparency, security, and compliance across the software supply chain.

SBOM Types

SBOMs can exist at different stages of the software lifecycle, including Design, Source, Build, Analyze, Deploye, and Runtime, as defined in the CISA guidelines.

Data Format

Data Format refers to the structure in which SBOM information is represented. Common formats include SPDX, CycloneDX, or other proprietary formats. For the purposes of this project, the SBOMs are represented using the SPDX format.

Package

A package is a reusable software unit, such as a library or module, distributed via package managers and accompanied by metadata including version, license, and authorship.

Package Manager

A package manager is a tool that automates the installation, upgrade, configuration, and removal of software packages, resolving both direct and transitive dependencies. Examples include npm, pip, Maven, Gradle, and Conan.

Software Composition Analysis (SCA) Tools

Tools that analyze software artifacts to identify components and extract metadata (e.g., licenses, versions, suppliers, vulnerabilities), commonly used to generate SBOMs for transparency, compliance, and security. These tools are often called SBOM generators.

Direct Dependency

A direct dependency is a first-level package or component explicitly declared in a project’s configuration or build file.

Transitive Dependency

A transitive dependency is an indirect package required by a direct dependency of a project.

3. Methodology

3.1. Tools Selection Criteria

• Open Source and Actively Maintained: Only open source tools with active development communities and regular updates were considered to ensure relevance, reliability, and transparency.

• Support for SPDX JSON Output: Since SPDX JSON was the standardized SBOM format selected for comparison, all selected tools needed to either natively support or allow conversion to this format.

• Language and Ecosystem Compatibility: Tools were required to support multiple programming languages and environments, particularly those used in our test targets (e.g., C, C++, Java, Python, Node.js, Go, container images).

• Alignment with OpenChain Telco SBOM Guide: Where applicable, tools were evaluated for their ability to generate SBOMs conforming to the recommendations and structure outlined in the OpenChain Telco SBOM Guide.

• Richness of Metadata: Tools were assessed on the granularity and completeness of SBOM content, including license information, versioning, source origin, cryptographic hashes, and component relationships.

3.2. Tools

This section lists the SBOM generation tools evaluated in this project, categorized into general-purpose and container-based tools.

General purpose tools:

1. Syft | Docs | v1.26.1

A CLI tool and Go library for generating an SBOM from container images and filesystems. Exceptional for SBOM creation and integrates well with vulnerability scanners like Grype.

Note: ammend/syft – A fork of Syft designed to generate SPDX JSON compliant with the OpenChain Telco SBOM Guide. Recommended for telecom sector requirements.

2. Trivy | Docs | v0.63.0

A comprehensive security tool for SBOM generation, vulnerability detection, license analysis, misconfiguration scanning, and secret discovery across container images, filesystems, repositories, VMs, and Kubernetes environments.

3. OSS Review Toolkit (ORT) | Docs | 62.2.0

A policy automation and orchestration toolkit for SBOM generation, license compliance, vulnerability detection, and open source risk management. ORT supports CycloneDX, SPDX, and custom attribution documents, while enabling policy-as-code checks, dependency analysis, and automated reporting across software projects.

4. SCANOSS | Docs | v1.26.2

The SCANOSS Python package provides a simple library for interacting with the SCANOSS APIs and engine, enabling SBOM generation, license compliance, and open source component identification.

Tools analyzing containers:

1. Syft | Docs | v1.30.0

A CLI tool and Go library for generating SBOMs from container images. It identifies installed packages and their metadata across multiple ecosystems, supporting images from registries, local Docker/OCI images, and tar archives.

2. Tern | v2.12.1

An inspection tool to collect metadata of packages installed in a container image. It analyzes each layer of the image, executes scripts in a chroot environment to gather package information, and generates a detailed report listing packages and their metadata, with optional mapping to Dockerfile instructions.

3. Distro2SBOM | 0.6.0

Generates an SBOM for either an installed application or a complete system installation in formats like SPDX and CycloneDX. It identifies all dependent components of a package and is intended for use in continuous integration systems to maintain accurate SBOM records and support audit requirements.

3.3. Test Targets Selection Criteria

• Real-World Relevance: Only publicly available, widely used open source projects were considered to ensure practical evaluation of SCA tools.
• Ecosystem Diversity: Projects cover multiple languages, build systems, both with and without package managers.
• Reference SBOM Availability: For each test target, a Reference SBOM was available. We used the SBOMs generated by GitHub for the projects.

3.4. Test Targets

The test targets linked in this part are the original, publicly available repositories of the respective projects.

1. C (No package manager)

libxcrypt | tag-v4.4.38 is a modern C library for one-way hashing of passwords, supporting various algorithms like bcrypt, md5crypt, and yescrypt. It provides traditional Unix crypt interfaces and extended functions for secure password handling. The project does not utilize a package manager, making it suitable for manual integration and analysis.

2. C++ (No package manager)

MeshOptimizer | tag-v0.24 is an open source C++ library developed by Arseny Kapoulkine, providing algorithms to optimize meshes for modern GPU vertex and index processing pipelines. It can reindex an existing index buffer or generate an entirely new set of indices from an unindexed vertex buffer. The project does not utilize a package manager, making it suitable for manual integration and analysis.

3. C++ (Conan)

Catch2 | tag-v3.9.0 is a modern, header-only testing framework for C++. It provides robust unit testing, micro-benchmarking, and test case management. For this study, the project is managed using the Conan package manager.

4. Go

Hugo | tag-v0.147.4 is a fast and flexible static site generator written in Go. It is widely used for websites, blogs, documentation, and portfolios.

5. Node.js

Express | tag-v5.1.0 is a minimal and flexible Node.js web application framework that provides robust features for building web and mobile applications.

6. Python FastAPI & GPT Engineer

• FastAPI | tag-0.116.0 is a modern, fast (high-performance) web framework for building APIs with Python 3.7+ based on standard Python type hints. It is widely used for developing web applications and APIs efficiently.

• GPT Engineer | tag-v0.3.1 is a Python project designed to facilitate building AI-driven solutions and applications, providing a structured environment for rapid prototyping and experimentation.

7. Java (Maven-managed)

JavaCV | tag-1.5.12 is a Java interface to OpenCV, FFmpeg, and other computer vision and machine learning libraries. It provides a comprehensive set of tools for image and video processing, machine learning, and computer vision tasks. The project is managed using the Maven package manager.

8. ContainerImage

pggb | tag-v0.7.4 builds pangenome variation graphs from input sequences using wfmash, seqwish, smoothxg, gfaffix, and odgi.

Note: The project provides a Dockerfile for containerized usage, enabling local builds or pulls from the GitHub Container Registry.

3.5. Reference SBOM

A Reference SBOM is a standardized, machine-readable inventory of a project's dependencies and associated metadata, such as versions, licenses, and transitive relationships. It is generated directly from the dependency graph of the original, publicly available repositories of the respective test targets.

These SBOMs provide a canonical representation of all software components, including direct and transitive dependencies, and serve as a baseline for comparison and validation. The dependency graph is a summary of the manifest and lock files stored in a repository and any dependencies that are submitted for the repository using the dependency submission API (GitHub Docs: Dependency Graph).

For each test target, the Reference SBOM was exported from its GitHub repository using automated SBOM generation tools. These SBOMs capture the original dependency tree and associated metadata (e.g., versions, licenses) before any modifications or local testing. GitHub facilitates this process by enabling users to export an SBOM from a repository's dependency graph through the UI or REST API, producing an SPDX-compatible JSON file (GitHub Docs: Exporting SBOM).

We are aware that these Reference SBOMs are not complete or perfect, but these SBOMs are generated with a consitent toolchain and in a consistent environment.

3.6. Comparison and Analysis

The generated SBOMs from each tool were benchmarked against the corresponding reference SBOMs.

The comparison focused on the following criteria:

• Dependency coverage (direct and transitive)
• Accuracy of versioning
• Presence of critical metadata (e.g., license)

3.7. Validation

OpenChain Telco SBOM Validator was used to validate the structural compliance and metadata quality of each generated SBOM. Validation criteria included:

• Conformance to SPDX specification
• Inclusion of mandatory and recommended fields
• Logical consistency (e.g., relationships, identifiers)

---

4. Observation & Result

4.1 General-Purpose SBOM Generation Tools

4.1.1 Syft

Command: For generating SBOMs for all test targets this command is used:

syft -o spdx-json=syft-sbom.spdx.json --enrich all --verbose .

4.1.1.1 Ecosystem: Node.js

• Default SBOM: syft-Nodejs.spdx.json

Generated directly from the project source without performing any compilation step.

• Compilation Step:

The project did not originally include a package-lock.json file, which is required to resolve and capture the full dependency tree. To generate it, the following command was executed in test target root:

npm install --package-lock

Generated Files: - package-lock.json , node_modules

Enriched SBOMs: syft-Nodejs-compilation.spdx.json

4.1.1.2 Ecosystem: Go

• Default SBOM: syft-Go.spdx.spdx.json

4.1.1.3 Ecosystem: Python (FastAPI)

• Default SBOM: syft-python.spdx.json

• Compilation Step:

The project did not include a pdm.lock file. To generate it and pin all dependencies, the following command was executed:

pdm lock

Dependencies must be installed for the environment using:

pdm install

Enriched SBOMs: syft-python-compilation.spdx.json

4.1.1.4 Ecosystem: Python (GPT Engineer)

• Default SBOM: syft-python2.spdx.json

• Compilation Step:

The project included both pyproject.toml and poetry.lock. To install only the production dependencies (excluding development packages), the following command was executed:

poetry install --no-dev

Enriched SBOMs: syft-python2-compilation.spdx.json

4.1.1.5 Ecosystem: C++ (Conan)

• Default SBOM: syft-C++-CONAN.spdx.json

4.1.1.6 Ecosystem: C (No package manager)

• Default SBOM: syft-C.spdx.json

4.1.1.7 Ecosystem: C++ (No package manager)

• Default SBOM: syft-C++.spdx.json

4.1.1.8 Ecosystem: Java (Maven-managed)

• Default SBOM: syft-Java.spdx.json

• Compilation Step: Build

The project was compiled to ensure that all direct and transitive dependencies were resolved and packaged into build artifacts (target/.jar)*. following command was executed in tese target root:

mvn clean package -DskipTests

Generated Files: - target/.jar*

Result:

Ensures a richer and more accurate SBOM including both direct and transitive dependencies.

Enriched SBOMs: syft-Java-compilation.spdx.json

Note: During the review of ammend/syft, it was observed that this fork made no actual changes compared to the original Syft project. All steps, processes, and the generated SBOMs are completely identical to those produced by the original Syft.

4.1.2. Scanoss

Generating an SBOM using scanoss-py involves two steps:

1. Scan the project and generate raw SBOM Command: (in test target root)

   scanoss-py scan -o scanoss-raw.spdx.json .
   

2. Convert raw SBOM to SPDX Lite format Command: (in test target root)

   scanoss-py convert --input scanoss-raw.spdx.json --format spdxlite --output scanoss.spdx.json
   

4.1.2.1 Ecosystem: Node.js

• Default SBOM: scanoss-Nodejs.spdx.json

• Compilation Step:

The project did not originally include a package-lock.json file, To ensure accurate scanning and SBOM generation, the project dependencies were prepared as follows(in root of project):

npm install --package-lock-only

Creates the package-lock.json file. (Does not install anything into node_modules)

To Install only production dependencies based on package-lock.json and Creates a clean environment containing runtime dependencies only, (reducing the risk of false positives from development or test packages during scanning), the following command was executed:

npm ci --only=production

- Enriched SBOMs: scanoss-compilation-Nodejs.spdx.json

4.1.2.2 Ecosystem: Go

• Default SBOM: scanoss-Go.spdx.json

• Compilation Step:

To ensure all required dependencies are included and the module files are consistent, the following command was executed in the project root:

go mod tidy

- Updated files: go.mod , go.sum

Additionally, to improve scanning accuracy, all required dependencies were copied into a local vendor/ directory, exposing the actual source of dependencies to SCANOSS . The following command was executed:

go mod vendor

- Enriched SBOMs: scanoss-Go-compilation.spdx.json

4.1.2.3 Ecosystem: Python (FastAPI)

• Default SBOM: scanoss-python.spdx.json
• Compilation Step:

To ensure accurate SBOM generation with ScanOSS, all project dependencies were locked, installed, and vendored in the project root: - Lock dependencies:

pdm lock

- Install dependencies in clean environment:

pdm install

- Export resolved dependencies:

pdm export -f requirements > requirements.txt

- Create vendor folder with actual package sources:

mkdir -p vendor

pip install --target=vendor -r requirements.txt

• Enriched SBOMs: scanoss-python-compilation.spdx.json

4.12.4 Ecosystem: Python (GPT Engineer)

• Default SBOM: scanoss-python2.spdx.json

4.1.2.5 Ecosystem: C++ (Conan)

• Default SBOM: scanoss-C++-CONAN.spdx.json

4.1 2.6 Ecosystem: C++ (No package manager)

• Default SBOM: scanoss-C++.spdx.json

• Compilation Step:

The project was configured and built in Release mode using CMake .following commands were executed: - Configure the project:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release

- Build the project:

cmake --build build

- Enriched SBOMs: scanoss-C++-compilation.spdx.json

4.1.2.7 Ecosystem: C (No package manager)

• Default SBOM: scanoss-C.spdx.json

• Compilation Step:

The project was configured and built using the standard autotools workflow. following commands were executed in project root: - Prepare the build system:

/autogen.sh

- Configure the project

./configure

- Compile the project

make

- Enriched SBOMs: scanoss-C-compilation.spdx.json

4.1.2.8 Ecosystem: Java (Maven-managed)

• Default SBOM: scanoss-Java.spdx.json

• Compilation Step:

To compile the project and resolve all dependencies, the following command was executed:

mvn clean install -DskipTests

- Enriched SBOMs: scanoss-Java-compilation.spdx.json

4.1.2.9 Using SCANOSS with API Key

To authenticate with the SCANOSS API, the API key must first be set as an environment variable:

export SCANOSS_API_KEY="your-api-key"

Then, run the scan command to analyze the project and output the raw results in JSON format:

scanoss-py scan --key $SCANOSS_API_KEY -o results.spdx.json .

Note: SBOMs were also generated using the SCANOSS API key for authenticated scanning. While the number of detected packages remained almost unchanged compared to unauthenticated runs, using the API key removes rate limits and allows continuous SBOM generation without waiting periods.

4.1.3. Trivy

Command: (In test target root):

trivy fs --format spdx-json --scanners vuln,license,secret,misconfig --output trivy-sbom.spdx.json .

4.1.3.1 Ecosystem:* Node.js

Default SBOM: trivy-Nodejs.spdx.json

• Compilation Step:

The project did not originally include a package-lock.json file, which is required to resolve and capture the full dependency tree without actually installing dependencies, the following command was executed in test target root:

npm install --package-lock-only

Created Files:

• package-lock.json

To create a clean environment with only production dependencies installed (excluding devDependencies), the following command was executed:

npm ci --only=production

Enriched SBOMs: trivy-Nodejs-compilation.spdx.json

4.1.3.2 Ecosystem: C++ (Conan)

Default SBOM: trivy-C++-CONAN.spdx.json

4.1.3.3 Ecosystem: Go

• Compilation Step:

To ensure all required dependencies are included and the module files are consistent, the following command was executed in the project root:

go mod tidy

Updated Files:

• go.mod , go.sum

Enriched SBOMs: trivy-Go.spdx.json

4.1.3.4 Ecosystem:**

C++ (No package manager)

Default SBOM: trivy-C++.spdx.json

4.1.3.5 Ecosystem:**

Python (FastAPI)

Default SBOM: trivy-python.spdx.json

• Compilation Step:

The project did not include a pdm.lock file. To generate it and pin all dependencies, the following command was executed:

pdm lock

Dependencies must be installed for the environment using:

pdm install

Enriched SBOMs: trivy-python-compilation.spdx.json

Note: As of now, Trivy does not support scanning Python projects managed with PDM. Specifically, Trivy does not parse the pdm.lock file, which means it cannot fully resolve and capture the project's dependency tree. This limitation affects the accuracy and completeness of Software Bill of Materials (SBOM) generation for PDM-managed Python projects.

For more information and to track the progress of this feature, refer to the following GitHub issue: Trivy GitHub Issue: Add support for PDM lockfile parsing

4.1.3.6 Ecosystem: Python (GPT Engineer)

Default SBOM: trivy-python2.spdx.json

3.7 Ecosystem: C (No package manager)

Default SBOM: trivy-C.spdx.json

3.8 Ecosystem: Java (Maven-managed)

Default SBOM: trivy-Java.spdx.json

4.1.4 ORT

Command:

For generating SBOMs for all test targets these commands are used:

Analyze phase:

ort analyze -i /path/to/your/project -o /path/to/output-dir

Report phase:

ort report -i /path/to/analyzer-result.yml -o /path/to/output-dir -f spdxDocument

Note: The resulting output is an SBOM compliant with SPDX version 2.2, delivered as a YAML file. Although the maintainers of ORT suggested the use of the following parameter to enforce SPDX version 2.3, no version change was observed in practice:

-O SpdxDocument=spdx_version=SPDX-2.3

The generation of an SBOM with the ORT is structured into several phases:

• Analyzer

• Scanner

• Advisor

• Evaluator

• Reporter

These phases can either be executed sequentially or selectively, depending on the requirements. The Analyzer Phase is mandatory, as its output serves as the essential input for all subsequent stages and therefore cannot be omitted.

In the examined workflow, only the Analyzer and Reporter phases were executed. The Analyzer phase produces the file analyzer-result.yml Then the Reporter phase use this file as input file.

4.1.4.1 Ecosystem: Go

In this project, the following files were identified:

• docs/go.mod , go.mod , docs/package.json , internal/warpc/js/package.json

Indicating the use of two different Package Managers:

• GoMod & NPM

Note: The SBOM was successfully generated without any errors and without the need for any modifications or special configurations.

Generated SBOM: ORT-GO.spdx.json

4.1.4.2 Ecosystem: C (No package manager)

Given that ORT relies on a package manager for SBOM generation, and no package manager was present in this project, the generated SBOM only included the project name as a package.

Generated SBOM: ORT-C-NP.spdx.json

4.1.4.3 Ecosystem: C++ (No package manager)

In this project, no package manager associated with C++ was identified; However, in the two files listed below, NPM-related Packages were detected.

• gltf/package.json , js/package.json

Generated SBOM: ORT-C++-NP.spdx.json

4.1.4.4 Ecosystem: Node.js

In this project, the following file was identified:

• Package.json

indicating the use of a package manager:

• NPM

Note: The SBOM was successfully generated without any errors and without the need for any modifications or special configurations.

Generated SBOM: ORT-Node-js.spdx.json

4.1.4.5 Ecosystem: Java (Maven-managed)

During the process, an error related to the Maven compiler was encountered, which necessitated modifications in the pom.xml file as described below:

    <artifactId>maven-compiler-plugin</artifactId>

    <version>3.12.1</version>

    <configuration>

      <source>1.8</source>

      <target>1.8</target>

In addition, the following three files were identified:

• platform/pom.xml , pom.xml , samples/pom.xml

Indicating that the project relied on a single package manager:

• Maven

After the version was corrected, ORT was ultimately able to generate the SBOM successfully without errors.

Generated SBOM: ORT-Java-Maven.spdx.json

4.1.4.6 Ecosystem: C++ (Conan)

In this section, several points should be highlighted.

• First, since ORT encountered errors with Conan versions 2.x, it was necessary to downgrade Conan to version 1.66.0.

• Second, because a Linux-based operating system was used, one of the viable approaches for installing Conan involved the creation of a conan-venv, which was adopted in this case.

It should also be noted that the project included Bazel; therefore, Bazel(version:8.3.1) and Buildizer(version:1.5.0) were installed, as ORT would otherwise have failed with errors.

• Finally, the following files were identified:

• MODULE.bazel , .conan/test_package/conanfile.py , Conanfile.py

and in total, two package managers were detected:

• Bazel & Conan

Generated SBOM: ORT-C++-CONAN.spdx.json

4.1.4.7 Ecosystem: Python (FastAPI)

Since the project utilized the PDM package manager, which was not included in the list of package managers supported by ORT, no SBOM was generated as a result.

4.1.4.8 Ecosystem: Python (GPT Engineer)

In this project, the following two files were identified:

• projects/example-improve/requirements.txt , poetry.lock

Ultimately two package managers were utilized:

• PIP & Poetry

The SBOM was successfully generated without any errors.

Generated SBOM: ORT-python.spdx.json

---

4.2 Container-Based SBOM Generation Tools

In the selected project, the container image was based on a Linux distribution. Therefore, in order to properly execute the image and generate its corresponding SBOM, all tasks were carried out within a Linux operating system environment.

4.2.1 Distro2SBOM

Observations:

In order to generate an SBOM using distro2sbom, direct access to the root filesystem is required. Since running the tool against the live system root may not always be feasible or safe, the system root was first exported into a separate directory. This exported filesystem served as an isolated input for the SBOM generation process.

By performing the export step, distro2sbom was able to analyze the complete set of installed packages and system files, ensuring that the resulting SBOM accurately represents the environment. After the export was completed, the tool was executed against the target directory, successfully producing the SBOM.

Command:

• Extract image:

  docker create --name tmp <image name>
  

• Export container filesystem:

  docker export tmp -o <fileName.tar>
  

• Extract the tar file into root filesystem:

  tar -C <directoryName>  -xf <fileName.tar>
  

• Remove the temporary container:

  docker rm tmp
  

• For Generating SBOM:

  distro2sbom --root <path-to-rootFileSystem> -s --sbom <spdx|cyclonedx> --format <json|xml|yaml> -o <path-to-output-file>
  

  Generated SBOM: distro2SBOM.spdx.json

Note: Since a Linux environment was used, a dedicated Python virtual environment was created to ensure isolation and reproducibility. In this environment, the distro2sbom tool was installed as the main tool for generating SBOM.

4.2.2 Tern

There are three main approaches to generating an SBOM with Tern:

1. Analyzing an exported root filesystem: In this method, the system’s root filesystem is exported into a dedicated directory. Tern is then executed against that directory to analyze all installed packages and generate the SBOM. Commands:

docker export $(docker create <image name>) | tar -C  </path/to/directory> -xvf –

tern report -l </path/to/directory>  -f spdxjson -o <path/to/output/name.spdx.json>

Generated SBOM: tern.spdx.spdx.json

2.⁠ ⁠Parsing a Dockerfile: Tern can process a Dockerfile directly to infer the layers and dependencies defined in the build instructions. Commands:

tern report -d Dockerfile -o <path/to/output/name.spdx.json> -f json

Generated SBOM: tern-docker.spdx.json

3.⁠ ⁠Directly analyzing a container image: (not used in this project) Tern can be pointed to a container image (local or remote) to generate an SBOM without needing the Dockerfile or exported filesystem.

Commands:

tern report -i <image:tag> > sbom_image.json

4.2.3 Syft

In this project, two approaches were applied to generate SBOM using Syft:

1. Image-based SBOM generation: - A Docker image was built locally from the project’s Dockerfile. - Syft was then executed against the built image (pggb:latest) to produce an SBOM that reflects all installed packages and dependencies inside the container environment.

Commands:

docker build -t <image:tag> .

docker run --rm -it <image:tag> bash

syft <image:tag>  -o spdx-json > <out put name.json>

Generated SBOM: syft.spdx.json

2. Source-based SBOM generation: - Instead of analyzing the image, Syft was run directly on the project’s source code directory (./). - This method inspects the manifests and dependency files present in the source to generate the SBOM.

Commands:

syft dir:./ -o spdx-json > <out put name.json>

Generated SBOM: syft-source.spdx.json

---

5. Conclusion

5.1. Ecosystem Perspective

5.1.1 General Ecosystems:

Ecosystems were categorized based on package manager maturity, which directly impacted SBOM completeness and accuracy in the test materials analyzed.

5.1.1.2 Mature, Rich, and Well-Integrated Ecosystems:

• Java(Maven): Building the project ensured all direct and transitive dependencies were captured, resulting in a complete SBOM.

• Node.js(npm): Preparing a production-only environment allowed SBOM tools to accurately capture runtime dependencies.

• Python(Poetry): Installing production dependencies produced richer SBOMs with full dependency resolution.

• Go: Tidying modules and including vendored dependencies ensured a complete, reproducible dependency graph.

Note: All have strong version resolution, standard registries, and are SBOM-ready.

5.1.1.3 Functional but Inconsistent / Intermediate:

• Python(PDM): Even after dependency preparation, SBOMs remained limited, indicating partial compatibility with tools like Trivy and SCANOSS.

Note: Good dependency management and lockfile support, but limited ecosystem maturity and metadata consistency.

5.1.1.4 Fragmented or Build-System-Dependent:

• C/C++(manual/CMake): SBOMs captured only source files; dependency metadata was largely absent.

• C++(Conan): Dependency structure improved, but SBOM completeness varied due to inconsistent metadata.

Note: Conan adds structure but depends on how developers define packages (no universal lockfile standard, no uniform registry metadata.

5.1.2 Container Ecosystem:

• Container images based on Linux distributions rely on the completeness of installed system packages and layers.

• Tools like Distro2SBOM, Tern, and Syft showed that SBOM richness depends on access to filesystem metadata and proper image preparation.

5.1.3 Key Finding:

SBOM richness correlates strongly with package manager maturity and standardization. Tools like Syft and Trivy performed best on Maven, npm, Poetry, and Go projects, while Python (PDM) and C/C++ projects had less complete results. SCANOSS required additional configuration (--dependencies) to detect more dependencies, but SBOMs for Python (PDM) remained less comprehensive. Container-based tools performed best when images were fully built and all layers were accessible.

---

5.2. Tools Perspective

Tool Comparison

Tools	Strengths (Pros)	Limitations (Cons)
Syft	• Fast and lightweight CLI • Good ecosystem coverage • Detects dependencies accurately when lockfiles exist • Works for both container and application scanning	• May miss dependencies if lockfiles are missing • Limited license detection for source-only projects
Trivy	• Detects dependencies accurately when lockfiles exist • Strong integration between SBOM and vulnerability scanning	• SBOM less detailed for pure source projects • May miss dependencies if lockfiles are missing
SCANOSS	• Works without build or lockfiles • Detects components directly from source code • Produces high-quality SBOMs (aligned with OpenChain Telco standard) • Excellent ecosystem coverage	• Weak dependency graph for Python Poetry • CLI requires a two-step process (raw JSON → SPDX) and often manual compilation steps for more accurate results
ORT (OSS Review Toolkit)	• Fully automated analysis without building projects • Detects all package manager files within supported ecosystems • Capable of scanning and identifying known vulnerabilities	• Does not support projects without a package manager • Only generates SPDX version 2.2 • Some package managers (e.g., Python PDM) not fully supported or produce errors • Compatibility issues with newer versions of certain package managers
Distro2SBOM (Container)	• Accurate SBOM from full Linux root • Captures all installed packages and system files • Supports offline analysis using exported root	• Requires access to root filesystem or exported copy • Not suitable for live system scanning without export • Limited integration with container image layers
Tern (Container)	• Multiple analysis approaches: exported rootfs, Dockerfile parsing, or direct image scanning • Good insight into layer-level dependencies in images • Supports automated image inspection pipelines	• Exported filesystem approach can be slow • Direct image analysis not always used, limiting flexibility • Requires Dockerfile or rootfs for accurate SBOMs
Syft (Container)	• Works on images & source directories • Fast and lightweight CLI • Detects dependencies and licenses in images accurately • Integrates well with CI/CD pipelines	• May miss packages if metadata is incomplete in source-based scans • License detection limited for complex or custom images • Requires image building or source prep for best results

---

5.3 Quality Perspective

Validation results and differences

Validation Results

The OpenChain Telco SBOM Validator is a Python-based tool used to verify SBOMs for compliance with the OpenChain Telco SBOM Guide (versions 1.0 or 1.1). It ensures that generated SBOMs meet mandatory and optionally recommended metadata requirements defined by the NTIA and OpenChain specifications.

To install the validator this command was used:

pip install openchain-telco-sbom-validator==0.3.0

Key Features (v0.3.0):

• Supports recursive validation of SBOMs linked via SPDX Relationships.

• Allows validation against Guide version 1.0 or 1.1 (default: 1.1).

• Enables checks for mandatory fields only or both mandatory and recommended fields.

To run the validator for each generated SBOM these commands were used:

python3 -m openchain_telco_sbom_validator.cli  path/generated-sbom.json   > validation_output.txt

openchain-telco-sbom-validator --guide-version 1.1 path/generated-sbom.json > validation_output.txt

The results were summarized in the following tables:

5.3.1 General-Purpose Tools

5.3.1.1 Syft

Ecosystem	Compliant	Error Types	Notes
Go, Python, Node.js, Java (Maven), C, C++ (Conan), C++, Python2	Not compliant	NTIA validation errors, Missing mandatory fields	All packages missing version, supplier, and CreatorComment; NTIA validation errors

Note: For Syft, both default and enriched SBOMs exhibited the same validation issues across all ecosystems. All packages were consistently non-compliant due to missing mandatory fields, including package version, supplier, and CreatorComment, resulting in NTIA validation errors. This uniform behavior indicates that the issue is systemic rather than specific to any single ecosystem.

5.3.1.2 Trivy

Ecosystem	Compliant	Error Types	Notes
Go, Python, Node.js, Java (Maven), C, C++ (Conan), C++, Python2	Not compliant	NTIA validation errors, Missing mandatory fields	All packages missing version, supplier, and CreatorComment; NTIA validation errors

Note: For Trivy, all generated SBOMs—across default, lockfile-based, and vendor-tidy formats—showed consistent validation failures for every ecosystem. The non-compliance was primarily due to missing critical metadata, specifically package version, supplier, and CreatorComment, which triggered NTIA validation errors. This consistent pattern suggests that the validation issues stem from the way Trivy constructs SBOMs, rather than being caused by ecosystem-specific data.

5.3.1.3 ORT

Ecosystem	Compliant	Error Types	Notes
Go	Not compliant	NTIA validation errors, Missing mandatory fields	Packages missing supplier; CreatorComment and Organization fields in CreationInfo missing
Python	-	-	No SBOM generated because PDM is not supported by ORT
Node.js	Not compliant	NTIA validation errors, Missing mandatory fields	Missing supplier and CreatorComment, causing NTIA validation errors
Java_Maven	-	Invalid SPDX file	The generated JSON is not recognized as a valid SPDX file by the validator, no validation could be performed
C	-	-	Only the project itself was included as a package
C++ (Conan)	Not compliant	NTIA validation errors, Missing mandatory fields	Multiple packages missing version or supplier; CreatorComment and Organization fields in CreationInfo missing
C++	-	-	Only the project itself and two NPM packages were included as packages
Python2	Not compliant	SPDX validation errors, NTIA validation errors, Missing mandatory fields	Several referenced SPDX IDs not found; packages missing supplier info and CreatorComment; unsupported PDM package manager

Note: ORT-generated SBOMs revealed recurring issues across ecosystems, including missing supplier information, CreatorComment, and Organization fields in CreationInfo. In some cases, no SBOM was created or the JSON was rejected as invalid SPDX, because ORT relies on a supported package manager to generate SBOMs.

5.3.1.4 SCANOSS

Ecosystem	Compliant	Error Types	Notes
Go, Python, Node.js, Java (Maven), C, C++ (Conan), C++, Python2	Compliant	None	All SBOMs were fully compliant with the OpenChain Telco SBOM Guide v1.1, with no validation errors detected.

Note: All SBOMs generated by ScanOSS were fully compliant with the OpenChain Telco SBOM Guide v1.1, showing strong consistency and reliability across all tested ecosystems.

5.3.2 Container-Based SBOM Generation Tools

5.3.2.1 Distro2SBOM

Ecosystem	Compliant	Error Types	Notes
Container Image	Not compliant	Missing mandatory field in CreationInfo	Missing Organization in CreationInfo (mandatory per v1.1)

5.3.2.2 Tern

Ecosystem	Compliant	Error Types	Notes
Container Image	Not compliant	File error, NTIA validation errors, Missing mandatory fields	File missing; missing supplier, CreatorComment, and Organization fields

5.3.2.3 Syft

Ecosystem	Compliant	Error Types	Notes
Container Image	Not compliant	NTIA validation errors, Missing mandatory fields	File missing or packages without version/supplier; missing CreatorComment and Organization fields

Notes: - NTIA validation errors: indicate missing required metadata fields such as version, supplier, or license. - Missing mandatory fields: generally refer to incomplete CreationInfo (e.g., CreatorComment, Organization) or missing package-level metadata. - Compliant: The SBOM meets all required specifications of the OpenChain Telco SBOM Guide version 1.1, including mandatory fields and SPDX/JSON format compliance. - Not compliant: The SBOM fails to meet one or more required specifications, such as missing mandatory fields, invalid SPDX format, or incomplete package metadata.

5.3.3 Diffs / Comparisons

The sbomdiff | v0.5.6 is a Python-based tool used to compare two SBOM files and identify differences between them. It supports both SPDX and CycloneDX formats and detects:

• Package version changes
• Added or removed packages
• License differences

Installation and usage: SBOMDiff was installed using:

pip install sbomdiff

Each comparison was performed between the reference SBOM (exported from GitHub dependency graph) and the generated SBOM from each tool.

The command used for comparison:

sbomdiff -f json -o name.json reference.json generated.json

Note: sbomdiff requires Python 3.7+ (in this project this version was used: 3.12.9). Using a virtual environment (python3 -m venv venv && source venv/bin/activate) is recommended to isolate dependencies.

The tool outputs differences in package versions, new or missing packages, and license changes.

• Version_Changes: packages exist in both SBOMs but have different version numbers.

• New_Packages: packages are only in the generated SBOM (file 2).

• Removed_Packages: packages are only in the reference SBOM (file 1).

• License_Changes: packages appear in both SBOMs but with different license declarations.

The results were summarized in the following tables:

5.3.3.1 Syft

Ecosystem	Compared Files	Difference file	Version Changes	New Packages	Removed Packages	License Changes
C	Ref-C.json vs C.json	Diff-C.json	7	1	1	7
C++ (Conan)	Ref-CONAN.json vs CONAN.json	Diff-CONAN.json	3	1	1	3
Python	Ref-python.json vs python.json , python-comp.json	Diff-python.json , Diff-python-comp.json	17 / 35	2 / 149	19 / 1	33 / 51
Node.js	Ref-Nodejs.json vs Nodejs.json , Nodejs-comp.json	Diff-Nodejs.json , Diff-Nodejs-comp.json	9 / 36	1 / 40	44 / 17	9 / 36
Java	Ref-Java.json vs Java.json , Java-comp.json	Diff-Java.json , Diff-Java-comp.json	1	42	43	4
Go	Ref-Go.json vs Go.json	Diff-Go.json	177	6	13	177
C++	Ref-C++.json vs C++.json	Diff-C++.json	3	1	1	5
Python2	Ref-python2.json vs python2.json , python2-comp.json	Diff-python2.json , Diff-python2-comp.json	9	1	1	198

5.3.3.2 Trivy

Ecosystem	Compared Files	Differences file	Version Changes	New Packages	Removed Packages	License Changes
C	Ref-C.json vs C.json	Diff-C.json	0	1	8	0
C++ (Conan)	Ref-CONAN.json vs CONAN.json	Diff-CONAN.json	0	1	4	0
Python	Ref-python.json vs python.json , python-comp.json	Diff-python.json , Diff-python-comp.json	0 / 4	1 / 7	52 / 48	0 / 4
Node.js	Ref-Nodejs.json vs Nodejs.json , Nodejs-comp.json	Diff-Nodejs.json , Diff-Nodejs-comp.json	0 / 0	1 / 67	47 / 47	0 / 0
Java	Ref-Java.json vs Java.json	Diff-Java.json	13	5	15	18
Go	Ref-Go.json vs Go.json	Diff-Go.json	166	7	24	166
C++	Ref-C++.json vs C++.json	Diff-C++.json	0	1	6	0
Python2	Ref-python2.json vs python2.json	Diff-python2.json	2	3	9	190

5.3.3.3 ORT

Ecosystem	Compared Files	Differences file	Version Changes	New Packages	Removed Packages	License Changes
C	Ref-C.json vs C.json	Diff-C.json	0	1	8	0
C++ (Conan)	Ref-CONAN.json vs CONAN.json	Diff-CONAN.json	0	24	4	0
Node.js	Ref-Nodejs.json vs Nodejs.json	Diff-Nodejs.json	35	288	10	43
Java	Ref-Java.json vs Java.json	Diff-Java.json	0	44	47	0
Go	Ref-Go.json vs Go.json	Diff-Go.json	8	62	21	169
C++	Ref-C++.json vs C++.json	Diff-C++.json	0	2	6	0
Python2	Ref-python2.json vs python2.json	Diff-python2.json	131	7	19	180

5.3.3.4 SCANOSS

Ecosystem	Compared Files	Differences file	Version Changes	New Packages	Removed Packages	License Changes
C	Ref-C.json vs C.json , C-comp.json	Diff-C.json , Diff-C-comp.json	0 / 0	2 / 17	8 / 8	0 / 0
C++ (Conan)	Ref-CONAN.json vs CONAN.json	Diff-CONAN.json	0 / 0	16 / 17	4 / 4	0 / 0
Python	Ref-python.json vs python.json , python-comp.json	Diff-python-comp.json , Diff-python-comp.json	0 / 4	3 / 29	52 / 48	0 / 4
Node.js	Ref-Nodejs.json vs Nodejs.json , Nodejs-comp.json	Diff-Nodejs.json , Diff-Nodejs-comp.json	0 / 20	14 / 54	53 / 33	0 / 20
Java	Ref-Java.json vs Java.json , Java-comp.json	Diff-Java.json , Diff-Java-comp.json	0 / 0	1 / 9	47 / 47	0 / 0
Go	Ref-Go.json vs Go.json , Go-comp.json	Diff-Go.json , Diff-Go-comp.json	0 / 0	15 / 352	190 / 190	0 / 0
C++	Ref-C++.json vs C++.json , C++-comp.json	Diff-C++-comp.json , Diff-C++-comp.json	0 / 0	8 / 16	6 / 6	0 / 0
Python2	Ref-python2.json vs python2.json	Diff-python2.json	0	2	199	0

*Note: Each pair of numbers (e.g., 2* / 17*) represents default / post-compilation results from the difference files, where the first value corresponds to the default SBOM and the second to the post-compilation SBOM.

Key considerations:

• The raw numbers in the difference tables do not by themselves indicate whether a tool performed correctly. Minor version differences (e.g., v1.2.3 vs 1.2.3) or licenses reported as NOASSERTION can inflate counts. However, newly added or removed packages are more significant and often provide clearer insight into a tool’s detection accuracy. To truly assess a tool’s accuracy and compatibility, the difference files must be examined to understand the nature of each change.

• Results vary across tools and ecosystems, reflecting differences in dependency detection, license resolution, and metadata handling.

5.3.3.5 distro2SBOM

Compared Files	Differences file	Version Changes	New Packages	Removed Packages	License Changes
Ref-Container.json vs distro.spdx.json	Diff-distro.json	0	363	7	0

5.3.3.6 Tern

Compared Files	Differences Found	Version Changes	New Packages	Removed Packages	License Changes
Ref-Container.json vs tern.spdx.json , tern-docker.json	Diff-tern.json , Diff-tern-docker	0 / 0	362 / 127	7 / 7	0 / 0

5.3.3.7 Syft

Compared Files	Differences Found	Version Changes	New Packages	Removed Packages	License Changes
Ref-Container.json vs syft-container.json , syft-source-container.json	Diff-syft-container.json , diff-syft-source-container.json	0 / 2	485 / 1	7 / 5	0 / 2

Note: Each pair of values (e.g., 0* / 2*) represents results from two phases — the first value corresponds to the SBOM generated in the first phase (default/root filesystem or image-based), and the second value corresponds to the second phase (Dockerfile-based or source-based) analysis.