Import

Beginner

Tutorial

Overview

The interoperability of canisters is a vital feature for many developers. dfx provides a consistent developer workflow for creating, integrating and testing third-party canisters with local developer environments.

Third-party canisters include canisters created by DFINITY or by developers in ICP community. Developers depend on third-party canisters to integrate with. They typically need a way to develop and test the integrations locally for:

• Validating the accuracy of the integration and other canister code.
• Testing without paying cycles.
• Using non-production data and environments.
• Faster completion time when run locally.

To pull these canisters from the mainnet to be tested locally, the dfx deps command and workflow can be used.

In this workflow, a service provider configures a canister to be pullable, then deploys the canister to the mainnet. A service provider can be any community developer creating a public, third-party canister.

Then a service consumer can pull the canister as a dependency directly from the mainnet and then deploy the dependency locally.

Determining if a canister should be pullable

First the canister must be configured to be pullable. Developers must ask question whether the canister should be pullable?

Pullable examples:

If a canister is providing a public service at a static canister ID, then it makes sense for the canister to be pullable.

If a service canister depends on other canisters, those dependencies should also be pullable.

Non-pullable examples:

If the canister is meant for personal use and not intended for others, the canister should not be pullable.

If a canister's Wasm is published for other developers to use, then the canister should not be pullable since the canister ID of the instance is not static. Users can test integrations locally and deploy them directly using the Wasm file directly.

Service provider workflow

First, a service provider must configure a canister to be pullable by setting it as such in the dfx.json file.

An example of a provider dfx.json which has a pullable "service" canister can be found below:

{
    "canisters": {
        "service": {
            "type": "motoko",
            "main": "src/main.mo",
            "pullable": {
                "wasm_url": "http://example.com/a.wasm",
                "wasm_hash": "d180f1e232bafcee7d4879d8a2260ee7bcf9a20c241468d0e9cf4aa15ef8f312",
                "dependencies": [
                    "yofga-2qaaa-aaaaa-aabsq-cai"
                ],
                "init_guide": "A natural number, e.g. 10."
            }
        }
    }
}

The Wasm module of a pullable canister must be hosted via a public URL where service consumers can download it.

GitHub Releases are a good, free option if the project is open source on GitHub. The GitHub URL schema is:

https://github.com/<USERNAME>/<REPONAME>/releases/latest/download/<FILENAME>

The pullable object will be serialized as a part of the dfx metadata and attached to the Wasm.

To better understand the pullable object, let's look at each property in depth:

• wasm_url: A URL used to download the canister Wasm module which will be deployed locally.
• wasm_hash: A SHA256 hash of the Wasm module located at wasm_url. This field is optional. In most cases, the Wasm module at wasm_url will be the same as the onchain Wasm module. This means that dfx can read the state tree to obtain and verify the module hash. In some cases, the Wasm module at wasm_url is not the same as the onchain Wasm module. For example, the Internet Identity canister provides a development variant to be integrated locally. In these cases, wasm_hash provides the expected hash, and dfx verifies the downloaded Wasm against this.

If the wasm_hash of the Wasm module at wasm_url does not match, dfx will abort with an error message indicating that there is a hash mismatch. In this scenario, the service consumer should contact the service provider. It is the responsibility of the service provider to assure that the correct Wasm module can be downloaded from the wasm_url.

• dependencies: An array of canister IDs (Principal) of direct dependencies.
• init_guide: A message to guide consumers how to initialize the canister.

Canister metadata requirements

A service provider canister used in production or in a production environment running on the mainnet should have public dfx metadata and public or private candid:service and candid:args metadata.

All metadata sections are handled by dfx when the canister is built.

Deployment process

Service providers should use the following deployment process to deploy their pullable canister.

• Step 1: From within your project's repo, deploy the canister to the mainnet with the command:

dfx deploy <canister-name> --network ic

• Step 2: If you're using GitHub, git tag and GitHub release with the commands:

git tag 0.1.0
git push --tags

You can follow this guide to create a release.

• Step 3: Attach the Wasm to the release assets.

Edit the release and attach the deployed Wasm as a release asset. The deployed Wasm file will be located at:

.dfx/ic/canisters/<CANISTER_NAME>/<CANISTER_NAME>.wasm

Automating the service provider process in CI

An example CI configuration demonstrates how to use a GitHub Action to automate the deploy routine described above.

The workflow with CI will follow these steps:

1. Push a git tag and wait for the GitHub release to complete.
2. Download the canister Wasm from the release assets (wget https://github.com/lwshang/pullable/releases/latest/download/service.wasm).
3. Install (upgrade) the canister using the downloaded Wasm (dfx canister --network ic install service --wasm service.wasm --argument '(1 : nat)' --mode upgrade).

Consumer workflow

The following workflow can be used for consumers to pull a pullable canister as a dependency.

• Step 1: Declare "pull" dependencies in dfx.json.

First, the dfx.json file must include the dependencies configuration for the canister.

An example dfx.json in which the consumer is developing a canister named "dapp" that has two pull dependencies can be found below:

• "dep_b" has canister ID of yhgn4-myaaa-aaaaa-aabta-cai on the mainnet.
• "dep_c" has canister ID of yahli-baaaa-aaaaa-aabtq-cai on the mainnet.

{
    "canisters": {
        "dapp": {
            "type": "motoko",
            "main": "src/main.mo",
            "dependencies": [
                "dep_b", "dep_c"
            ]
        },
        "dep_b": {
            "type": "pull",
            "id": "yhgn4-myaaa-aaaaa-aabta-cai"
        },
        "dep_c": {
            "type": "pull",
            "id": "yahli-baaaa-aaaaa-aabtq-cai"
        }
    }
}

• Step 2: Pull the dependencies using the dfx deps pull command.

dfx deps pull connects to the mainnet by default (--network ic). You can choose other network as usual, e.g. --network local.

Running this command will do the following:

1. First, it will resolve the dependency graph by fetching the dependencies field in the dfx metadata recursively.
2. Then, it will download the Wasm of all direct and indirect dependencies from wasm_url into the shared cache.
3. Next, the hash of the downloaded Wasm will be verified against wasm_hash metadata or the hash of the canister deployed on mainnet.
4. Then, candid:args, candid:service, dfx metadata will be extracted from the downloaded Wasm.
5. The deps/ folder is created in the project root.
6. The candid:service of direct dependencies is saved as deps/candid/<CANISTER_ID>.did.
7. The deps/pulled.json which contains major info of all direct and indirect dependencies is saved.

For the example project, you will find the following files in deps/:

• yhgn4-myaaa-aaaaa-aabta-cai.did and yahli-baaaa-aaaaa-aabtq-cai.did: Candid files that can be imported by "dapp".
• pulled.json: A json file with the following content:

{
  "canisters": {
    "yofga-2qaaa-aaaaa-aabsq-cai": {
      "dependencies": [],
      "wasm_hash": "e9b8ba2ad28fa1403cf6e776db531cdd6009a8e5cac2b1097d09bfc65163d56f",
      "init_guide": "A natural number, e.g. 10.",
      "candid_args": "(nat)"
    },
    "yhgn4-myaaa-aaaaa-aabta-cai": {
      "name": "dep_b",
      "dependencies": [
        "yofga-2qaaa-aaaaa-aabsq-cai"
      ],
      "wasm_hash": "f607c30727b0ee81317fc4547a8da3cda9bb9621f5d0740806ef973af5b479a2",
      "init_guide": "No init arguments required",
      "candid_args": "()"
    },
    "yahli-baaaa-aaaaa-aabtq-cai": {
      "name": "dep_c",
      "dependencies": [
        "yofga-2qaaa-aaaaa-aabsq-cai"
      ],
      "wasm_hash": "016df9800dc5760785646373bcb6e6bb530fc17f844600991a098ef4d486cf0b",
      "init_guide": "A natural number, e.g. 20.",
      "candid_args": "(nat)"
    }
  }
}

In this file, you can see there are three dependencies:

• yhgn4-myaaa-aaaaa-aabta-cai: "dep_b" in dfx.json.

• yahli-baaaa-aaaaa-aabtq-cai: "dep_c" in dfx.json.

• yofga-2qaaa-aaaaa-aabsq-cai: An indirect dependency that both "dep_b" and "dep_c" depend on.

• Step 3: Set init arguments using dfx deps init

Running the command dfx deps init will iterate over all dependencies in the pulled.json file and set an empty argument for any that do not need an init argument. Then, it will print the list of dependencies that do require an init argument.

Running the command dfx deps init <CANISTER> --argument <ARGUMENT> will set the init argument for an individual dependency. The init arguments will be recorded in deps/init.json.

Using the example above, you can run the following commands:

• To set the init arguments:

dfx deps init

Output

WARN: The following canister(s) require an init argument. Please run `dfx deps init <NAME/PRINCIPAL>` to set them individually:
yofga-2qaaa-aaaaa-aabsq-cai
yahli-baaaa-aaaaa-aabtq-cai (dep_c)

• If you try to set an init argument for an individual dependency without an argument, it will result in the following error:

Error: Canister yofga-2qaaa-aaaaa-aabsq-cai requires an init argument. The following info might be helpful:
init_guide => A natural number, e.g. 10.
candid:args => (nat)

• To set an init argument with an argument using the --argument flag, the following commands can be used:

dfx deps init yofga-2qaaa-aaaaa-aabsq-cai --argument 10
dfx deps init deps_c --argument 20

The resulting generated file init.json will have the following content:

{
  "canisters": {
    "yofga-2qaaa-aaaaa-aabsq-cai": {
      "arg_str": "10",
      "arg_raw": "4449444c00017d0a"
    },
    "yhgn4-myaaa-aaaaa-aabta-cai": {
      "arg_str": null,
      "arg_raw": null
    },
    "yahli-baaaa-aaaaa-aabtq-cai": {
      "arg_str": "20",
      "arg_raw": "4449444c00017d14"
    }
  }
}

• Step 4: Deploy the pulled dependencies on a local replica using the dfx deps deploy command.

Running the dfx deps deploy command will:

1. First, create the dependencies on the local replica with the same mainnet canister ID.
2. Then, it will install the downloaded Wasm with the init arguments in the init.json file.

You can also specify the canister name or principal to deploy one particular dependency.

Using the example above, you can run the following command to deploy all dependencies:

dfx deps deploy

Output

Creating canister: yofga-2qaaa-aaaaa-aabsq-cai Installing canister: yofga-2qaaa-aaaaa-aabsq-cai Creating canister: yhgn4-myaaa-aaaaa-aabta-cai (dep_b) Installing canister: yhgn4-myaaa-aaaaa-aabta-cai (dep_b) Creating canister: yahli-baaaa-aaaaa-aabtq-cai (dep_c) Installing canister: yahli-baaaa-aaaaa-aabtq-cai (dep_c)

dfx deps deploy always creates the canister with the anonymous identity so that dependencies and application canisters will have different controllers. It will also always install the canister in "reinstall" mode so that the canister status will be discarded.

Frequently asked questions

• Why download the Wasm into shared cache instead of a project subfolder?

It is not encouraged to include binary files in version control. On the Internet Computer, every canister only has one latest version running on mainnet. Service consumers should integrate with that latest version.

dfx deps pull always gets the latest dependencies instead of locking on a particular run. Every pulled canister has the latest version in the shared cache and can be reused by different projects.

• Should I include deps/ folder in version control?

Yes. deps/ files enable the dependent canister to build and get IDE support. If the required Wasm files are also available in the shared cache, all application and dependencies can be deployed and tested integrally.

Considering a canister developer team:

1. Developer 1 follows the service consumer workflow and includes all generated deps/ files in source control.
2. Developer 2 pulls the branch by Developer 1 and runs the dfx deps pull command again.
   • If the pulled.json has no change, then all dependencies are still up to date. Developer 2 can run dfx deps deploy without setting init arguments again.
   • If there are changes in pulled.json, Developer 2 can try to run dfx deps deploy to see if all init arguments are still valid. Then Developer 2 can run dfx deps init if necessary and update source control.

These files also help CI to detect outdated dependencies.

Resources

• dfx deps CLI reference.
• GitHub example project.
• Pull dependencies concept documentation.