EDITO uses S3-compatible object storage for personal and project data.

Depending on your context, the storage you want to access may be:

The commands in this article use the same syntax for all of these cases.

This article applies when you are working outside a running EDITO service, for example from:

Inside a running EDITO service such as Jupyter, the selected S3 configuration and credentials are normally injected automatically and the MinIO Client alias s3 is already configured. See Manage files in EDITO Storage and Access EDITO Storage from code, services and processes for that use case.

To keep all commands independent of the storage provider, the examples below use three values:

The generic remote path is therefore:

For shell commands, it can be convenient to define:

All transfer commands can then use $REMOTE_ROOT, regardless of where the bucket is hosted.

The default personal storage is hosted on EDITO MinIO.

Typical values are:

You might also need:

Get the credentials and exact values from My Account → Connect to storage.

This section displays the storage variables for your personal bucket and lets you copy or download an initialization script in a supported programming language. The Datalab also exposes S3 snippets for several languages and libraries, so you can use the client that best fits your workflow.

Check the current project's configuration in Project Settings → S3 Configurations:

For an additional or external bucket, use the values supplied by that provider or displayed in Project Settings:

Select the relevant project, then open Project Settings → S3 Configurations:

Use the endpoint and working-directory path shown in the project configuration.

For example:

If your project requires a lot of storage, you can request that user support increase your quota. In that case, the team will set up a bucket for your project that is hosted directly by the cloud provider, CloudFerro.

Select the project and open Project Settings → S3 Configurations:

The configuration may look conceptually like:

The exact endpoint and path depend on the project configuration.

A project may have more than one S3 configuration. The configuration with Use for EDITO explorers enabled is the one displayed by default in File Explorer, but the other configurations can still represent valid storage targets.

For an external bucket managed outside EDITO, use the values supplied by the storage provider:

Credentials are provided by the external storage provider.

Install the MinIO Client command-line tool, named mc, on your computer or cluster.

​

After installation, confirm that it is available:

Use the installation method appropriate to your operating system or cluster environment. On a managed HPC platform, you may need to install it in your home directory, use a module or ask the platform administrator.

The endpoint identifies the S3 service. The alias is the local name that mc will use to refer to it.

Once the alias is configured, the endpoint no longer appears in the transfer commands via:

How the alias is configured depends on whether the credentials include a session token:

Temporary EDITO credentials normally contain:

mc alias set only takes the endpoint, access key and secret key in its basic syntax. For credentials requiring a session token, configure an MC_HOST_<ALIAS> environment variable instead.

For example, for:

Configure:

The suffix of MC_HOST_<ALIAS> is the alias used by subsequent commands:

And:

Set the alias with:

Then verify the connection:

And your configured storage root:

As explained above, before transferring a large amount of data, test that the remote storage is accessible:

If this fails, check:

All commands below are identical for personal, project, CloudFerro and external storage.

Only ALIAS and WORKING_DIRECTORY_PATH change.

All files are stored under the destination prefix.

For repeated transfers, use mc mirror:

mc cp displays a progress indicator for transfers.

​

​Large files are uploaded using multipart transfer automatically; no additional multipart configuration is normally required.

​

mc cat is useful for a small configuration or text object, or for piping its content into another tool:

mc can also transfer objects directly between two configured S3 endpoints.

This works regardless of whether the source and destination are hosted on EDITO MinIO, CloudFerro or another S3-compatible provider.

For example:

For a complete prefix/folder:

Or synchronize them with:

Both aliases must be configured with credentials that allow the required operation.

For large prefixes, remember that object storage may implement a move as a copy followed by deletion.

Preview the operation first:

Then, if the target is correct:

Libraries such as s3fs, fsspec, xarray and Zarr can read supported formats directly from private S3 storage.

​

This can reduce data transfer when the format supports partial or chunked access and you only need a subset of a large dataset.

​

For the canonical authenticated xarray and s3fs examples, see Access EDITO storage from code, services and processes.

​

Because mc is entirely command-line based, the same commands can be used in SSH sessions, SLURM jobs or other automated workflows.

Before running a large transfer from compute nodes:

Use boto3 when the transfer is part of an existing Python pipeline.

boto3 does not provide a direct one-command equivalent of mc mirror for a whole local directory. To synchronize a directory, iterate through the local files, build each object key and call upload_file, or use a dedicated synchronization tool.

​

For in-memory uploads and additional Python examples, see Access EDITO storage from code, services and processes.

​

Check that the credentials have permission to access the requested bucket and prefix. Credentials for one personal space, project or provider may not be valid for another.

Also check that <WORKING_DIRECTORY_PATH> points to the path accessible with those credentials.

Obtain a new temporary credential set, including a new session token, then recreate the MC_HOST_<ALIAS> variable or Python client.

Check the bucket name and whether the value shown in Project Settings is a bucket or a working-directory path inside a bucket.

Use the exact value supplied by the storage configuration rather than guessing it from the project name.

​

Check <ENDPOINT>, including the URL scheme.

For the default EDITO MinIO storage: https://minio.dive.edito.eu

Also check local proxy, firewall, DNS and outbound-network rules.

HPC login nodes and compute nodes may have different network-access policies.

S3 object keys should not normally begin with /. Compare the source and destination paths carefully and remember that a trailing / changes how some recursive copy commands construct the destination keys.

For operations performed inside an EDITO service, see:

For configuring another S3 provider in EDITO, see:

For project-specific storage, see:

If you have any questions, problems, or suggestions, please feel free to contact us via chat using the widget available at the bottom right of the page.