Skip to main content

File storage

File storage is a persistent data storage solution that remains available across instance launches and terminations. It provides a reliable way to maintain your data independently of compute instances.

Overview​

Our file storage system offers virtual file systems that can be used to store:

  • Large datasets and model weights
  • Source code repositories
  • Configuration files
info

Jarvislabs supports file storage volumes of up to 2TB, allowing you to scale your storage needs as your projects grow. If you need more storage, please contact us at hello@jarvislabs.ai.

Getting Started with File Storage​

Creating and Attaching Storage​

  1. When launching a new instance, you can create and attach file storage during the instance configuration.
  2. You can also create a file storage here and attach it to an instance later.
  3. Select your desired storage capacity (up to 2TB).
  4. Once your instance launches, you'll find a mounted directory named 'jl_fs' in your instance.

Using File Storage​

  • The file storage is mounted to the /home/jl_fs directory in your instance.
  • All data stored in the /home/jl_fs directory is automatically persisted to your file storage volume
  • The same file storage can be attached to multiple instances, enabling data sharing
note

File storage can be used only if the instance is in the same region.

Common Use Cases​

  • Training Data: Store and access large datasets across multiple training runs
  • Model Artifacts: Save trained models and checkpoints
  • Collaboration: Share data and code between team members using different instances

Pricing and Billing​

File storage is billed based on the provisioned capacity:

  • Rate: $0.00014/GB/Hour (approximately $0.1008/GB/month)
  • Example: 500GB storage for one month (720 hours) = $50.40

Important Notes:

  • Pricing is for the allocated capacity, regardless of actual usage
  • Storage size can only be increased, not decreased
  • Current pricing will be displayed during file system creation

Best Practices for File Storage​

File storage system is a distributed file system that provides excellent scalability and reliability. Here are some best practices to optimize your usage:

  1. Large Files vs Small Files

    • Performs best with larger files (>1MB)
    • Avoid storing many small files (few KB) in a single directory
    • Consider combining small files into archives (tar, zip) when possible

  2. Directory Organization

    • Keep the number of files per directory under 1000 for optimal performance
    • Use subdirectories to organize large collections of files
    • Avoid deep directory hierarchies (keep under 8 levels when possible)

  3. Performance Tips

    • For read-heavy workloads, consider using /home temporary files
    • Avoid running intensive file operations on multiple instances simultaneously

  4. Data Management

    • Regularly clean up unnecessary files to optimize storage usage
    • Use .gitignore-style files when working with git repositories
    • Back up critical data to external storage periodically

Note: While File storage is highly reliable, it's designed for general-purpose storage. For workloads requiring extremely high IOPS, consider using /home storage instead.

Troubleshooting​

Common issues and solutions:

  1. Slow Performance

    • Check if you're following the best practices for file organization
    • Consider using home for temporary files and intensive I/O operations

  2. Storage Full

    • Regular cleanup of unnecessary files
    • Consider increasing storage capacity if needed

For additional support, please contact our support team through the dashboard.