Pluggable file systems in Java

Java 7 (2011) brought with NIO.2 a major update to the Input / Output system. It introduced the classes java.nio.file.Path and java.nio.file.Files to the runtime and decoupled the file operations from the underlying native file system.

The java.io.File class that exists since Java 1.0 is still available. With Java 7, the implementation of this class changed and used the new file system abstraction as well. But java.io.File only supports the default implementation that talks to the native platform file system.

When your code calls a file operation on Path and Files, they do not directly "talk" to the native file system. Instead, they delegate the call to an implementation of the class java.nio.file.FileSystem. The runtime uses by default an implementation that talks with the native file system on which the Virtual Machine is running on.

You see that when you look at the source code for Paths.get():

    public static Path get(String first, String... more) {
        return FileSystems.getDefault().getPath(first, more);
    }

getDefault() returns the java.nio.file.FileSystem implementation for the native file system.

Here a simple example with a bunch of file operations. All these operations are executed in the current directory where the Java program resides.

    Path p1 = Paths.get("one.txt");
    List<String> lines = Arrays.asList("one", "two", "three");
    Files.write(p1, lines, StandardCharsets.UTF_8);

    Path p2 = Paths.get("two.txt");
    Files.copy(p1, p2, StandardCopyOption.REPLACE_EXISTING);

    Path p2b = Paths.get("two_renamed.txt");
    Files.move(p2, p2b, StandardCopyOption.REPLACE_EXISTING);

    Path dir = Paths.get("directory");
    Files.createDirectory(dir);

    Path p3 = Paths.get("directory/three.txt");
    Files.write(p3, Arrays.asList("three"));

    Files.delete(p1);

Default.java

Zip ¶

Besides the default file system implementation, the Java runtime provides one additional implementation: ZipFileSystem

With this file system, a program can "mount" a zip file and do file operations as if it is a regular file system.

The following example shows you how to use this file system. FileSystems.newFileSystem instantiates a new filesystem. We have to provide a URI and a configuration object. The URI points to a zip file. The only option that ZipFileSystem supports is create; when true, it creates the zip file when it does not exist.

Java automatically selects the ZipFileSystem because it is internally registered to handle URIs with the jar: scheme. This code does the same as the previous example, but everything is contained in the test1.zip file. Note that the example calls getPath() from the FileSystem instance instead of Paths.get. Paths.get always references paths on the native file system.

    Map<String, String> env = new HashMap<>();
    env.put("create", "true");
    URI uri = URI.create("jar:file:/test1.zip");

    try (FileSystem fs = FileSystems.newFileSystem(uri, env)) {
      Path p1 = fs.getPath("one.txt");
      List<String> lines = Arrays.asList("one", "two", "three");
      Files.write(p1, lines, StandardCharsets.UTF_8);

      Path p2 = fs.getPath("two.txt");
      Files.copy(p1, p2, StandardCopyOption.REPLACE_EXISTING);

      Path p2b = fs.getPath("two_renamed.txt");
      Files.move(p2, p2b, StandardCopyOption.REPLACE_EXISTING);

      Path dir = fs.getPath("directory");
      Files.createDirectory(dir);

      Path p3 = fs.getPath("directory/three.txt");
      Files.write(p3, Arrays.asList("three"));

      Files.delete(p1);
    }

Zip1.java

Here another example that "mounts" an existing jar file and lists all entries.

    URI uri = URI.create("jar:file:///D:/ws/dacs/license-manager/target/license.jar");

    try (FileSystem fs = FileSystems.newFileSystem(uri, Collections.emptyMap())) {

      try (Stream<Path> paths = Files.walk(fs.getPath("/"))) {
        paths.filter(Files::isRegularFile).forEach(System.out::println);
      }
    }

Zip2.java

An application can work with multiple file systems at once. This example copies a file from one zip file to another.

    Map<String, String> env = new HashMap<>();
    env.put("create", "true");
    URI uri1 = URI.create("jar:file:/test1.zip");
    URI uri2 = URI.create("jar:file:/test2.zip");

    try (FileSystem fs1 = FileSystems.newFileSystem(uri1, Collections.emptyMap());
        FileSystem fs2 = FileSystems.newFileSystem(uri2, env)) {
      Path p1 = fs1.getPath("two_renamed.txt");
      Path p2 = fs2.getPath("copy.txt");
      Files.copy(p1, p2);
    }

Zip3.java

Cryptomator ¶

Cryptomator is a desktop application that encrypts files. It's especially useful for users that want to store private information on a cloud drive like Dropbox, Google Drive, or Microsoft OneDrive. When you start with Cryptomator, you configure an empty directory where the application stores the encrypted files. When you mount such a directory with Cryptomator, it provides a virtual hard drive through which the files can be accessed in unencrypted form. It's similar to VeraCrypt (former TrueCrypt), but instead of storing the files in one big container, Cryptomator encrypts and stores each file individually. You still see the directories and files in the file system, although directory and file names are encrypted as well.

The reason why I mention Cryptomator in this context is that the source code of the whole application is open and accessible from their GitHub account and one part of the application is the cryptofs library, a java.nio.file.FileSystem implementation.

To use the library, you add this dependency to your Maven or Gradle project. You don't have to install the Cryptomator desktop application. The library works independently.

<dependency>
  <groupId>org.cryptomator</groupId>
  <artifactId>cryptofs</artifactId>
  <version>1.9.2</version>
</dependency>

Before your application can use the file system, you need to prepare an empty directory on the native file system. The library provides the method CryptoFileSystemProvider.initialize for this purpose. Alternatively, you can create the directory with the Cryptomator desktop application.

After the initialization, the program can instantiate the file system with FileSystems.newFileSystem and use it like an ordinary file system.

    Path storageLocation = Paths.get("E:\\temp\\vault");

    Files.createDirectories(storageLocation);
    CryptoFileSystemProvider.initialize(storageLocation, "masterkey.cryptomator",
        "password");

    URI uri = CryptoFileSystemUri.create(storageLocation);

    try (FileSystem fs = FileSystems.newFileSystem(uri,
        CryptoFileSystemProperties.cryptoFileSystemProperties().withPassphrase("password")
            // .withFlags(FileSystemFlags.READONLY)
            .build())) {

      Path p1 = fs.getPath("/one.txt");
      List<String> lines = Arrays.asList("one", "two", "three");
      Files.write(p1, lines, StandardCharsets.UTF_8);

      Path p2 = fs.getPath("/two.txt");
      Files.copy(p1, p2, StandardCopyOption.REPLACE_EXISTING);

      Path p2b = fs.getPath("/two_renamed.txt");
      Files.move(p2, p2b, StandardCopyOption.REPLACE_EXISTING);

      Path dir = fs.getPath("/directory");
      Files.createDirectory(dir);

      Path p3 = fs.getPath("/directory/three.txt");
      Files.write(p3, Arrays.asList("three"));

      Files.delete(p1);

    }

Cryptomator.java

Memory ¶

Another interesting file system implementation is Jimfs from Google. Jimfs is an in-memory file system, and all the data is non-persistent. This is especially useful for testing, and you can specify how the file system should behave. Jimfs supports Windows, Unix, and OSX like file systems. You find the source code for this project on GitHub: https://github.com/google/jimfs

To use it, add this dependency to your Maven or Gradle project.

<dependency>
  <groupId>com.google.jimfs</groupId>
  <artifactId>jimfs</artifactId>
  <version>1.1</version>
</dependency>

Jimfs.newFileSystem is a factory method that simplifies the file system instantiation. This method calls FileSystems.newFileSystem under the hood.

    try (FileSystem fs = Jimfs.newFileSystem(Configuration.unix())) {
      Path p1 = fs.getPath("one.txt");
      List<String> lines = Arrays.asList("one", "two", "three");
      Files.write(p1, lines, StandardCharsets.UTF_8);

      Path p2 = fs.getPath("two.txt");
      Files.copy(p1, p2, StandardCopyOption.REPLACE_EXISTING);

      Path p2b = fs.getPath("two_renamed.txt");
      Files.move(p2, p2b, StandardCopyOption.REPLACE_EXISTING);

      Path dir = fs.getPath("directory");
      Files.createDirectory(dir);

      Path p3 = fs.getPath("directory/three.txt");
      Files.write(p3, Arrays.asList("three"));

      Files.delete(p1);

      try (Stream<Path> paths = Files.walk(fs.getPath("/"))) {
        paths.filter(Files::isRegularFile).forEach(System.out::println);
      }
    }

Memory.java

If you want to write your own java.nio.file.FileSystem implementation, check out this page
https://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/filesystemprovider.html