Working with MongoDB TTL Indexes from Java

In the previous blog post, we have seen how to leverage the ability of MongoDB to delete documents based on the size with Capped Collections automatically.

Another everyday use case is to delete documents based on an expiry date. For instance, log events that you only want to store for a specific amount of days or session data that should only be stored for a few hours. MongoDB supports this use case, not with a special collection, but with a particular index: Time to live (TTL) index

You can create a TTL index on any date field or any date array field. When you create the index, you specify the amount of time after the document should expire.

      collection.createIndex(Indexes.ascending("date"),
          new IndexOptions().expireAfter(1L, TimeUnit.MINUTES));

Ttl1.java

You can use a TTL index for queries like a regular index. When you index a date array with a TTL index, MongoDB chooses the earliest (lowest) date in the array for computing the expiry date.
MongoDB automatically deletes documents after the indexed date + specified amount of time has passed.
If the indexed field in the document is missing or not a date or not a date array, the document is never going to expire.

      MongoCollection<Document> collection = db.getCollection("log");

      // TTL Index
      collection.createIndex(Indexes.ascending("date"),
          new IndexOptions().expireAfter(1L, TimeUnit.MINUTES));

      for (int j = 0; j < 5; j++) {
        Document logMessage = new Document();
        logMessage.append("date", new Date());
        logMessage.append("severity", "INFO");
        logMessage.append("message", String.valueOf(j));
        collection.insertOne(logMessage);
      }

      System.out.println(collection.countDocuments()); // 5

      TimeUnit.SECONDS.sleep(120);

      System.out.println(collection.countDocuments()); // 0

Ttl1.java

MongoDB runs a background thread every 60 seconds that checks the TTL indexes and deletes the documents. Therefore, a TTL index does not guarantee that a document is immediately deleted when the expiry date is reached. Most of the time, there is a delay between the expiration date and the time MongoDB deletes the document.

You can also insert documents with an expiry date in the past. Such a document is not going be deleted immediately, MongoDB removes it the next time the background thread runs.

      MongoCollection<Document> collection = db.getCollection("log");

      // TTL Index
      collection.createIndex(Indexes.ascending("date"),
          new IndexOptions().expireAfter(1L, TimeUnit.MINUTES));

      Document logMessage = new Document();
      logMessage.append("date", toDate(LocalDateTime.now().minusMinutes(5)));
      logMessage.append("severity", "INFO");
      logMessage.append("message", "in the past");
      collection.insertOne(logMessage);

      System.out.println(collection.countDocuments()); // 1

      TimeUnit.MINUTES.sleep(1);

      System.out.println(collection.countDocuments()); // 0

Ttl2.java

You may also add a TTL index to an existing collection. The same behavior applies here; the documents are not immediately deleted only the next time the background thread runs.

Specific expiry date ¶

The previous example expired all the documents after date + 1 minute. But what if we want to expire the documents with different expiration dates. For instance, we store log events, and we want to expire logs with the severity INFO after 1 hour, with the severity WARN after 7 days and ERROR logs after 1 month.

For that use case, we add a field to our document that specifies the expiration date and create a TTL index on this field with an expireAfter of 0.

      MongoCollection<Document> collection = db.getCollection("log");
      collection.createIndex(Indexes.ascending("expireAt"),
          new IndexOptions().expireAfter(0L, TimeUnit.SECONDS));

Ttl3.java

When we insert the documents, we can now set the exact expiration date and time.

      Document logMessage = new Document();
      logMessage.append("date", new Date());
      logMessage.append("expireAt", toDate(LocalDateTime.now().plusSeconds(10)));
      logMessage.append("severity", "INFO");
      logMessage.append("message", "a debug message");
      collection.insertOne(logMessage);

      logMessage = new Document();
      logMessage.append("date", new Date());
      logMessage.append("expireAt", toDate(LocalDateTime.now().plusMinutes(2)));
      logMessage.append("severity", "WARN");
      logMessage.append("message", "an info message");
      collection.insertOne(logMessage);

      logMessage = new Document();
      logMessage.append("date", new Date());
      logMessage.append("expireAt", toDate(LocalDateTime.now().plusMinutes(5)));
      logMessage.append("severity", "ERROR");
      logMessage.append("message", "an error message");
      collection.insertOne(logMessage);

Ttl3.java

MongoDB deletes the documents at the specified date and time. As usual, there is a little delay between the specified expiration date and MongoDB deleting the document, because the background thread only runs every 60 seconds.

      System.out.println(collection.countDocuments()); // 3
      TimeUnit.SECONDS.sleep(60);
      System.out.println(collection.countDocuments()); // 2
      TimeUnit.SECONDS.sleep(120);
      System.out.println(collection.countDocuments()); // 1
      TimeUnit.SECONDS.sleep(180);
      System.out.println(collection.countDocuments()); // 0

Ttl3.java

Restrictions ¶

There are a few restrictions on TTL indexes you need to be aware of:

A TTL index must be a single-field index. If you create a compound index with expireAfter, MongoDB ignores the option.
The _id field does not support TTL indexes.
You cannot create a TTL index on a capped collection, because these collections do not support removing documents

See the official documentation for a more in-depth description of TTL indexes.
https://www.mongodb.com/docs/manual/core/index-ttl/

You find all the code example from this blog post here:
https://github.com/ralscha/blog/tree/master/capped