Working with MongoDB TTL Indexes from Java

In the previous blog post, we saw how to leverage MongoDB's ability to automatically delete documents based on size using Capped Collections.

Another common use case is deleting documents based on an expiry date. For instance, you might want to store log events for a specific number of days or session data for only a few hours. MongoDB supports this use case, not with a special collection, but with a particular index: a Time-To-Live (TTL) index.

You can create a TTL index on any date field or any date array field. When you create the index, you specify the amount of time after which the document should expire.

      collection.createIndex(Indexes.ascending("date"),
          new IndexOptions().expireAfter(1L, TimeUnit.MINUTES));

Ttl1.java

You can use a TTL index for queries like a regular index. When you index a date array with a TTL index, MongoDB chooses the earliest (lowest) date in the array for computing the expiry date. MongoDB automatically deletes documents after the indexed date plus the specified amount of time has passed. If the indexed field in the document is missing, is not a date, or is not a date array, the document will never expire.

      MongoCollection<Document> collection = db.getCollection("log");

      // TTL Index
      collection.createIndex(Indexes.ascending("date"),
          new IndexOptions().expireAfter(1L, TimeUnit.MINUTES));

      for (int j = 0; j < 5; j++) {
        Document logMessage = new Document();
        logMessage.append("date", new Date());
        logMessage.append("severity", "INFO");
        logMessage.append("message", String.valueOf(j));
        collection.insertOne(logMessage);
      }

      System.out.println(collection.countDocuments()); // 5

      TimeUnit.SECONDS.sleep(120);

      System.out.println(collection.countDocuments()); // 0

Ttl1.java

MongoDB runs a background thread every 60 seconds to check the TTL indexes and delete the expired documents. Therefore, a TTL index does not guarantee that a document will be deleted immediately when the expiry date is reached. Most of the time, there is a delay between the expiration date and the time MongoDB deletes the document.

You can also insert documents with an expiry date in the past. Such documents will not be deleted immediately; MongoDB will remove them the next time the background thread runs.

      MongoCollection<Document> collection = db.getCollection("log");

      // TTL Index
      collection.createIndex(Indexes.ascending("date"),
          new IndexOptions().expireAfter(1L, TimeUnit.MINUTES));

      Document logMessage = new Document();
      logMessage.append("date", toDate(LocalDateTime.now().minusMinutes(5)));
      logMessage.append("severity", "INFO");
      logMessage.append("message", "in the past");
      collection.insertOne(logMessage);

      System.out.println(collection.countDocuments()); // 1

      TimeUnit.MINUTES.sleep(1);

      System.out.println(collection.countDocuments()); // 0

Ttl2.java

You can also add a TTL index to an existing collection. The same behavior applies here; the documents are not deleted immediately, only the next time the background thread runs.

Specific Expiry Date ¶

The previous example expired all documents after date + 1 minute. But what if we want to expire documents with different expiration dates? For instance, if we store log events, we might want to expire logs with the severity INFO after 1 hour, logs with the severity WARN after 7 days, and ERROR logs after 1 month.

For that use case, we add a field to our document that specifies the expiration date and create a TTL index on this field with an expireAfter value of 0.

      MongoCollection<Document> collection = db.getCollection("log");
      collection.createIndex(Indexes.ascending("expireAt"),
          new IndexOptions().expireAfter(0L, TimeUnit.SECONDS));

Ttl3.java

When we insert the documents, we can now set the exact expiration date and time.

      Document logMessage = new Document();
      logMessage.append("date", new Date());
      logMessage.append("expireAt", toDate(LocalDateTime.now().plusSeconds(10)));
      logMessage.append("severity", "INFO");
      logMessage.append("message", "a debug message");
      collection.insertOne(logMessage);

      logMessage = new Document();
      logMessage.append("date", new Date());
      logMessage.append("expireAt", toDate(LocalDateTime.now().plusMinutes(2)));
      logMessage.append("severity", "WARN");
      logMessage.append("message", "an info message");
      collection.insertOne(logMessage);

      logMessage = new Document();
      logMessage.append("date", new Date());
      logMessage.append("expireAt", toDate(LocalDateTime.now().plusMinutes(5)));
      logMessage.append("severity", "ERROR");
      logMessage.append("message", "an error message");
      collection.insertOne(logMessage);

Ttl3.java

MongoDB deletes the documents at the specified date and time. As usual, there is a slight delay between the specified expiration date and MongoDB deleting the document, because the background thread only runs every 60 seconds.

      System.out.println(collection.countDocuments()); // 3
      TimeUnit.SECONDS.sleep(60);
      System.out.println(collection.countDocuments()); // 2
      TimeUnit.SECONDS.sleep(120);
      System.out.println(collection.countDocuments()); // 1
      TimeUnit.SECONDS.sleep(180);
      System.out.println(collection.countDocuments()); // 0

Ttl3.java

Restrictions ¶

There are a few restrictions on TTL indexes that you need to be aware of:

A TTL index must be a single-field index. If you create a compound index with expireAfter, MongoDB ignores the option.
The _id field does not support TTL indexes.
You cannot create a TTL index on a capped collection, because these collections do not support removing documents.

See the official documentation for a more in-depth description of TTL indexes. https://www.mongodb.com/docs/manual/core/index-ttl/

You can find all the code examples from this blog post here:
https://github.com/ralscha/blog/tree/master/capped