Working with MongoDB TTL Indexes from Java

Published: February 05, 2018  •  database, mongodb, java

In the previous blog post we have seen how to leverage the ability of MongoDB to automatically delete documents based on the size with Capped Collections.

Another common use case is to delete documents based on a expiry date. For instance log events that you only want to store for certain days or session data that should only be stored for a few hours. MongoDB supports this use case, not with a special collection, but with a special index: Time to live index

You can create a TTL index on any date field or any date array field. When you create the index you specify the amount of time after the document should expire.

 collection.createIndex(Indexes.ascending("date"),
          new IndexOptions().expireAfter(1L, TimeUnit.MINUTES));

Ttl1.java

You can use a TTL index for queries like a normal index. When you index a date array with a TTL index MongoDB chooses the earliest (lowest) date in the array for computing the expiry date.
MongoDB automatically deletes documents after the indexed date + specified amount of time has passed.
If the indexed field in the document is missing or not a date or not a date array the document will never expire.

  for (int j = 0; j < 5; j++) {
    Document logMessage = new Document();
    logMessage.append("date", new Date());
    logMessage.append("severity", "INFO");
    logMessage.append("message", String.valueOf(j));
    collection.insertOne(logMessage);
  }

  System.out.println(collection.count()); // 5

  TimeUnit.SECONDS.sleep(120);

  System.out.println(collection.count()); // 0

Ttl1.java

MongoDB runs a background thread every 60 seconds that checks the TTL indexes and deletes the documents. Therefore a TTL index does not guarantee that a document is immediately deleted when the expiry date is reached. Most of the time there is a delay between expiration date and the time MongoDB deletes the document.


You can also insert documents with an expiry date in the past. Such a document will not be deleted immediately, MongoDB removes it the next time the background thread runs.

  collection.createIndex(Indexes.ascending("date"),
          new IndexOptions().expireAfter(1L, TimeUnit.MINUTES));

  Document logMessage = new Document();
  logMessage.append("date", toDate(LocalDateTime.now().minusMinutes(5)));
  logMessage.append("severity", "INFO");
  logMessage.append("message", "in the past");
  collection.insertOne(logMessage);

  System.out.println(collection.count()); // 1

  TimeUnit.MINUTES.sleep(1);

  System.out.println(collection.count()); // 0

Ttl2.java

You may also add a TTL index to an existing collection. Same behavior applies here, the documents are not immediately deleted only the next time the background thread runs.


Specific expiry date

The previous example expired all the documents after date + 1 minute. But what if we want to expire the documents with different expiration dates. For instance, we store log events and we want to expire logs with the severity INFO after 1 hour, with the severity WARN after 7 days and ERROR logs after 1 month.

For that use case we add a field to our document that specifies the expiration date and create a TTL index on this field with a expireAfter of 0.

  collection.createIndex(Indexes.ascending("expireAt"),
          new IndexOptions().expireAfter(0L, TimeUnit.SECONDS));

Ttl3.java

When we insert the documents we can now set the exact expiration date and time.

  Document logMessage = new Document();
  logMessage.append("date", new Date());
  logMessage.append("expireAt", toDate(LocalDateTime.now().plusSeconds(10)));
  logMessage.append("severity", "INFO");
  logMessage.append("message", "a debug message");
  collection.insertOne(logMessage);

  logMessage = new Document();
  logMessage.append("date", new Date());
  logMessage.append("expireAt", toDate(LocalDateTime.now().plusMinutes(2)));
  logMessage.append("severity", "WARN");
  logMessage.append("message", "an info message");
  collection.insertOne(logMessage);

  logMessage = new Document();
  logMessage.append("date", new Date());
  logMessage.append("expireAt", toDate(LocalDateTime.now().plusMinutes(5)));
  logMessage.append("severity", "ERROR");
  logMessage.append("message", "an error message");
  collection.insertOne(logMessage);

Ttl3.java

MongoDB deletes the documents at the specified date and time. As usual there will a little delay between the specified expiration date and MongoDB deleting the document, because the background thread only runs every 60 seconds

  System.out.println(collection.count()); // 3
  TimeUnit.SECONDS.sleep(60);
  System.out.println(collection.count()); // 2
  TimeUnit.SECONDS.sleep(120);
  System.out.println(collection.count()); // 1
  TimeUnit.SECONDS.sleep(180);
  System.out.println(collection.count()); // 0

Ttl3.java


Restrictions

There are a few restrictions on TTL indexes you need to be aware of:


See the official documentation for a more in depth description of TTL indexes.
https://docs.mongodb.com/manual/core/index-ttl/

You find all the code example from this blog post here:
https://github.com/ralscha/blog/tree/master/capped