After diving deep into IndexedDB land in the previous blog post, I wanted to create a complete app with IndexedDB. A use case for client-side storage is to make web applications work when offline. The goal of this blog post is to create such an application.
As a result, I have built another Earthquake app. You may have seen the initial version of the app in action in my blog post about Lovefield. By the way, Lovefield uses IndexedDB under the hood as a storage engine.
The web app I created for this blog post, downloads earthquake data from the USGS Earthquake Hazard Program site, stores the records into IndexedDB and displays them. The user can filter the records and specify the sort order.
You find the source code for the application on Github: https://github.com/ralscha/blog/tree/master/indexeddb
and you can test it online: https://omed.hplar.ch/indexeddb/
The app is built with the Ionic Framework and based on the blank starter template (ionic start earthquake blank).
I also added the Angular Service Worker
ng add @angular/pwa --project app
to cache all the resources in the browser to make the application work offline.
I won't go into detail about the user interface and focus on the part that stores and queries the database.
All the database code is encapsulated in the EarthquakeService class: https://github.com/ralscha/blog/blob/master/indexeddb/src/app/earthquake.service.ts
Initialize Database ¶
The initDb() method creates a database Earthquake with version 1. In the upgradeneeded handler, it creates the object store earthquakes with a primary key that references the id property and is not auto-generated. Additionally, the method creates two indexes on the properties mag and time.
private initDb(): Promise<void> {
if (this.db) {
this.db.close();
}
return new Promise(resolve => {
const openRequest = indexedDB.open('Earthquake', 1);
openRequest.onupgradeneeded = event => {
const target: EventTarget | null = event.target;
const db = (target as any).result;
const store = db.createObjectStore('earthquakes', {keyPath: 'id'});
store.createIndex('mag', 'mag');
store.createIndex('time', 'time');
The success handler assigns the database object to an instance variable because we need a reference to the database in other parts of the service.
The open() call is wrapped in a Promise, and the success handler resolves it.
The cast (event.target as any).result is a workaround because TypeScript does not recognize the result property on the EventTarget object.
openRequest.onsuccess = event => {
this.db = (event.target as any).result;
this.db.onerror = e => {
console.log(e);
};
resolve();
Load and insert data ¶
The method loadData() downloads the CSV file from the USGS site. I chose CSV instead of JSON because the file is smaller in size, and with the help of the Papa Parser library, parsing is as easy as parsing a JSON file.
The Angular HTTP client converts the response by default to JSON, but because we want to download a CSV file, we have to set the response type to text: responseType: 'text'.
After that, the method parses the text file with Papa Parse to an array of objects (data.data). The application process one record after the other with a for of loop, converts it into an object and stores it with store.put() into the database.
The code only stores the properties that it needs for the application to save space.
The if (row.id) statement is needed because the file from the USGS site contains an empty last line, and Papa Parse creates an empty last array element.
We don't have to worry about duplicate entries when we process the same file multiple times. Each record has a unique key (id), and the put method does either an insert when the primary key does not yet exist or an update when the record already exists.
After every record is stored in the database, the code stores a lastUpdate key with the current time as a value into localStorage. You see, in a moment, what the application uses this entry for. For storing simple primitives and objects, I prefer localStorage. The application does not need to query this object, and there is only one entry.
Like the initDb() method, the whole body of the loadData() method is wrapped in a Promise, and the transaction complete handler resolves it.
The complete event is emitted as soon as all asynchronous store.put() calls complete successfully
private loadData(dataUrl: string): Promise<void> {
return new Promise(resolve => {
const response = this.http.get(dataUrl, {responseType: 'text'}).pipe(map(data => parse<{
id: string, time: string, place: string, mag: string,
depth: string, latitude: string, longitude: string
}>(data, {header: true})));
response
.subscribe(data => {
const tx = this.db.transaction('earthquakes', 'readwrite');
const store = tx.objectStore('earthquakes');
for (const row of data.data) {
if (row.id) {
store.put({
time: new Date(row.time).getTime(),
place: row.place,
mag: Number(row.mag),
depth: Number(row.depth),
latLng: [Number(row.latitude), Number(row.longitude)],
id: row.id
});
}
}
tx.oncomplete = () => {
localStorage.setItem('lastUpdate', Date.now().toString());
resolve();
};
});
});
Delete old data ¶
The next method deleteOldRecords() is responsible for deleting old records. Because the application loads new data from time to time into the database, it would grow indefinitely without such a cleanup job. The application calls this method every time it inserted new records. The deleteOldRecords() method deletes all records that are older than 30 days. The time property stores the time in milliseconds since 1970-1-1.
The code starts a read-write transaction, gets a reference to the earthquakes object store, and then starts a cursor operation with openCursor() and an upperBound query.
The cursor loops over each record that is older than 30 days and the method calls cursor.delete() to delete the underlying record. cursor.continue() moves the cursor to the next record. When event.target.result is null, the cursor reached the end.
I added a transaction complete handler that resolves the Promise, because cursor.delete() is an asynchronous operation, and the application has to wait before it can continue.
private deleteOldRecords(): Promise<void> {
const tx = this.db.transaction('earthquakes', 'readwrite');
const store = tx.objectStore('earthquakes');
const timeIndex: IDBIndex = store.index('time');
const thirtyDaysAgo = Date.now() - EarthquakeService.THIRTY_DAYS;
return new Promise(resolve => {
timeIndex.openCursor(IDBKeyRange.upperBound(thirtyDaysAgo)).onsuccess = event => {
const cursor = (event.target as any).result;
if (cursor) {
cursor.delete();
cursor.continue();
}
};
tx.oncomplete = () => resolve();
});
Initialization ¶
The next method in our service is initProvider() that is called when the application starts (from the HomePage ngOnInit method). This method calls the three methods we discussed so far.
First, it initializes the database with a call to this.initDb().
Then it tests if the device is online. The next step needs Internet access, and we can simply stop here when the device is offline
initProvider(): Promise<void> {
let promise = this.initDb();
if (!navigator.onLine) {
return promise;
Then the method reads the lastUpdate entry from localStorage.
The trick here is that the USGS site provides several files with different periods: past hour, past day, past 7 days, and past 30 days.
To save bandwidth, the method compares the time of the last update with the current time and downloads the file from the USGS that is the most suitable. For example, when the last update was 5 days ago, there is no need to download the full 30 days file. It only has to download the 7 days file to keep the database up to date.
If there is no lastUpdate entry in localStorage or the last update was more than 30 days in the past, the method downloads and inserts the 30 days file.
const lastUpdate = localStorage.getItem('lastUpdate');
if (lastUpdate) {
const lastUpdateTs = parseInt(lastUpdate, 10);
const now = Date.now();
if (lastUpdateTs + EarthquakeService.SEVEN_DAYS < now) {
// database older than 7 days. load the 30 days file
promise = promise.then(() => this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv'));
} else if (lastUpdateTs + EarthquakeService.ONE_DAY < now) {
// database older than 1 day. load the 7 days file
promise = promise.then(() => this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.csv'));
} else if (lastUpdateTs + EarthquakeService.ONE_HOUR < now) {
// database older than 1 hour. load the 1 day file
promise = promise.then(() => this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.csv'));
} else if (lastUpdateTs + EarthquakeService.FOURTYFIVE_MINUTES < now) {
// database older than 45 minutes. load the 1 hour file
promise = promise.then(() => this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.csv'));
}
} else {
// no last update. load the 30 days file
promise = promise.then(() => this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv'));
And as the last step, the initProvider() method calls this.deleteOldRecords() to delete all records in the database that are older than 30 days.
This concludes the initialization of the database. But an application that only downloads and inserts data into IndexedDB would not be that useful, and therefore we need some code to query the data.
Query ¶
The Earthquake service does all this in the filter() method. This method returns a promise like all the other methods of this class. It starts a read-only transaction, gets a reference to the two indexes, and runs the queries. The user can filter on the magnitude, time of the earthquake, and the distance, and the filter() method has to handle all these cases.
filter(filter: Filter): Promise<Earthquake[]> {
const tx = this.db.transaction('earthquakes', 'readonly');
const store = tx.objectStore('earthquakes');
const magIndex: IDBIndex = store.index('mag');
const timeIndex: IDBIndex = store.index('time');
const hasMagFilter = !(filter.mag.lower === -1 && filter.mag.upper === 10);
const hasDistanceFilter = !(filter.distance.lower === 0 && filter.distance.upper === 20000);
The method can only filter on the magnitude and time property in the database. When the user only filters on the magnitude property, the method calls getAll() on the magIndex with a range object.
let promise = new Promise<Earthquake[]>(resolve => {
if (hasMagFilter && !hasTimeFilter) {
magIndex.getAll(IDBKeyRange.bound(filter.mag.lower, filter.mag.upper))
I use a similar code for the time filter. When the user only filters on the time, the method can call getAll() on the timeIndex and returns all records that match the requested period.
.onsuccess = e => resolve((e.target as any).result);
} else if (!hasMagFilter && hasTimeFilter) {
const now = new Date();
now.setHours(now.getHours() - parseInt(filter.time, 10));
timeIndex.getAll(IDBKeyRange.lowerBound(now.getTime()))
It gets a bit more complicated when the user filters on magnitude and time. For that case, the method calls getAllKeys on both indexes to get the primary keys, then creates an intersection of these two result arrays and finally fetches each record with store.get.
.onsuccess = e => resolve((e.target as any).result);
} else if (hasMagFilter && hasTimeFilter) {
const magPromise = new Promise<string[]>(res => {
magIndex.getAllKeys(IDBKeyRange.bound(filter.mag.lower, filter.mag.upper))
.onsuccess = e => res((e.target as any).result);
});
const now = new Date();
now.setHours(now.getHours() - parseInt(filter.time, 10));
const timePromise = new Promise<string[]>(res => {
timeIndex.getAllKeys(IDBKeyRange.lowerBound(now.getTime()))
.onsuccess = e => res((e.target as any).result);
});
Promise.all([magPromise, timePromise]).then(results => {
const intersection = results[0].filter(id => results[1].includes(id));
const result: Earthquake[] = [];
intersection.forEach(id => {
store.get(id).onsuccess = e => result.push((e.target as any).result);
});
tx.oncomplete = () => resolve(result);
When the user does not filter on any property, the method calls getAll() on the magIndex. It does not matter if we use store.getAll() or magIndex.getAll() both calls would return the same amount of records (but in a different order).
});
} else {
magIndex.getAll()
.onsuccess = e => resolve((e.target as any).result);
The distance filtering happens outside the scope of IndexedDB and for that the filter() method processes the results it got from the previous queries, calculates the distance for each record to the current position of the device (geolib) and filters out all records that are not in the requested distance range. The method also does this when the user requested a sort on the distance. The distance property is not stored in the database because of the current position of a mobile device changes.
if (hasDistanceFilter || filter.sort === 'distance') {
promise = promise.then(e => {
const filtered: Earthquake[] = [];
e.forEach(r => {
const distanceInKilometers = geolib.getDistance(
{latitude: r.latLng[0], longitude: r.latLng[1]},
{latitude: filter.myLocation.latitude, longitude: filter.myLocation.longitude}) / 1000;
if (hasDistanceFilter) {
if (filter.distance.lower <= distanceInKilometers && distanceInKilometers <= filter.distance.upper) {
r.distance = distanceInKilometers;
filtered.push(r);
}
} else {
r.distance = distanceInKilometers;
filtered.push(r);
}
});
return filtered;
});
The last task in the filter() method is sorting. This happens in memory by calling the built-in sort() method on the JavaScript array. I'm sure there are smarter ways to do that because the indexes are already sorted. For example, when the user filters and sorts on the magnitude property, there is no need to sort the result again. But on the other hand, when the user filters on magnitude but sorts on distance, the method has to sort it in memory.
if (filter.sort === 'mag') {
return promise.then(e => e.sort((a, b) => b.mag - a.mag));
}
if (filter.sort === 'distance') {
return promise.then(e => e.sort((a, b) => {
if (a.distance && b.distance) {
return a.distance - b.distance;
} else if (!a.distance && b.distance) {
return -1;
} else if (a.distance && !b.distance) {
return 1;
} else {
return 0;
}
}));
Promisify ¶
You have seen a lot of IndexedDB code in this and the previous blog post that is wrapped in a Promise.
Promises make working with IndexedDB much more enjoyable and easier, but when you do a lot of IndexedDB programming, wrapping all these calls with Promises becomes a bit tedious.
Fortunately, there is a library that solves that problem. IndexedDB Promised is a thin layer around IndexedDB and wraps the IDBRequest in a Promise. You find the library and documentation on GitHub: https://github.com/jakearchibald/idb
Google's documentation about IndexedDB uses this library for its code examples too: https://web.dev/articles/indexeddb
When you work with the library, you use the object idb instead of indexedDB as the main entry point. The idb.open() method returns a promise and takes three arguments: the database name, the version, and the upgrade function.
const dbPromise = idb.open('contatcsDb', 1, upgradeDB => {
upgradeDB.createObjectStore('contacts', {keyPath: 'id'});
});
The common programming pattern looks like the following code fragment. The application uses the Promise it gets from the open() call and calls the then() method. The then handler receives a reference to the database (db) and uses the usual IndexedDB methods to do his job. What's new is the property tx.complete. This property points to a promise that resolves when the transaction completes and rejects when the transaction aborts or throws an error.
dbPromise.then(db => {
const tx = db.transaction('contacts', 'readwrite');
tx.objectStore('contacts').put({
id: 1,
firstName: 'John',
lastName: 'Doe',
email: 'john@email.com'
});
return tx.complete;
});
Another example with getAll(). All the methods that return a IDBRequest in the IndexedDB API return a Promise with the IndexedDB Promised library.
dbPromise.then(db => {
return db.transaction('contacts').objectStore('contacts').getAll();
}).then(allObjs => console.log(allObjs));