In the previous blog post, I wrote about Dexie, an IndexedDB wrapper library that simplifies IndexedDB programming.
In this blog post, I'll show you a complete application and how to integrate Dexie into a TypeScript (Ionic) application.
The application is a clone of the application I wrote for this blog post about IndexedDB. It downloads a list of earthquakes that happened in the last month from the USGS Earthquake Hazard Program website, inserts the records into an IndexedDB database, and displays the data in a list. The user can filter the data based on various criteria, like magnitude and time.
Dexie with TypeScript ¶
When you try to add Dexie using the method I showed you in the previous blog post, with code like this, you'll encounter an error:
const db = new Dexie("MyDb");
db.version(1).stores({
earthquakes: 'id'
});
const allObjects = await db.earthquakes.toArray();
In vanilla JavaScript, you get implicit properties for each object store you define in stores(). However, since they are defined at runtime, TypeScript does not recognize them, and this code will not compile.
A workaround is to use the table() method:
const allObjects = await db.table('earthquakes').toArray();
But you won't get any code completion or type safety, which is a primary reason for using TypeScript in the first place.
Therefore, the recommended way is to create a TypeScript class. Before we do that, we'll define an interface for our entity that we want to store in the database.
export interface Earthquake {
id: string;
time: number;
place: string;
mag: number;
depth: number;
distance?: number;
latLng: [number, number];
}
Now, we'll create the special Dexie database class. It has to be a subclass of Dexie. In the constructor, you must call the constructor of the superclass with the name of the database as a parameter.
export class EarthquakeDb extends Dexie {
earthquakes!: Dexie.Table<Earthquake, string>;
constructor() {
super('Earthquake');
this.version(1).stores({
earthquakes: 'id,mag,time'
});
}
}
The class contains an instance variable for each object store. In this example, we only have one object store with the name earthquakes. The instance variable is of type Dexie.Table, and the generic data types specify the type of the object (Earthquake) and the type of the primary key (string). Make sure that the name of the instance variable matches the name of the object store.
In the constructor, after the super call, we add the code for the schema definition. Each time you need to change the schema, you add a new this.version(...) statement to the constructor.
Instead of using an interface, as we do in this example, you can also use classes as entity objects. See the documentation for an example: https://dexie.org/docs/Typescript
With the Dexie class in place, we can now instantiate it in the constructor of our service.
constructor() {
this.db = new EarthquakeDb();
}
This is all you need to do to set up Dexie in a TypeScript application. From here on, accessing the database works the same way as in all the other examples you see in the documentation or my previous blog post.
Data Load ¶
Every time the application starts, or the user uses the pull-to-refresh function, the application calls the initProvider() method of the service. Here, we first fetch the timestamp of the last update.
const lastUpdate = localStorage.getItem('lastUpdate');
After each successful download and insert, a timestamp is stored in the localStorage. With that timestamp, we can determine how old the data in our database is and what data file we have to download to keep it up-to-date. Fortunately for us, the USGS website hosts the earthquake data with different periods (hour, day, week, month). So, we don't have to download the whole month if our database is just a few hours old.
if (lastUpdate) {
const lastUpdateTs = parseInt(lastUpdate, 10);
const now = Date.now();
if (lastUpdateTs + EarthquakeService.SEVEN_DAYS < now) {
// database older than 7 days. load the 30 days file
await this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv');
} else if (lastUpdateTs + EarthquakeService.ONE_DAY < now) {
// database older than 1 day. load the 7 days file
await this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.csv');
} else if (lastUpdateTs + EarthquakeService.ONE_HOUR < now) {
// database older than 1 hour. load the 1 day file
await this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.csv');
} else if (lastUpdateTs + EarthquakeService.FOURTYFIVE_MINUTES < now) {
// database older than 45 minutes. load the 1 hour file
await this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.csv');
}
} else {
// no last update. load the 30 days file
await this.loadData('https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv');
}
return this.deleteOldRecords();
The loadData() method downloads the file from the USGS website using the Fetch API, parses it with the PapaParse library, creates an Earthquake object for each entry, and inserts it into the database using the bulkPut() method.
We don't have to worry about importing duplicates because a primary key is assigned to each earthquake, and bulkPut() only inserts a record if the key does not already exist. If it does exist, it overwrites the existing entry.
private async loadData(dataUrl: string): Promise<void> {
const response = await fetch(dataUrl);
const text = await response.text();
const data = parse<{
id: string, time: string, place: string, mag: string,
depth: string, latitude: string, longitude: string
}>(text, {header: true});
const earthquakes: Earthquake[] = [];
for (const row of data.data) {
if (row.id) {
earthquakes.push({
time: new Date(row.time).getTime(),
place: row.place,
mag: Number(row.mag),
depth: Number(row.depth),
latLng: [Number(row.latitude), Number(row.longitude)],
id: row.id
});
}
}
this.db.transaction('rw', this.db.earthquakes, async () => {
await this.db.earthquakes.bulkPut(earthquakes);
localStorage.setItem('lastUpdate', Date.now().toString());
});
}
Note that bulkPut() should (in most cases) be surrounded by a manual transaction. If you don't do that and the operation fails halfway, the successfully imported records will not be rolled back. You'll end up with a partially imported dataset.
Delete Old Data ¶
We don't want to fill the database indefinitely. Therefore, after each successful import, we delete the earthquake objects that are older than 30 days.
private deleteOldRecords(): Promise<number> {
const thirtyDaysAgo = Date.now() - EarthquakeService.THIRTY_DAYS;
return this.db.earthquakes.where('time').below(thirtyDaysAgo).delete();
}
Query ¶
The filter() method is responsible for applying the filter criteria and returns a promise with the matching Earthquake objects.
async filter(filter: Filter): Promise<Earthquake[]> {
const hasMagFilter = !(filter.mag.lower === -1 && filter.mag.upper === 10);
const hasDistanceFilter = !(filter.distance.lower === 0 && filter.distance.upper === 20000);
const hasTimeFilter = filter.time !== '-1';
const now = new Date();
let result: Earthquake[];
We check what filter is enabled and build the query accordingly. For the magnitude filter, we use between() to filter earthquakes with a minimum and maximum magnitude. Both bounds are included (the 3rd and 4th parameters of the between() call).
if (hasMagFilter && !hasTimeFilter) {
result = await this.db.earthquakes.where('mag').between(filter.mag.lower, filter.mag.upper, true, true).toArray();
When we filter on time, we use aboveOrEqual().
} else if (!hasMagFilter && hasTimeFilter) {
now.setHours(now.getHours() - parseInt(filter.time, 10));
result = await this.db.earthquakes.where('time').aboveOrEqual(now.getTime()).toArray();
When the user enables both the magnitude and time filters, we have to combine the two conditions. IndexedDB does not natively support AND queries. Dexie provides the and() method for this purpose. The first part of the query runs in IndexedDB, and the and() function runs in JavaScript.
result = await this.db.earthquakes.where('time').aboveOrEqual(now.getTime())
.and(e => e.mag >= filter.mag.lower && e.mag <= filter.mag.upper).toArray();
I struggled a bit with this query because it was slow. On my computer with Chrome and about 9400 earthquake objects, it took about 1 second.
After experimenting with different queries, I ended up with the following solution, which is much faster (30-40 milliseconds). It first executes the time query and then filters the objects by magnitude in JavaScript.
The reason this is much faster than the and() query is that toArray() calls the native getAll() method from IndexedDB, which is natively implemented in Chrome, whereas and() uses an IndexedDB cursor, which is slower.
This might not be the best solution for every use case. The problem is that toArray() loads all objects into memory at once. Fortunately, the individual objects in this example are not that large.
} else if (hasMagFilter && hasTimeFilter) {
now.setHours(now.getHours() - parseInt(filter.time, 10));
result = await this.db.earthquakes.where('time').aboveOrEqual(now.getTime()).toArray();
result = result.filter(e => e.mag >= filter.mag.lower && e.mag <= filter.mag.upper);
The distance filtering is done entirely in JavaScript. Because the app can run on a mobile device, we have to recalculate the distance to the earthquakes each time we filter the data. I use the geolib library for this calculation.
let filtered: Earthquake[] = [];
if (hasDistanceFilter || filter.sort === 'distance') {
result.forEach(r => {
const distanceInKilometers = geolib.getDistance(
{latitude: r.latLng[0], longitude: r.latLng[1]},
{latitude: filter.myLocation.latitude, longitude: filter.myLocation.longitude}) / 1000;
if (hasDistanceFilter) {
if (filter.distance.lower <= distanceInKilometers && distanceInKilometers <= filter.distance.upper) {
r.distance = distanceInKilometers;
filtered.push(r);
}
} else {
r.distance = distanceInKilometers;
filtered.push(r);
}
});
} else {
filtered = result;
}
At the end of the filter() method, the code sorts the results in JavaScript according to the selected sort order.
if (filter.sort === 'mag') {
return filtered.sort((a, b) => b.mag - a.mag);
}
if (filter.sort === 'distance') {
return filtered.sort((a, b) => {
if (a.distance && b.distance) {
return a.distance - b.distance;
} else if (!a.distance && b.distance) {
return -1;
} else if (a.distance && !b.distance) {
return 1;
} else {
return 0;
}
});
}
You can find the complete source code for this version of the app with Dexie and the native IndexedDB version on GitHub.
- Dexie version: https://github.com/ralscha/blog/tree/master/dexiejs
- IndexedDB version: https://github.com/ralscha/blog/tree/master/indexeddb
When you compare the two versions, you'll see that the database code with Dexie is more concise and easier to read.