Unity: Use PersistentDictionary and ESENT to store large collection object in Unity

This post shows how to add ESENT to Unity game and how to use PersistentDictionary in ESENT to store a very large collection of objects in Unity game.

Firstly, we need to download ESENT and make it available to Unity game development, double-click any C# script in the Unity project panel to bring up the Visual Studio, right-click the project in the Visual Studio and select "Managed NuGet Packages...". In the "Manage NuGet Packages ...", key in "ESENT" to search for the Extensible Storage Engine, and click the "Install" button for the first result (which is "ManagedEsent 1.6" in my case) coming up. This will install the ESENT references for development in Visual Studio.

Next we need to make ESENT libraries available to Unity. To this end, create a new folder under "Assets" in the Unity project panel (e.g., name it "References"). Now go to the "Packages\ManagedEsent.1.6\lib\net20" folder in your Unity project folder, and drag the "Esent.Collections.dll" and "Esent.Interop.dll" into the "References" folder in the Unity project folder. Upon this, Unity will now recognize the ESENT library and you can use it in C#. Below is a C# code snippet which shows how to uses a PersistentDictionary object from ESENT to store a large number of objects.

using Microsoft.Isam.Esent.Collections.Generic;
using UnityEngine;

public class Employee
{
 public string ID;
 public override string ToString()
 {
  return ID;
 }
 public static Employee ParseEmployee(string serializedContent)
 {
  Employee person = new Employee();
  person.ID = serializedContent;
  return person;
 }
}
public class EmployeePool 
{
 private string mID;
 protected PersistentDictionary<string, string> mData = null;

 public EmployeePool(string id)
 {
  mID = id;
  
 }

 public void Add(Employee person)
 {
  if (mData == null)
  {
   CreatePersistentDictionary<string, string>(ref mData, mID);
  }
  mData[person.ID] = person.ToString();
 }

 public void Clear()
 {
  CreatePersistentDictionary<string, string>(ref mData, mID);   
 }

 private static void CreatePersistentDictionary<T, V>(ref PersistentDictionary<T, V> current, string directory)
    where T : IComparable<T>
 {
  string full_directory_path = Path.Combine(Application.persistentDataPath, directory);

  if (PersistentDictionaryFile.Exists(full_directory_path))
  {
   PersistentDictionaryFile.DeleteFiles(full_directory_path);
  }

  current = new PersistentDictionary<T, V>(full_directory_path);
 }

 private static void DeletePersistentDictionary<T, V>(ref PersistentDictionary<T, V> current, string directory)
  where T: IComparable<T>
 {
  string full_directory_path = Path.Combine(Application.persistentDataPath, directory);

  if (PersistentDictionaryFile.Exists(full_directory_path))
  {
   PersistentDictionaryFile.DeleteFiles(full_directory_path);
  }
  current=null;
 }

 public int Count
 {
  get
  {
   if (mData == null)
   {
    return 0;
   }
   return mData.Count;
  }
 }

 public void Remove(Employee e)
 {
  mData.Remove(e.ID);
 }

 public Employee this[string index]
 {
  get
  {
   if (mData.ContainsKey(index))
   {
    return Employee.ParseEmployee(mData[index]);
   }
   return null;
  }
 }

 public IEnumerable<Employee> Enumerator
 {
  get
  {
   foreach (string key in mData.Keys)
   {
    yield return this[key];
   }
  }
 }
}

As PersistentDictionary is quite peculiar about the key type and the value type, in this particular case, each of the Employee object is given a id which is a string and serves as the key for the PersistentDictionary as it has a IComparable interface already. the Employee object itself is serialize into a string to be store (the serialization can be a JSON string, e.g.), the Enumerator function allows each Employee object in the collection to be iterated.

One thing to note is the CreatePersistentDictionary method, the method is implemented such that whenever a collection is created with an existing mID, the old data stores in the database with that mID as name will be wiped out. This is intended to let user starts with an empty database when clear is called (all when the constructor is called).