January 2015
Volume 30 Number 1
Microsoft Azure - Occasionally Connected Data in Cross-Platform Mobile Apps
By Kevin Ashley
Most mobile appsneed to stay offline for some periods of time, and mobile users expect their apps to work gracefully in both connected and disconnected states. Hundreds of millions of mobile device users might not be aware of needs that the apps have with regard to online and offline states; they simply expect the apps to work under any circumstances. In this article I’ll show you ways to enable your mobile app to work in both states and synchronize data gracefully with the cloud in Windows, iOS and Android using cross-platform Xamarin tooling and Microsoft Azure Mobile Services.
As a mobile app developer myself, I encountered the need to synchronize offline data early on. For my Winter Sports ski app (winter-sports.co) and Active Fitness activity tracking app (activefitness.co), it’s somewhat expected that you can’t get a decent connection on the slopes or on the run. So those apps need to be able to synchronize data collected offline, without significant impact on battery life and reliability. In other words, the apps need to work efficiently in any conditions.
There’s more than meets the eye when considering persistent storage. To start, there are multiple approaches to synchronizing, meaning it can be done from within an app or as a background process in the OS. In addition, there are different types of data stores, such as semistructured data from sensors and potentially relational data stored in SQLite. It’s also important to implement a conflict resolution policy so data loss and degradation are minimized. Finally, there are a wide variety of data formats to manage, such as binary, JSON, XML and custom.
Mobile apps typically store several types of data. Structured data, such as JSON or XML, is typically used for local settings, local files or caching. In addition to storing data in files, you also have a choice of storage engines, such as SQLite, to store and query data. Mobile apps may also store blobs, media files and other types of large binary data. For these types of data, I’ll demonstrate techniques that make data transfer more reliable on occasionally connected devices. I’ll provide an overview of several technologies. For example, instead of focusing strictly on offline sync for structured data, I’ll show you a broader picture of both structured and unstructured (blob) data synchronization techniques. I’ll also use a cross-platform approach everywhere throughout these examples.
The Cross-Platform Approach
Because it’s becoming increasingly popular to connect sensor-enabled devices to the cloud, I included device sensor data in my project to demonstrate various methods to sync it with the cloud. I’ll discuss three scenarios: offline data sync, manual data synchronization, and synchronizing large media and binary data. The accompanying code sample is completely cross-platform, with 100 percent reusability for Android, iOS and Windows. To achieve that, I used Xamarin.Forms, cross-platform XAML/C# tooling that works well on iOS, Android and Windows and is getting integrated with Visual Studio tools (see the Microsoft Channel 9 video, “Cross-Platform Mobile Development Using Visual Studio,” at bit.ly/1xyctO2).
In the code sample are two classes that manage cross-platform data models: SensorDataItem and SensorModel. This approach may be used by many sports and fitness tracking apps, such as Active Fitness, or apps that need to synchronize structured data from local storage with the cloud. I added latitude, longitude, speed and distance to the SensorDataItem class as an example of data collected by sensors, such as GPS, to illustrate the idea. Of course, the data structure in the actual app might be more complicated—and include dependencies—but my example will give you an idea of the concept.
Synchronizing Structured Data with Offline Sync
Offline sync is a powerful new feature in Azure Mobile Services. You can reference the Azure Mobile Services package in your Visual Studio project using NuGet. More important, it’s also supported in cross-platform apps with the new version of the Azure Mobile Services SDK. That means you can use this feature in your Windows, iOS and Android apps that occasionally need to connect to the cloud and synchronize their states.
I’ll start with a few concepts.
Sync Table This is a new object in the Azure Mobile Services SDK created to distinguish tables that support synchronization from “local” tables. Sync tables implement the IMobileServiceSyncTable<T> interface and include additional “sync” methods, such as PullAsync, PushAsync and Purge. If you want to synchronize your offline sensor data with the cloud, you need to use sync tables instead of standard tables. In my code sample, I initialize my sensor data sync table by using the GetSyncTable<T> call. In the Azure Mobile Services portal, I created a regular table called SensorDataItem and added the code in Figure 1 to my client initialization (you can download the full source code from bit.ly/11yZyhN).
The Synchronization Context This is responsible for synchronizing data between local and remote stores. Azure Mobile Services ships with SQLiteStore, which is based on the popular SQLite library. The code in Figure 1 does a few things. It checks whether a synchronization context has already been initialized and, if not, it creates a new instance of the SQLite store from the local.db file, defines the table based on the SensorDataItem class and initializes the store. To handle pending operations, the synchronization context uses a queue, accessible via the PendingOperations property. The synchronization context provided by Azure Mobile Services is also “smart” enough to distinguish update operations happening in the local store. Synchronization is done automatically by the system, so you don’t have to manually and unnecessarily call into the cloud to persist data. This is good because it lowers traffic and increases device battery life.
Figure 1 Leveraging the MobileServiceSQLiteStore Object for Synchronization
// Initialize the client with your app URL and key
client = new MobileServiceClient(applicationURL, applicationKey);
// Create sync table instance
todoTable = client.GetSyncTable<SensorDataItem>();
// Later in code
public async Task InitStoreAsync()
{
if (!client.SyncContext.IsInitialized)
{
var store = new MobileServiceSQLiteStore(syncStorePath);
store.DefineTable<SensorDataItem>();
await client.SyncContext.InitializeAsync(store,
new MobileServiceSyncHandler ());
}
}
The Push Operation This lets you explicitly synchronize data between the local store and the cloud store by pushing local data to the server. It’s important to point out that in the current version of the Azure Mobile Services SDK, you need to invoke push and pull explicitly to synchronize the context. The push operation executes on the whole synchronization context to help you preserve relationships between tables. For example, if I have relationships between tables, my first insert will give me an Id of the object, and subsequent inserts will preserve referential integrity:
await client.SyncContext.PushAsync();
The Pull Operation This lets you explicitly synchronize data by pulling data from a remote store into the local store. You can use LINQ to specify a subset of data, or any OData query. Unlike push operations, which happen on an entire context, pull is executed on the table level. If items are pending in the synchronization queue, those items are pushed first—before pull operations execute—to prevent data loss (yet another benefit from using Azure Mobile Services for data synchronization). In this example, I pull data that has a nonzero speed (collected by my GPS sensor, for example), stored earlier in the server:
var query = sensorDataTable.Where(s => s.speed > 0);
await sensorDataTable.PullAsync(query);
The Purge Operation This clears data specified from local and remote tables, causing synchronization. Similar to pull, you can use LINQ to specify a subset of data, or any OData query. In this example, I purge data that has zero distance (which might also come from my GPS sensor) from my tables:
var query = sensorDataTable.Where(s => s.distance == 0);
await sensorDataTable.PurgeAsync(query);
Appropriately Handling Conflicts This is an important part of your data synchronization strategy when devices come online and offline. Conflicts will happen, and the Azure Mobile Services SDK provides ways to deal with conflicts. For conflict resolution to work, I enabled the Version property column on the SensorDataItem object. In addition, I created the ConflictHandler class, which implements the IMobileServiceSyncHandler interface. When you need to resolve a conflict, you have three options: keep the client value, keep the server value or abort the push operation.
In my example, check the ConflictHandler class. When it’s initialized, in the constructor I set it with one of three conflict resolution policies:
public enum ConflictResolutionPolicy
{
KeepLocal,
KeepRemote,
Abort
}
Depending on the method, each time a conflict happens, in the ExecuteTableOperationAsync method, I automatically apply my conflict resolution policy. When I initialize my synchronization context, I pass my ConflictHandler class to the synchronization context with my default conflict resolution policy:
await client.SyncContext.InitializeAsync(
store,
new ConflictHandler(client, ConflictResolutionPolicy.KeepLocal)
);
To read more on conflict resolution, see the MSDN sample, “Azure Mobile Services - Handling Conflicts with Offline WP8,” at bit.ly/14FmZan and the Azure documentation article, “Handling Conflicts with Offline Data Sync in Mobile Services,” at bit.ly/1zA01eo.
Synchronizing Serialized Data Manually
Before Azure Mobile Services started offering offline synchronization, developers had to implement data synchronization manually. So, if you develop an app that occasionally needs to synchronize data and you don’t use the Azure Mobile Services offline sync feature, you can manually do it (although I strongly recommend looking at the offline sync feature). You can use either direct object serialization (such as a JSON serializer) into files, or data storing engines such as SQLite. The main difference between the offline sync mechanism and manual synchronization is that you need to do most of the work yourself in the latter. One of the methods to detect whether data has been synchronized is by using the Id property of any object in your data model. For example, see the SensorDataItem class used in my earlier example—note the Id and Version fields—shown in Figure 2.
Figure 2 Data Structure for Data Synchronization
public class SensorDataItem
{
public string Id { get; set; }
[Version]
public string Version { get; set; }
[JsonProperty]
public string text { get; set; }
[JsonProperty]
public double latitude { get; set; }
[JsonProperty]
public double longitude { get; set; }
[JsonProperty]
public double distance { get; set; }
[JsonProperty]
public double speed { get; set; }
}
When a record is inserted into a remote database, Azure Mobile Services automatically creates an Id and assigns it to the object, so the Id will be a non-null value when the record has been inserted, and null when the record has never been synchronized with the database:
// Manually synchronizing data
if (item.Id == null)
{
await this.sensorDataTable.InsertAsync(item);
}
Synchronizing deletes and updates manually is a far more challenging and involved process and is beyond the scope of this article. If you’re looking for a comprehensive synchronization solution, look at the offline sync features of the Azure Mobile Services SDK. Of course, this example is simple compared with real-life scenarios, but if you’re looking to implement manual data synchronization, this can give you an idea on where to start. Certainly, because the Azure Mobile Services SDK offers a tested, well-thought-out solution for synchronizing data, I recommend trying the offline sync approach, especially in apps that require a solid, tested method to keep local and remote data in sync.
Transferring Binary Data, Photos and Media to the Cloud
In addition to structured data, apps often need to synchronize unstructured or binary data or files. Think of a mobile photo app or an app that needs to upload a binary file to the cloud, such as a photo or video. Because I explore this subject in a cross-platform context, different platforms have different capabilities. But are they really that different? Synchronizing blob data can be achieved in multiple ways, such as with an in-process service or by leveraging an out-of process, platform-specific background transfer service. To manage downloads, I also provided a simple TransferQueue class, based on ConcurrentQueue. Every time I need to submit a file for upload or download, I add a new Job object to the queue. This is also a common pattern in the cloud, where you put unfinished work in a queue and then have some other background process reading from the queue and completing the work.
In-Process File Transfer Sometimes you need to transfer files directly from within your app. This is the most obvious way of handling blob transfers, but as mentioned earlier, it has disadvantages. To protect the UX, the OS puts a cap on an app’s use of bandwidth and resources. However, this assumes the user is using the app to work. In the case of occasionally disconnected apps, it may not be the best idea. The upside of transferring files directly from an app is that it has full control over data transfer. With full control, an app can take advantage of shared access signature methods to manage uploads and downloads. You can read about the advantages in the Microsoft Azure Storage Team Blog post, “Introducing Table SAS (Shared Access Signature), Queue SAS and Update to Blob SAS,” at bit.ly/1t1Sb94. Although not all platforms provide this functionality built in, if you’re willing to use a REST-based approach, you can certainly take advantage of SAS keys from Azure Storage Services. The downside to transferring files directly from an app is twofold. First, you must write more code. Second, the app must be running, potentially draining battery life and limiting the UX. The best solutions leverage some of the elegance of built-in data synchronization techniques.
I provided source code for a basic upload and download operation in a cross-platform Xamarin app in BlobTransfer.cs (see accompanying code download). This code should work on iOS, Android and Windows. In order to use platform-independent file storage, I used the PCLStorage NuGet package (Install-Package PCLStorage), which allows me to abstract file operations on iOS, Android and Windows.
To initiate an in-process transfer, I call my TransferQueue AddInProcessAsync method:
var ok = await queue.AddInProcessAsync(new Job {
Id = 1, Url = imageUrl, LocalFile = String.Format("image{0}.jpg", 1)});
This schedules a typical in-process download operation, defined in the BlobTransfer object as shown in Figure 3.
Figure 3 Download Operation (Cross-Platform Code)
public static async Task<bool> DownloadFileAsync(
IFolder folder, string url, string fileName)
{
// Connect with HTTP
using (var client = new HttpClient())
// Begin async download
using (var response = await client.GetAsync(url))
{
// If ok?
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
// Continue download
Stream temp = await response.Content.ReadAsStreamAsync();
// Save to local disk
IFile file = await folder.CreateFileAsync(fileName,
CreationCollisionOption.ReplaceExisting);
using (var fs =
await file.OpenAsync(PCLStorage.FileAccess.ReadAndWrite))
{
// Copy to temp folder
await temp.CopyToAsync(fs);
fs.Close();
return true;
}
}
else
{
Debug.WriteLine("NOT FOUND " + url);
return false;
}
}
}
Of course, if you wish to upload a file, you can do it in-process by using the method shown in Figure 4.
Figure 4 Upload Operation (Cross-Platform Code)
public static async Task UploadFileAsync(
IFolder folder, string fileName, string fileUrl)
{
// Connect with HTTP
using (var client = new HttpClient())
{
// Start upload
var file = await folder.GetFileAsync(fileName);
var fileStream = await file.OpenAsync(PCLStorage.FileAccess.Read);
var content = new StreamContent(fileStream);
// Define content type for blob
content.Headers.Add("Content-Type", "application/octet-stream");
content.Headers.Add("x-ms-blob-type", "BlockBlob");
using (var uploadResponse = await client.PutAsync(
new Uri(fileUrl, UriKind.Absolute), content))
{
Debug.WriteLine("CLOUD UPLOADED " + fileName);
return;
}
}
}
Out-of-Process File Transfer Using OS-Specific Transfer Service Downloading and uploading using built-in file transfer services has many advantages. Most platforms offer a service that can transfer large files (both uploading and downloading) autonomously as a background service. You should leverage such services whenever possible because they run out of process—that is, your app isn’t capped on the actual data transfer, which may be fairly expensive in terms of resources consumed. Also, your app doesn’t have to stay in memory all the time to transfer files, and the OS usually provides a conflict resolution (retry) mechanism to restart uploads and downloads. Other advantages include less code to write, the app doesn’t need to be active (the OS manages its own queue of uploads and downloads) and the app is more memory/resource efficient. The challenge, however, is that this method requires platform-specific implementation: iOS, Windows Phone and so on have their own background transfer implementations.
Conceptually, a reliable upload for files in a mobile app using an OS-specific out-of-process service looks similar to the in-app implementation. But the actual upload/download queue management is outsourced to the OS transfer service. For Windows Phone Store apps and Windows Store apps, developers can use BackgroundDownloader and BackgroundUploader objects. For iOS 7 and higher, NSUrlSession provides CreateDownloadTask and CreateUploadTask methods to initiate downloads and uploads.
Using my earlier example, I now need to call my out-of-process method to invoke a call using an OS-specific background transferring service. In fact, because the service is handled by the OS, I’ll schedule 10 downloads to demonstrate the app isn’t blocking and the execution is handled by the OS (in this example, I used the iOS background transferring service):
for (int i = 0; i < 10; i++)
{
queue.AddOutProcess(new Job { Id = i, Url = imageUrl,
LocalFile = String.Format("image{0}.jpg", i) });
}
For an example of a background transfer service for iOS, check BackgroundTransferService.cs. In iOS you first need to initialize a background session with CreateBackgroundSessionConfiguration (note this only works on iOS 8 or later):
using (var configuration = NSUrlSessionConfiguration.
CreateBackgroundSessionConfiguration(sessionId))
{
session = NSUrlSession.FromConfiguration(configuration);
}
Then you can submit a long upload or download operation and the OS will handle it independently from your app:
using (var uri = NSUrl.FromString(url))
using (var request = NSUrlRequest.FromUrl(uri))
{
downloadTask = session.CreateDownloadTask(request);
downloadTask.Resume();
}
You also need to think of a queueing mechanism to reliably upload and download blobs.
Sample Code and Next Steps
All sample code for this article is available on GitHub at bit.ly/11yZyhN. To use this source code, you can use Visual Studio with Xamarin or Xamarin Studio, which is available from xamarin.com. The project uses cross-platform Xamarin.Forms and the Azure Mobile Services library with offline sync. For the next steps, it would be interesting to see an out-of-process service added to community libraries, such as Xamarin Labs, as well as queuing capabilities and conflict resolution similar to what’s currently provided for structured data in the Azure Mobile Services Offline Sync SDK.
To recap, Microsoft Azure Mobile Services provide a powerful and efficient way to synchronize offline data. You can use these services in a cross-platform scenario on Windows, Android and iOS. Microsoft also provides easy-to-use native SDKs that work on every platform. You can improve your app’s reliability in disconnected scenarios by integrating these services and adding offline sync to your apps.
Kevin Ashley is an architect evangelist at Microsoft. He’s a coauthor of “Professional Windows 8 Programming” (Wrox, 2012) and a developer of top apps and games, most notably Active Fitness (activefitness.co). He often presents on technology at various events, industry shows and Webcasts. He works with startups and partners, advising on software design, business and technology strategy, architecture, and development. Follow him at his blog at kevinashley.com and on Twitter at twitter.com/kashleytwit.
Thanks to the following Microsoft technical experts for reviewing this article: Greg Oliver and Bruno Terkaly