Porting Palm Pilot to Nexus 7
July 19, 2014

1. Summary
2. My Palm Pilot Plan
3. Long-Term Problems with Palm IIIxe
4. Advantages of Nexus 7
5. Requirements for PIM Software
6. PIM Software Alternatives
   1. PIM Applications and Services
   2. Outliners
      • Advantages and Disadvantages of Outliners
      • Examples of Outliner Applications
   3. No Existing PIM Software is Good Enough
7. Building a PIM Application With Vault 3 and Git
1. Vault 3
2. Git
3. Porting PIM Data from Palm Pilot
8. The PIM Synchronization Problem
1. How Hard is It?
2. Simplifying Principle
9. Synchronization Solution
1. Synchronization Web Page
2. Overview of Synchronization Algorithm
3. The Three Perl Scripts
10. Conclusion

Summary

A Palm Pilot of the 1990’s was much better than any paper-based organizer; updates, searches, and backups all were far superior. Furthermore, Palm handled synchronization conflicts between the desktop and mobile device correctly, without losing any data.

But Palm, Inc. failed and I needed a replacement. To my great disappointment, no other solutions synchronized correctly, even though the newer devices were much more powerful. Furthermore, too many offerings were cloud-based systems that could not preserve privacy. Finally, I had to write my own synchronization software, built upon Git and the Vault 3 organizer.

My Palm Pilot Plan

I got my first Palm Pilot in late 1997. It was so much better than the paper planners that I had used since the mid-1980’s! Moving an item from one day to another was easy and it did not leave a messy note on the day it was originally scheduled. Old items could be searched electronically rather than found only by laborious, manual reading through numerous pages of paper notes. The Palm Pilot and computer desktop could be synchronized with the push of a button. Whereas paper records always suffered the risk of permanent loss if/when they got lost or destroyed, creating backups of electronic records was easy.

Furthermore, the technology was improving steadily. As the size of my PIM (Personal Information Management) records increased, the newer devices should be able to accommodate all of them. My records since late 1997 should always be accessible easily in the latest Palm Pilots. Eventually, I upgraded to a Palm IIIxe and then waited for technology to advance with my PIM needs.

It did not turn out as I expected. Palm, Inc. failed. The technology moved toward smart phones, which soon sported hardware powerful enough to handle all my PIM data, but the software moved in another direction. Many PIM-related products and services arose, but none could replace the original Palm Pilot applications, mainly because they all failed to handle merge conflicts correctly. For example, if the same calendar item was changed in two different devices, then the synchronization typically would clobber one version with the other version rather than noticing the conflict, saving both sets of changes, and then letting the user resolve the conflict.

Long-Term Problems with Palm IIIxe

One option was to continue using the old Palm IIIxe, but that was not a long-term solution.

• It runs on only old hardware (released in 2000) and old software (Palm Desktop) that no longer is supported. Both the hardware and software are “abandonware”.
• Its internal limits prevent it from storing my entire Datebook (Calendar), which has over 20,000 entries. Fortunately, the Palm Desktop can store more entries via archived data, but that leaves the Palm Pilot with an incomplete database.
• The data is in a proprietary format. (Fortunately, some enterprising people have reverse engineered most of it.)
• A datebook search is slow on the Palm IIIxe because (1) the old hardware is slow and (2) searches always are global, not just for the range of dates wanted.

Advantages of Nexus 7

For a platform to replace the Palm IIIxe, I chose a Nexus 7. It is much faster, has much more memory, greater screen resolution, WiFi connectivity, is much easier to program, has native support for SQLite, supports full disk encryption, and has a very good price. But what about the PIM software?

Requirements for PIM Software

Given a Nexus 7 mobile device and a desktop computer, in addition to the normal PIM-related operations, the software must support the following features.

• Reliability. Randomly dropping or corrupting data is not acceptable. (That is not a problem with Palm Pilots, when used within their data size limitations, but some later PIM products and services do suffer that problem.)
• Conflict resolution must be handled gracefully during synchronization. Except for Palm, apparently nobody gets this right. Even systems that claim to perform two-way merging of individual records, rather than just clobbering one entire database with another, still end up clobbering one record with another rather than merging the conflicting changes in that record.
• PIM data is personal data, not public data. There should be no plain-text hosting on another vendor's PIM web site (or even my own), since no vendor promises or security mechanisms can be trusted long-term. All PIM data must be encrypted. That alone eliminates almost all the cloud-based PIM services.
• Synchronization must work for PIM data in multiple, mobile and/or desktop devices, not just a single mobile device and a single desktop.
• No vendor lock-in. It must be easy to export all the PIM data to a non-proprietary format. Use PIM standards, if possible.
• It would be nice to retain access to previous versions of the PIM data. If/when an item gets lost or mangled, due to user error when making updates, that will provide a historical record for recovering it.
• Fine-grained control over sharing PIM data would be nice.

PIM Software Alternatives

PIM Applications and Services

• Absolute Privacy is a Windows desktop encrypted PIM application that claims to support a PDA version and synchronization via Microsoft Outlook. But I question the expertise of the author because of the claim that the “likelihood of the same key being used more than once in 50000 records is almost nil” due to its use of Thousands of Encryption Keys . The author apparently never heard of the “birthday paradox“.
• AnCal is a free, open source PalmOS Style calendar, tasks, and notes for Android. It lacked the polish that I wanted for an Android application and completely lacked a desktop version, so it would require a lot of programming to convert it to a suitable PIM application.
• CalCurse is a Linux/UNIX desktop application that supports export to the iCalendar format. It does not have any corresponding Android application.
• CompanionLink / DejaOffice was the most promising-looking alternative that I found. The free Android application (DejaOffice) worked well. The problem was that CompanionLink software for synchronizing the Android data with the Windows desktop had two major flaws: (1) it did not handle conflicts properly, since the changes on one device would clobber the changes on the other and (2) in my experience, it randomly and silently dropped records during the synchronization. While (1) is typical, (2) is completely unacceptable. (FYI: I tried CompanionLink synchronization with both Palm Desktop and Outlook on Windows.)
• Evolution is a Linux Gnome and Windows desktop application. Surprisingly, its search function worked only for a month at a time. But I needed to search many years of data, not just one month. There had been some work to synchronize Evolution with other devices, but since the desktop version was so clunky, it did not seem worth pursuing.
• Google Calendar is built into Android. It is a cloud-based solution with the expected privacy limitations and also lack of proper conflict resolution during synchronization. If one wanted to use it from a desktop, though, one could use either a web browser or install an application (such as BlueStacks) for running Android applications on a Windows desktop.
• Kontact is a Linux KDE application that I also was able to run in Ubuntu (Gnome, before Unity). It was less clunky than Evolution and may be acceptable as a stand-alone PIM application, but the synchronization options are underwhelming. (MemoToo sort-of supported synchronization with Kontact, but Kontact then would lose data.)
• MemoToo is a cloud-based solution that (attempts to) support synchronization with several kinds of desktop clients. It is a Funambol application, though, and cannot support proper conflict resolution during two-way synchronization; either the client version clobbers the MemoToo version or vice-versa. (SyncML) Of course, privacy of the data relies entirely on the security of the MemoToo server.
• Microsoft Outlook works well on the Windows desktop. It does keep insisting on being the preferred email client, though, even if you want to use something else for your email. (As noted above, CompanionLink claims to synchronize Outlook with DejaOffice on Android, but in my experience that was not reliable.) I have not investigated Microsoft’s Outlook.com cloud solution.
• OwnCloud supports PIM data, but it emphasizes file synchronization, not PIM data synchronization. One could host OwnCloud on a VPS, but then would need to administrate the VPS.
• RememberTheMilk is a cloud-based task list service that has API kits in several different languages (C, C#, Java, Lisp, Perl, Python, and Ruby). It has some support for synchronization; the “getList” method supports “last_sync” information. (I do not know if it supports proper conflict resolution during synchronization.) Creating tasks to simulate contacts, calendar items, and notes may be klutzy, though. The cloud-based system also suffers the usual privacy limitations.
• ToDoist is a cloud-based TODO list (task list) service. Mobile applications are available, but no API. The limitations otherwise are similar to those for RememberTheMilk.
• Toodledo is yet another cloud-based TODO list service. It does support an API and the site has a detailed comparison page. Otherwise, it is similar to the other cloud-based task list services.
• Yahoo Calendar is another cloud-based service. Unlike the other cloud-based services, it does not provide any details about the service until one opens an account. If you are still interested, check its Wikipedia page.

Outliners

Since none of the PIM products and services (above) satisfied the requirements, I switched to Outliner applications, which also can be used to support PIM data, with some limitations.

Advantages and Disadvantages of Outliners

Advantages

• A tree-based data structure has great flexibility. It can emulate most of the PIM data types easily and tree-based update operations are simple and intuitive.
• Outliner applications usually have a simple database schema, thereby reducing lock-in.

Disadvantages

• An outliner has no understanding of PIM data and therefore does not support any PIM standards such as iCalendar or vCalendar.
• Each date in the Calendar must be inserted manually.
• Repeating events are not handled automatically.
• For contact records, each field must be inserted manually as part of a multi-line text field.
• There is no automated sanity check for properly formatted data.

Examples of Outliner Applications

• Android Outliner is an Android app, with both a free version and a paid version. The paid version supports search, but not the free version. It understands due dates associated with entries in the tree. Both versions can import / export several desktop formats (Natara Bonsai, Treepad, and OPML format) via USB or DropBox, but only the paid version will synchronize with them. (My expectation is that it supports only file level synchronization, not record level and certainly not any graceful conflict resolution.)
• B-Folders offers a free Android app. As of July 2012, the Windows/Mac/Linux desktop application costs $30. The search always is a global search, although one can restrict the type of objects searched (tasks, journals, etc.). It has several options for repeating events. Synchronization between Android and the Linux desktop succeeded, but when there is a conflict, the version with the latest change always silently clobbers the other. In particular, if the latest conflicting change is to delete an item, the synchronization deletes it silently. The export mechanism works well and it supports encryption via the open source Bouncy Castle libraries. The Android version performance is poor for a large database (25,000 entries), though, resulting in a time out on my Nexus 7.
• Microsoft Office Word (Outliner Mode) is a nice desktop outliner, but I would not consider trying to synchronize it with an Android app because it is a Microsoft Office product.
• NoteLynX Pro Outliner is a paid Android app. (Just $2.38.) It claims to support cloud synchronization but not synchronization with any desktop outliner. (It says “Use on Desktop via BlueStacks”.) Its data structure is not limited to trees since a child can link to multiple parents. The cloud synchronization probably is file-level since it relies on services such as DropBox. (Developer’s website.) I do not have details on how well its search works or how well it performs with a large database.
• TreePad is a Windows desktop application. (It also can run under Linux-Wine.) There is a free version plus several tiers of paid versions. It supports import/export from/to several formats. It does not have an Android version but there is some synchronization capability between TreePad and Android Outliner.
• Vault 3 Outliner is an open source project written in Java, which has a free (Windows, Linux, Mac OS X) desktop version and also both free and paid Android apps. Both the desktop and Android versions quickly load, search, and update the same SQLite database format, even when it holds over 20,000 entries! Searches can be limited to a sub-tree. It supports 128-bit AES encryption, which operates on the content of individual database fields, not the entire database at once. Although no synchronization capability is built into the Vault 3 Outliner, the database schema is so simple that writing one looks feasible. Even if it did not include import/export capabilities, once could use a SQLite client to perform those functions, too.

No Existing PIM Software is Good Enough

This should be a solved problem already. The Palm Pilot did a credible job in the mid 1990’s and there are many PIM options available for the more powerful devices available today. However:

• None of the existing alternatives satisfy my requirements.
• Doing two-way synchronization correctly, without losing any changes even when there are conflicts, is not simple.
• Technology trends favor always-online, cloud-based systems.

For my purposes, an outliner provides the best platform for a PIM solution, but it requires some assistance since it was not designed to be used that way.

Building a PIM Application With Vault 3 and Git

Vault 3

Vault 3 is the best outliner application that I found.

• It is fast, on both Windows and Android, for display, search, and update.
• It uses a SQLite database, which is well-supported on many platforms and easily handles large databases, and, of course, supports SQL queries.

Admittedly, Vault 3 is the work of just one person; it does not have the support of an established large company. But even an established large company, such as Palm, Inc., can drop support and even fail completely. Fortunately, the database format is simple enough that it should not be difficult to port the Vault 3 data to another application if/when needed.

Git

With some additional software, Git can support synchronization of changes to a Vault 3 document among two or more (desktop or mobile) devices. It has a built-in capability to merge changes from separate branches. (Handling merge conflicts automatically is tricky, though. See the section on Synchronization below.) It also provides a version history, which a Palm Pilot never did.

Git does add some maintenance overhead. The repository eventually does get large, so periodic “git -gc” commands are needed to keep it from getting bloated. Furthermore, if you do many synchronizations, you may want to keep not only archived annual snapshots of the repository, but also a repository updated just once a month as well as the daily repository.

Porting PIM Data from Palm Pilot

Before I could use the PIM solution on my Nexus 7, I needed to port my old Palm Pilot data to Vault 3. The Palm Pilot had four main data types: (1) Datebook (Calendar), (2) Addressbook (Contacts), (3) TODO (Tasks), and (4) Memos. I found software for extracting a text dump of the Datebook. Palm Desktop can export the memos and address book entries in several formats, including some simple text formats. But the TODO items exported only to a TODO archive format. (Fortunately, I did not have many TODO items.) Once I had text representations for the Palm Pilot data, it was easy to write Perl scripts to transform it to a format that could be imported into Vault 3.

The PIM Synchronization Problem

How Hard is It?

How hard is it to synchronize the PIM data in two or more devices? Here is some background reading.

PIM Data Synchronization: Why is it so hard?

For an outliner application, synchronization requires a diff/merge solution for a tree structure, such as XML. A survey of potential XHTML diff/merge algorithms

Simplifying Principle

A simplifying principle is needed, given that the general tree merge problem is difficult, especially when there are merge conflicts. Fortunately, only merges of trees structured for Vault 3 must be supported, not the general tree merge problem. Also, a perfect merge is not needed. What is most important is:

1. preserve all the changes in a merge conflict so that no change is lost and
2. point the user to the existence and location of those merge conflicts so that the user can fix them manually in the Vault 3 GUI.

Synchronization Solution

Synchronization Web Page

The synchronization is not performed over any USB "conduits" connecting the desktop and mobile device. Instead, it is performed by scripts running on a web server on the desktop PC (but not in the cloud).

To perform a synchronization, do the following:

1. Start the web server on the desktop PC.
2. Access the URL for the PIM Synchronization page from your device.
3. Ensure that the correct device is selected in the dropdown list.
4. If any changes were made on the device since the last synchronization:
   • Click on the “Browse” button and select the Vault 3 SQLite file.
   • Wait for the synchronization to complete. Check for any messages about synchronization conflicts. Click on the link to download the merged Vault 3 SQLite file.
   • Move the downloaded SQLite file to the location that Vault 3 expects to find it on your device.
   • If there were any synchronization conflicts, resolve them manually and then rerun the synchronization.
5. If no changes were made on the device since the last synchronization, save time as follows:
   • Click on the “Clobber” button and agree to the “clobber” since no changes will be lost. (There is no need to click on the “Browse” button since no SQLite database file needs to be uploaded.)
   • Click on the link for downloading the latest Vault 3 SQLite file.
   • Move the downloaded SQLite file to the location that Vault 3 expects to find it on your device.
6. Stop the web server on the desktop PC.

Overview of Synchronization Algorithm

Black = master branch, Red = desktop branch, Green = mobile branch

The Git repository has a “master” branch and also a branch for each device. Rather than directly managing versions of the Vault 3 SQLite databases for each device, it manages versions of XML dumps of those Vault 3 SQLite databases.

The synchronization algorithm includes the following steps:

• Upload the Vault 3 SQLite database from the device to the synchronization web server.
• Generate an XML dump of the SQLite database for the device.
• Commit that XML file to the branch for that device.
• Merge the branch for that device with the master branch.
• If there are one or more merge conflicts:
  • Run a script to patch the XML file so that (1) it includes both versions of text in the merge conflict, (2) it has distinctive markers that show where the merge conflicts occurred, and (3) it also is formatted as a valid Vault 3 XML file.
  • Complete the Git merge, successfully this time.
• Reset the Git branch for the device to point to the current master branch.
• Convert the merged XML file to a Vault 3 SQLite database.
• Return a download link for the new Vault 3 database for the device.

The Three Perl Scripts

The synchronization software is a combination of HTML, JavaScript, PHP, Perl, and Git bash scripts. Here is an overview of the three Perl scripts that the synchronization algorithm (above) uses:

1. sqlitedb2xml.pl
   Dump a Vault 3 SQLite database to a pretty-printed XML file, using a depth-first pre-order tree traversal. It is formatted so that it can be imported back into Vault 3. That is the XML format saved in the Git repository.
2. sqlitexml2db.pl
   Convert a Vault 3 XML file into a Vault 3 SQLite database. (Yes, Vault 3 can import the XML file in the GUI, but I wanted a fully automated solution.)
3. conflict_resolve.pl
   Analyze a Vault 3 XML file with Git merge conflict markers and convert it to another XML file that (1) includes both versions of text in the merge conflict, (2) has distinctive markers that show where the merge conflicts occurred, and (3) also is formatted as a valid Vault 3 XML file. (Note: Git does provide support for a custom merge algorithm that can be called directly from Git, but that proved difficult to incorporate, so this script is run only after a Git merge fails with one or more merge conflicts.)

Conclusion

Although wonderful advances were made in mobile devices since the 1990’s, an important feature of the original Palm Pilots was lost. This project restored the ability to synchronize two or more devices without losing any data when there is a synchronization conflict. It accomplishes that with a modern device (Nexus 7 tablet) and without relying on a vendor offering a cloud-based solution.