Table of Contents
NLNet Grant 3
In November 2014, The Serval Project commenced a third round of work for the NLnet Foundation to build and demonstrate the first functional prototype of the Serval Chat app for iOS.
From the grant proposal:
Deliver a demonstration-quality iPhone app that has private, one-to-one messaging and rudimentary contact management.
Work items:
Streams
PROGRESS
W1 - COMPLETED
- In early September 2014, the Serval Project acquired two new Apple MacBook Pros and five used iPhones.
- In September 2014, senior developer Andrew Bettison attended a one-day iOS developer course for Flinders University staff, “Introduction to iOS Development using Swift”.
- During October, November and December 2014, senior developers installed Xcode, and created and installed rudimentary “Hello World” iOS apps.
W2 - COMPLETED
- During February-July 2015, visiting student Imane Rai from INSA Lyon worked on a prototype iOS app, supervised by senior developer Andrew Bettison:
- design of UI screens and transitions: First prototype
- code on GitHub: https://github.com/servalproject/servalchat-ios-prototype.git
W3 - IN PROGRESS
- In April 2015, senior developer Jeremy Lakeman started work on making the Serval DNA daemon process run as a single thread within a multi-threaded Java application. This work is also directly applicable to the iOS port, since iOS does not allow applications to call fork(2) or create long-running processes, so the daemon must be run as a thread.
- In May 2015, senior developer Andrew Bettison improved the performance of the HTTP REST API operation
GET /restful/meshms/<toSID>/<fromSID>/newsince/<token>/messagelist.json
:- created a generic “trigger” mechanism within Serval DNA that removes the need to poll for events:
- instead of the HTTP request polling the Rhizome store to detect new content, it now suspends and is woken by the “new bundle” trigger whenever new content is added to the Rhizome store, giving lower battery drain and slightly faster message arrival times (no inter-poll delay to report a new incoming message)
- In October-December 2015, senior developer Andrew Bettison formally documented the HTTP REST API to support efforts by third party contributors and to fix the remaining grey areas in the specification and implementation of the API:
- the resulting Technical Documentation is available on GitHub:
- see REST-API.md
- the commits that created and revised this document are:
- serval-dna commit e189bcf created the original document
- serval-dna commit 419364b formalised the HTTP response status codes and CORS support
- serval-dna commit 3c993f0 formalised bundle author deduction
- numerous issues were resolved in Rhizome, the most major being:
- unhelpful mapping from Rhizome operation results to HTTP response status codes:
- serval-dna commit 6123503 improved some status codes:
- replaced 403 “Forbidden” with 404 “Not Found”, 422 “Unprocessable Entity”, 423 “Locked”, 429 “Too Many Requests”
- insert responds with 202 “Accepted” for out-of-space and already-expired, since these are not strictly errors
- serval-dna commit 419364b improved the remaining status codes:
- replaced all remaining uses of 403 “Forbidden” with 415 “Unsupported Media”, 414 “Request-URI Too Long” and 419 “Authentication Timeout”
- now the daemon only responds 403 for requests that do not originate from the local host
- inadequate bundle author identification from the
GET /restful/rhizome/bundlelist.json
request:- serval-dna commit 3c993f0 added a new value in the
fromhere
column
- inadequate diagnostic messages in many failure cases:
- serval-dna commit 98ed040 refactored the Rhizome result handling to make it easy to return a diagnostic message from most contexts
- serval-dna commit 3f8f0f6 improved Rhizome diagnostics for HTTP clients:
- added a “Manifest Too Big” result code
- returned messages in the HTTP response that previously only went into the daemon's log files
- In December 2015 - February 2016, senior developer Andrew Bettison resolved the following issues on OS-X 10.11 El Capitan:
- some Autoconf M4 macros were missing, so the build process failed on hosts that did not provide them:
- serval-dna commit f6c2706 added these macros, with some local modifications
- serval-dna commit e51189a removed the redundant macro cache file and updated the standard build instructions
- the standard build procedure gave link errors on OS-X:
- serval-dna commit ae18d76 fixed these link errors
- the behaviour of the
recvmsg(2)
system call on BSD appears to differ from Linux, and the resultant missing-nul-terminator bug caused most routing, MDP and MSP tests to fail on OS-X:- serval-dna commit c3375d0 fixed the bug and factored away some redundant code
- some test cases depend on GNU grep, GNU sed and GNU awk, but OS-X supplies the BSD variants of these utilities:
- serval-dna commit 0361b99 fixed the tests to use the GNU variants and adds developer instructions for installing them
- In September-October 2016, senior developer Andrew Bettison resolved the following issues on OS-X 10.11 El Capitan:
- Servald DNA had acquired a new dependency on the externally-built package
libsodium
:- serval-dna commit c45e92b and fc8fce5 added a new
build-libsodium.sh
script and updated the build instructions - serval-dna commit 7e39698 (and others) improved the documentation for developers
- fixed some non-deterministic test failures that had appeared since February:
- serval-dna commit 43bd0bd fixed a keyring test
- serval-dna commit 7f0fef2 fixed a server test
- serval-dna commit 298b83f fixed a keyring restful test
- serval-dna commit 94c58e8 and 5191d42 fixed a server start pidfile race condition
- serval-dna commit 6627c86 fixed a log file rotation race condition
- fixed some recently-added test cases that were failing on OS-X:
- serval-dna commit 21679e3 fixed a rhizomeprotocol test
- serval-dna commit 0657e27 fixed a meshms restful test
- prepared for an iOS build of the Serval DNA daemon static library:
- serval-dna commit e34141a upgraded to the latest version of SQLite (3.14.2)
- modernised the build system:
- serval-dna commit 4b20605, 29a771e and a72f6ae prepared the autoconf Makefile to be invoked on OS-X when cross-compiling to many iOS architectures (“slices”)
- serval-dna commit 192fefd updated the documentation for developers
- serval-dna commit 1efe668 moved configured build-time settings into
config.h
- removed Java interface (JNI) code from all source files that will be compiled on iOS:
- serval-dna commit c8bf8a7 moved JNI code from the CLI and server main loop into separate, Java-only source files
- serval-dna commit 42f62d2 moved JNI code from the instance path into a separate, Java-only source file
- serval-dna commit 5191d42 fixed the build on Android
- introduced “feature-driven” linking to support linking an executable from a static library:
- serval-dna commit 6dd823a renamed a header file to reduce confusion
- serval-dna commit 8cb3afa and 6201b97 fixed a two-way dependency problem
- serval-dna commit 71cbe86 added the new
“feature.h”
mechanism - serval-dna commit 8325aac fixed the build on Android
W7 - IN PROGRESS
- In July 2014, student Tobias Wooldridge built a preliminary version of the Keyring REST API:
- serval-dna commit e79e74f created the
/restful/keyring/identities.json
operation for querying existing keyring identities
- In August 2015, senior developer Andrew Bettison added missing operations and support for PINs (passwords) to the Keyring REST API:
- serval-dna commit 586c6b3 created the
/restful/keyring/add
operation - serval-dna commit 73ced0f added the
pin
parameter to the add operation - serval-dna commit 380a721 added the
pin
parameter to the identities operation - serval-dna commit 8ba612f created the
/restful/keyring/SID/set
operation - serval-dna commit 714ea63 improved the JSON output of keyring operations
- In November 2015, senior developer Andrew Bettison documented the Keyring REST API:
- serval-dna commit e189bcf created the original document
- serval-dna commit bd45186 documented Serval ID and identity lock/unlock
- In February 2016, senior developer Andrew Bettison expanded the existing REST API Technical Documentation to define the new Peer List REST API:
- serval-dna commit 62ce14f split the REST-API.md document into separate files per service:
- In November 2016, senior developer Andrew Bettison finished implementing the Keyring REST API:
- serval-dna commit a8e394d created the
/restful/keyring/SID/remove
operation and added more detail to the API documentation
- In November 2016, senior developer Andrew Bettison created a Keyring API for the Swift 3 language:
- serval-dna commit 24266b5 added
api.restful.authorization
config option to support testing on Linux
WORK ITEMS
W1. [Front-end, Back-end] Commission hardware and set up iOS developer environment
Developers set up and familiarise themselves with the iOS development tools. This also includes the time to source the used iPhone and Apple Mac Book Pro laptops and install OS-X Mavericks, Xcode and other development tools.
- Deliverables: Statements of readiness by developers, accompanied by trivial iOS applications developed during the activity.
W2. [Front-end] Minimal UI design and prototype
Design the UI for one-to-one private messaging app based on existing back-end functions (MeshMS and Peer List), with five screens:
- On-boarding (first use) screen, where the user can set a displayed name
- “Contacts” screen displaying existing contacts (private conversations)
- “Conversation” screen, displaying a single, conversation thread of private text messages to and from one other person, with input box for entering and sending a new message
- “New Contact” screen with three tabs: “Requests”, “Nearby” and “Blocked” each listing identities by name
- “Add Contact” screen for editing the name associated with an identity, and choosing to save or block the contact
Create a UI prototype app that encapsulates the whole visual design with static mocked-up data on each screen to allow scrolling, tabs, data entry and notifications to be exercised in the absence of back-end functionality. This will allow visual and usability design issues to be spotted and corrected before the fully functional app is ready.
This prototype app will evolve into the final app when work item 8 connects the back-end functions.
- Deliverable: a visually attractive one-to-one private messaging UI prototype iOS application that allows interaction with the five screens populated with mock data (“lorem ipsum”), but with no back-end functionality.
W3. [Back-end] Finish iOS port of Serval DNA
Finish the integration of Serval DNA into an iOS application, continuing the work of an independent iOS developer in early 2014. Resolve iOS build issues in a sustainable fashion, ensure that the most recent version of Serval DNA builds on iOS, runs properly and passes all integration test scripts.
- Deliverable: an development branch of Serval DNA that builds on both OS-X and iOS, build instructions in INSTALL.md, all test scripts pass on OS-X.
W4. [Back-end] Achieve MDP ping between iOS apps
Investigation of the iOS Multipeer Connectivity Framework suggests that the most effective approach to implementing Rhizome sync will be to implement Mesh Datagram Protocol (MDP) and selectively disable parts of it.
Integrate Serval DNA as the background thread of an experimental iOS app that sends periodic MDP ping packets to all of its visible peers and reports the ping replies on the screen. This will involve setting iOS mDNS to announce the Serval Identity (SID) and replacing the Serval route detection packets on peer-to-peer links with logic to populate the routing table from the other visible mDNS peers (one hop).
Find the best way to access the Serval DNA logs from the developer laptop, which may involve integrating Serval DNA's logging with the iOS logging API.
- Deliverable: an experimental iOS application that successfully MDP pings its nearby peers.
W5. [Back-end] Achieve Rhizome Sync between iOS apps
Once MDP packets are successfully exchanged between iOS apps (work item 4), implement Rhizome sync between iOS apps. This will involve suppressing unnecessary MDP packets (such as periodic route discovery) and re-working the payload transfer to use the Multipeer Connectivity Framework reliable stream protocol instead of the current sliding window MDP retransmission.
Test by temporarily allowing the experimental app's Rhizome API to accept requests from the network (instead of localhost), then make HTTP requests from the developer laptop to add bundles to Rhizome on one iPhone and to list the bundles on other iPhones.
- Deliverable: an experimental iOS application that automatically synchronises all Rhizome contant with nearby peers.
W6. [Back-end] Persistent contact data
Modify the existing MeshMS conversation-list bundle, which persists in the Rhizome store, to contain the user's contact list. At present every MeshMS conversation records only the Serval Identity (SID) and the read/received byte offsets for every conversation. We would supplement them with:
- tri-state flag: whether the contact has been accepted, blocked, or neither
- the contact's published Name at the time the contact was accepted
- ancilliary data, eg, “last seen” and “last used” times, to help sort the contact list by favourite
The new contact data record format will be extensible (length-delimited type-tagged records) to ensure that all future versions of the app can be made back- and forward-compatible with each other. A new MeshMS API operation will assign the per-conversation extensible data. Existing MeshMS API queries will retrieve the extensible per-conversation data.
- Deliverable: a new MeshMS API in Serval DNA to store and retrieve contact list data, with working integration tests.
Temporarily modify the MeshMS protocol to add a name field to the Rhizome manifest of all outgoing MeshMS ply bundles. This will make it possible to display the sender's name on the “Request” tab of the “New Contact” screen when the sender is not currently in the peer list. This modification will eventually be scrapped when the Identity Card layer is implemented.
- Deliverable: a modified MeshMS protocol that sets the names field of ply manifests, with working integration tests.
W7. [Back-end] Peer List and Keyring APIs
Serval DNA's recently developed REST HTTP server already provides a Rhizome API and MeshMS API. New APIs for the Peer List and Keyring are needed:
- implement a new REST request to report the current list of peers reachable on the network, including their Name, as such data becomes available
- implement a new set of REST requests to manage the Keyring, allowing an HTTP client to add an identity, set the Name of an identity, list identities, and remove an identity.
Both these APIs already exist in the Serval DNA command-line API, so this work item can progress independently of other work items.
- Deliverable: a new REST HTTP API in Serval DNA to query the current network peers, with working integration tests.
- Deliverable: a new REST HTTP API in Serval DNA to manage the local keyring, with working integration tests.
W8. [Front-end] Demonstrable app for one-to-one private messaging
Once the new back-end Peer List and Keyring APIs are complete, modify the UI prototype app developed in work item 2 to use the back-end HTTP REST APIs to populate the screens as follows:
- the “Contacts” screen from persistent contact data, sorting contacts with unread incoming messages to the top, then in chronological order of recent messages sent or received
- the “Nearby” tab of the “New Contact” screen from the peer list if not marked as blocked, updated as peers enter and leave
- the “Request” tab of the “New Contact” screen using incoming messages from contacts not marked as blocked or accepted
- the “Blocked” tab of the “New Contact” screen from contacts marked as blocked
- the “Conversation” screen from a single MeshMS conversation, updated as new messages arrive
The following user actions will invoke the HTTP REST API:
- store the name that the user enters on the On-boarding screen in the keyring
- store the name and accept/block decision that the user enters on the “Add Contact” screen
- accept and block contacts that are visible on the “Nearby” tab of the “New Contact” screen
- block contacts that are shown on the “Contacts” screen
- un-block and accept contacts that are shown on the “Blocked” tab of the “New Contact” screen
- send new text messages entered on the “Conversation” screen
The app will raise notifications when receiving events via HTTP REST API:
- whenever a new message is received, the notification will open the “Conversation” screen if the contact is already accepted, or the “Requests” tab of the “New Contact” screen if the contact is not accepted and not blocked
Test the app without needing Serval mesh network connectivity, by using the existing Serval DNA command-line interface to side-load data via the HTTP REST API.
- Deliverable: a iOS app for one-to-one private messaging, with all screens and controls functional, fulfilling the objectives of the grant.