Table of Contents

NLNet Grant 3

In November 2014, The Serval Project commenced a third round of work for the NLnet Foundation to build and demonstrate the first functional prototype of the Serval Chat app for iOS.

From the grant proposal:

Deliver a demonstration-quality iPhone app that has private, one-to-one messaging and rudimentary contact management.

Work items:

Streams

  • Front-end is concerned with iOS and the user experience: app data persistence, human interaction, visual presentation, graphic design, internationalisation and localisation for different human languages.
  • Back-end is concerned with Serval DNA: integration into iOS, network protocols, Rhizome data transport and persistence, encryption, battery use and performance, and infrastructure like internal configuration and logging.

PROGRESS

W1 - COMPLETED

  • In early September 2014, the Serval Project acquired two new Apple MacBook Pros and five used iPhones.
  • In September 2014, senior developer Andrew Bettison attended a one-day iOS developer course for Flinders University staff, “Introduction to iOS Development using Swift”.
  • During October, November and December 2014, senior developers installed Xcode, and created and installed rudimentary “Hello World” iOS apps.

W2 - COMPLETED

W3 - IN PROGRESS

  • In April 2015, senior developer Jeremy Lakeman started work on making the Serval DNA daemon process run as a single thread within a multi-threaded Java application. This work is also directly applicable to the iOS port, since iOS does not allow applications to call fork(2) or create long-running processes, so the daemon must be run as a thread.
  • In May 2015, senior developer Andrew Bettison improved the performance of the HTTP REST API operation GET /restful/meshms/<toSID>/<fromSID>/newsince/<token>/messagelist.json:
    • created a generic “trigger” mechanism within Serval DNA that removes the need to poll for events:
    • instead of the HTTP request polling the Rhizome store to detect new content, it now suspends and is woken by the “new bundle” trigger whenever new content is added to the Rhizome store, giving lower battery drain and slightly faster message arrival times (no inter-poll delay to report a new incoming message)
  • In October-December 2015, senior developer Andrew Bettison formally documented the HTTP REST API to support efforts by third party contributors and to fix the remaining grey areas in the specification and implementation of the API:
    • the resulting Technical Documentation is available on GitHub:
    • numerous issues were resolved in Rhizome, the most major being:
      • unhelpful mapping from Rhizome operation results to HTTP response status codes:
        • serval-dna commit 6123503 improved some status codes:
          • replaced 403 “Forbidden” with 404 “Not Found”, 422 “Unprocessable Entity”, 423 “Locked”, 429 “Too Many Requests”
          • insert responds with 202 “Accepted” for out-of-space and already-expired, since these are not strictly errors
        • serval-dna commit 419364b improved the remaining status codes:
          • replaced all remaining uses of 403 “Forbidden” with 415 “Unsupported Media”, 414 “Request-URI Too Long” and 419 “Authentication Timeout”
          • now the daemon only responds 403 for requests that do not originate from the local host
      • inadequate bundle author identification from the GET /restful/rhizome/bundlelist.json request:
      • inadequate diagnostic messages in many failure cases:
        • serval-dna commit 98ed040 refactored the Rhizome result handling to make it easy to return a diagnostic message from most contexts
        • serval-dna commit 3f8f0f6 improved Rhizome diagnostics for HTTP clients:
          • added a “Manifest Too Big” result code
          • returned messages in the HTTP response that previously only went into the daemon's log files
  • In December 2015 - February 2016, senior developer Andrew Bettison resolved the following issues on OS-X 10.11 El Capitan:
    • some Autoconf M4 macros were missing, so the build process failed on hosts that did not provide them:
    • the standard build procedure gave link errors on OS-X:
    • the behaviour of the recvmsg(2) system call on BSD appears to differ from Linux, and the resultant missing-nul-terminator bug caused most routing, MDP and MSP tests to fail on OS-X:
    • some test cases depend on GNU grep, GNU sed and GNU awk, but OS-X supplies the BSD variants of these utilities:
  • In September-October 2016, senior developer Andrew Bettison resolved the following issues on OS-X 10.11 El Capitan:

W7 - IN PROGRESS

WORK ITEMS

W1. [Front-end, Back-end] Commission hardware and set up iOS developer environment

Developers set up and familiarise themselves with the iOS development tools. This also includes the time to source the used iPhone and Apple Mac Book Pro laptops and install OS-X Mavericks, Xcode and other development tools.

  • Deliverables: Statements of readiness by developers, accompanied by trivial iOS applications developed during the activity.

W2. [Front-end] Minimal UI design and prototype

Design the UI for one-to-one private messaging app based on existing back-end functions (MeshMS and Peer List), with five screens:

  1. On-boarding (first use) screen, where the user can set a displayed name
  2. “Contacts” screen displaying existing contacts (private conversations)
  3. “Conversation” screen, displaying a single, conversation thread of private text messages to and from one other person, with input box for entering and sending a new message
  4. “New Contact” screen with three tabs: “Requests”, “Nearby” and “Blocked” each listing identities by name
  5. “Add Contact” screen for editing the name associated with an identity, and choosing to save or block the contact

Create a UI prototype app that encapsulates the whole visual design with static mocked-up data on each screen to allow scrolling, tabs, data entry and notifications to be exercised in the absence of back-end functionality. This will allow visual and usability design issues to be spotted and corrected before the fully functional app is ready.

This prototype app will evolve into the final app when work item 8 connects the back-end functions.

  • Deliverable: a visually attractive one-to-one private messaging UI prototype iOS application that allows interaction with the five screens populated with mock data (“lorem ipsum”), but with no back-end functionality.

W3. [Back-end] Finish iOS port of Serval DNA

Finish the integration of Serval DNA into an iOS application, continuing the work of an independent iOS developer in early 2014. Resolve iOS build issues in a sustainable fashion, ensure that the most recent version of Serval DNA builds on iOS, runs properly and passes all integration test scripts.

W4. [Back-end] Achieve MDP ping between iOS apps

Investigation of the iOS Multipeer Connectivity Framework suggests that the most effective approach to implementing Rhizome sync will be to implement Mesh Datagram Protocol (MDP) and selectively disable parts of it.

Integrate Serval DNA as the background thread of an experimental iOS app that sends periodic MDP ping packets to all of its visible peers and reports the ping replies on the screen. This will involve setting iOS mDNS to announce the Serval Identity (SID) and replacing the Serval route detection packets on peer-to-peer links with logic to populate the routing table from the other visible mDNS peers (one hop).

Find the best way to access the Serval DNA logs from the developer laptop, which may involve integrating Serval DNA's logging with the iOS logging API.

  • Deliverable: an experimental iOS application that successfully MDP pings its nearby peers.

W5. [Back-end] Achieve Rhizome Sync between iOS apps

Once MDP packets are successfully exchanged between iOS apps (work item 4), implement Rhizome sync between iOS apps. This will involve suppressing unnecessary MDP packets (such as periodic route discovery) and re-working the payload transfer to use the Multipeer Connectivity Framework reliable stream protocol instead of the current sliding window MDP retransmission.

Test by temporarily allowing the experimental app's Rhizome API to accept requests from the network (instead of localhost), then make HTTP requests from the developer laptop to add bundles to Rhizome on one iPhone and to list the bundles on other iPhones.

  • Deliverable: an experimental iOS application that automatically synchronises all Rhizome contant with nearby peers.

W6. [Back-end] Persistent contact data

Modify the existing MeshMS conversation-list bundle, which persists in the Rhizome store, to contain the user's contact list. At present every MeshMS conversation records only the Serval Identity (SID) and the read/received byte offsets for every conversation. We would supplement them with:

  • tri-state flag: whether the contact has been accepted, blocked, or neither
  • the contact's published Name at the time the contact was accepted
  • ancilliary data, eg, “last seen” and “last used” times, to help sort the contact list by favourite

The new contact data record format will be extensible (length-delimited type-tagged records) to ensure that all future versions of the app can be made back- and forward-compatible with each other. A new MeshMS API operation will assign the per-conversation extensible data. Existing MeshMS API queries will retrieve the extensible per-conversation data.

  • Deliverable: a new MeshMS API in Serval DNA to store and retrieve contact list data, with working integration tests.

Temporarily modify the MeshMS protocol to add a name field to the Rhizome manifest of all outgoing MeshMS ply bundles. This will make it possible to display the sender's name on the “Request” tab of the “New Contact” screen when the sender is not currently in the peer list. This modification will eventually be scrapped when the Identity Card layer is implemented.

  • Deliverable: a modified MeshMS protocol that sets the names field of ply manifests, with working integration tests.

W7. [Back-end] Peer List and Keyring APIs

Serval DNA's recently developed REST HTTP server already provides a Rhizome API and MeshMS API. New APIs for the Peer List and Keyring are needed:

  • implement a new REST request to report the current list of peers reachable on the network, including their Name, as such data becomes available
  • implement a new set of REST requests to manage the Keyring, allowing an HTTP client to add an identity, set the Name of an identity, list identities, and remove an identity.

Both these APIs already exist in the Serval DNA command-line API, so this work item can progress independently of other work items.

  • Deliverable: a new REST HTTP API in Serval DNA to query the current network peers, with working integration tests.
  • Deliverable: a new REST HTTP API in Serval DNA to manage the local keyring, with working integration tests.

W8. [Front-end] Demonstrable app for one-to-one private messaging

Once the new back-end Peer List and Keyring APIs are complete, modify the UI prototype app developed in work item 2 to use the back-end HTTP REST APIs to populate the screens as follows:

  • the “Contacts” screen from persistent contact data, sorting contacts with unread incoming messages to the top, then in chronological order of recent messages sent or received
  • the “Nearby” tab of the “New Contact” screen from the peer list if not marked as blocked, updated as peers enter and leave
  • the “Request” tab of the “New Contact” screen using incoming messages from contacts not marked as blocked or accepted
  • the “Blocked” tab of the “New Contact” screen from contacts marked as blocked
  • the “Conversation” screen from a single MeshMS conversation, updated as new messages arrive

The following user actions will invoke the HTTP REST API:

  • store the name that the user enters on the On-boarding screen in the keyring
  • store the name and accept/block decision that the user enters on the “Add Contact” screen
  • accept and block contacts that are visible on the “Nearby” tab of the “New Contact” screen
  • block contacts that are shown on the “Contacts” screen
  • un-block and accept contacts that are shown on the “Blocked” tab of the “New Contact” screen
  • send new text messages entered on the “Conversation” screen

The app will raise notifications when receiving events via HTTP REST API:

  • whenever a new message is received, the notification will open the “Conversation” screen if the contact is already accepted, or the “Requests” tab of the “New Contact” screen if the contact is not accepted and not blocked

Test the app without needing Serval mesh network connectivity, by using the existing Serval DNA command-line interface to side-load data via the HTTP REST API.

  • Deliverable: a iOS app for one-to-one private messaging, with all screens and controls functional, fulfilling the objectives of the grant.

Login