The Serval Project Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== New America Foundation - Contractor Agreement #31A1D0T1CY14 “NAF4” ======

In November 2013, [[:content:about]] commenced a fourth round of work for the [[http://www.newamerica.net/|New America Foundation]]'s [[http://oti.newamerica.net/|Open Technology Institute]] to develop a [[http://en.wikipedia.org/wiki/Representational_state_transfer|RESTful]] [[http://en.wikipedia.org/wiki/Api|API]] for [[:content:tech:Rhizome]].  The aim is to allow Commotion Wireless and other third party Android apps to use Serval's secure, authenticated and non-centralised file distribution.

**THE FOLLOWING DESCRIPTION OF WORK DOES NOT SPECIFY THE CURRENT RHIZOME OR MESHMS REST APIS.  SEE [[:content:tech:Rhizome API|RHIZOME API]] AND [[:content:tech:MeshMS API|MESHMS API]] FOR THE UP-TO-DATE DEFINITIONS.**

===== Section 1: Scope of Work =====

1. Extend the [[:content:tech:Rhizome]] C API within [[:content:tech:Serval DNA]] to support additional interconnected applications, implementing the deliverables listed below:

  * <BOOKMARK:R1a>**R1(a)**. Implement an access control scheme, using [[http://en.wikipedia.org/wiki/Basic_access_authentication|HTTP Basic authentication]] in the first instance, that controls access to the following APIs, identified in subsequent deliverable items by their URL. A password and permissions file will be used to record the authorities given to each password. Requests will only be accepted on the loopback interface.
  * <BOOKMARK:R2>**R2**. Implement URL ''localhost:4110/restful/rhizome/bundlelist.json'' in servald that returns the list of all Rhizome bundles as JSON formatted text. The list will include a token that can be supplied to ''GET /restful/rhizome/newsince/<token>/bundlelist.json''. The token should be considered an opaque symbol, whose format may vary between implementations and instances of servald (see technical notes **[[#N1]]**, **[[#N2]]**, **[[#N3]]** and **[[#N4]]**).
  * <BOOKMARK:R3>**R3**. Implement URL ''localhost:4110/restful/rhizome/newsince/<token>/bundlelist.json'' in servald that returns the list of all Rhizome bundles added or updated after the Rhizome database state indicated by <token> as JSON formatted text. The token should be considered an opaque symbol, whose format may vary between implementations and instances of servald (see **[[#N2]]**). The request will block for up to 60 seconds until results are available, making this request suitable for efficient polling for new bundles (see **[[#N5]]**). If no results are available the request will eventually timeout and complete with HTTP response code 204 and no data (see technical variation **[[#V1]]**).
  * <BOOKMARK:R4>**R4**. Implement URL ''localhost:4110/restful/rhizome/<bundleID>/manifest.bin'' (see **[[#V2]]**) in servald that returns the manifest of the specified bundle.
  * <BOOKMARK:R5>**R5**. Implement URL ''localhost:4110/restful/rhizome/<bundleID>/raw.bin'' in servald that returns the raw associated file of the specified bundle. If the bundle is encrypted, it is the encrypted file that will be returned.
  * <BOOKMARK:R6>**R6**. Implement URL ''localhost:4110/restful/rhizome/<bundleID>/decrypted.bin'' in servald that returns the associated file of the specified bundle, if possible. If the bundle is encrypted, it is the decrypted file that will be returned. If it is not possible to decrypt the bundle, respond with HTTP response code 412.
  * <BOOKMARK:R7>**R7**. Implement URL ''localhost:4110/restful/rhizome/insert'' in servald that inserts the bundle manifest and file provided through the POST request (see **[[#N7]]**).
  * <BOOKMARK:R8>**R8**. Merge new MeshMS code into servald. Currently MeshMS is implemented in the batphone Java code, and considerable work has been undertaken to remedy this.
  * <BOOKMARK:R9>**R9**. Implement URL ''localhost:4110/restful/meshms/<SID>/conversationlist.json'' in servald that returns the list of all MeshMS conversations as JSON formatted text.
  * <BOOKMARK:R10>**R10**. Implement URL ''localhost:4110/restful/meshms/<toSID>/<fromSID>/messagelist.json'' that returns the list of messages between the two parties as JSON formatted text (see **[[#N10]]**, **[[#N12]]** and **[[#N13]]**). Each message will be identified by a unique token. The tokens should be considered opaque symbols, whose format may vary between implementations and instances of servald (see **[[#N11]]**). Each message will also be accompanied by a delivery status of either unacknowledged, delivered (for messages sent by this node), or received (for messages sent by the remote party) (see technical variation **[[#V3]]**).
  * <BOOKMARK:R11>**R11**. Implement URL ''localhost:4110/restful/meshms/<toSID>/<fromSID>/newsince/<token>/messagelist.json'' that returns the list of messages between the two parties as JSON formatted text (see **[[#N10]]**, **[[#N12]]** and **[[#N13]]**). Only messages newer than the message corresponding to the supplied token (see **[[#N11]]**) will be returned. The request will block for up to 60 seconds until results are available (see **[[#N14]]** and **[[#N15]]**), making this request suitable for efficient polling for new bundles. If no results are available the request will eventually timeout and complete with HTTP response code 204 and no data (see technical variations **[[#V1]]** and **[[#V3]]**).
  * <BOOKMARK:R12>**R12**. Implement URL ''localhost:4110/restful/meshms/<toSID>/<fromSID>/sendmessage/<toDID>/<fromDID/<URL-coded-message-text>'' that attempts to send a MeshMS message. Total URL length is limited to 1000 characters. Message body must be representable as a UTF-8 string (see technical variations **[[#V3]]**, **[[#V4]]** and **[[#V5]]**).
  * <BOOKMARK:R13>**R13**. Create a cross-platform Java library that provides bindings to each of the APIs introduced above (see **[[#N16]]**).
  * <BOOKMARK:R14>**R14**. Expand servald test suite to cover the above APIs and Java library.
  * <BOOKMARK:R15>**R15**. Reimplement the MeshMS interface in the Serval Mesh Android App to use the Java library (see **[[#N16]]**).

==== Technical variations ====
The following variations to requirements were made during the course of the contract without prior consultation because they did not alter the intent or functionality of the original requirements.

  * <BOOKMARK:V1>**V1**.  In **[[#R3]]** and **[[#R11]]**, instead of returning 204 No Data, the ''GET'' request instead returns 200 OK and a JSON object with a "header" line and an empty "rows" array.
  * <BOOKMARK:V2>**V2**.  In **[[#R4]]**, the URL was changed to ''/restful/rhizome/<bundleID>.rhm'' so that applications which store the downloaded manifest will use ''<BundleID>.rhm'' as the default file name.  The RHM file extension was chosen because a manifest file is not strictly a text file (TXT), since the text portion is followed by a NUL byte and the binary signature block.
  * <BOOKMARK:V3>**V3**.  In **[[#R10]]**, **[[#R11]]** and **[[#R12]]**, the orders of the SIDs in the URL were reversed, eg, ''/restful/meshms/<fromSID>/<toSID>/messagelist.json'' to make the RESTful URL name space more regular; ie, one "sub-directory" per local identity (''<fromSID>'')
  * <BOOKMARK:V4>**V4**.  In **[[#R12]]**, the DID fields were removed from the URL, since they are not carried in MeshMS2 messages (they were a legacy from the MeshMS1 design)
  * <BOOKMARK:V5>**V5**.  In **[[#R12]]**, in order to respect RESTful architectural style, the request was changed from GET to POST and the text of the message supplied as a POST body parameter.  This, together with **[[#V3]]** and **[[#V4]]** changes the URL path to ''/restful/meshms/<fromSID>/<toSID>/sendmessage''

==== Technical notes ====
The following implementation decisions were made during the course of the contract.

  * <BOOKMARK:N1>**N1**.  In **[[#R2]]** and **[[#R3]]**, the JSON output of ''GET /restful/rhizome/bundlelist.json'' and ''GET /restful/rhizome/newsince/<token>/bundlelist.json'' has the following structure:<code>
{
    "header":[".token","_id","service","id","version","date",".inserttime",".author",".fromhere","filesize","filehash","sender","recipient","name"],
    "rows":[
        ["-luCgeURRKqvbRKQilANXwcAAAAAAAAA",7,"file","F81645C6693A98EB4D2727A7A3D9F478FF77CC60F68949679BFD2A7C65A29E89",1384921558580,1384921558581,1384921558648,"A8BFD18D5BF6E005E96E62188F553F0149B10AD03B47B7A35181457B2A1ABF69",1,1006,"A49C795AE4687CB14F6F1F01265C1DC6472935B125F52ADE4F166A31770E508AF2E0AF9F48D9F6826D60C7B5B820C63028EFD78AD32B9EBA5E43A2B60D74C9E8",null,null,"file6"],
        ...
    ]
}
</code>This is more compact than the most convenient JSON representation, which would be an array of JSON objects.  An array of JSON objects would redundantly repeat all the header labels within every single object.  The chosen representation can easily be transformed into an array of JSON objects.  The test script depends on version 1.3 of the [[http://stedolan.github.io/jq/|jq(1)]] utility to perform this transformation.
  * <BOOKMARK:N2>**N2**.  The token implemented in **[[#R2]]** and **[[#R3]]** is encoded using [[http://en.wikipedia.org/wiki/Base64#URL_applications|Base64 for URL]], and comprises the unique [[http://en.wikipedia.org/wiki/Universally_unique_identifier|UUID]] of the Rhizome database plus the [[http://www.sqlite.org/autoinc.html|ROWID]] of the SQLite MANIFESTS table row, which increases monotonically with every insert or alteration.
  * <BOOKMARK:N3>**N3**.  In **[[#R2]]**, ''GET /restful/rhizome/bundlelist.json'' returns bundles in reverse chronological order (most recent first).  This was chosen because, in general, applications are likely to be interested in the most recent bundles (even were filtering functions added in future), and it allows applications to cut off the response once they have received enough rows simply by closing the HTTP connection prematurely.
  * <BOOKMARK:N4>**N4**.  In **[[#R2]]** and **[[#R3]]**, the token is embedded in the JSON output of ''GET /restful/rhizome/bundlelist.json'' and ''GET /restful/rhizome/newsince/<token>/bundlelist.json'' as an optional per-row attribute which is only present in the first row and in any row thereafter which is more recent than the row containing the previous token.   This means that any future changes which alter the row order will not introduce incompatibilities in any client which follows the correct semantic of always using the last token received when forming the ''GET /restful/rhizome/newsince/<token>/bundlelist.json'' URL.
  * <BOOKMARK:N5>**N5**.  In **[[#R3]]**, ''GET /restful/rhizome/newsince/<token>/bundlelist.json'' returns bundles in chronological order (oldest first).  This is because the temporally incremental nature of the request makes it impossible to return bundles in reverse chronological order (as in **[[#N3]]**) except by waiting for the end of the 60-second timeout and sending them all in a single burst, which would not yield the kind of immediate interactive functionality that is desired.  One consequence of this is that every row will contain a new token (see **[[#N4]]**), which has the benefit that the HTTP client can close the connection at any point and initiate a new request using the latest token it received, without missing any bundles.
  * <BOOKMARK:N6>**N6**.  In **[[#R4]]**, **[[#R5]]**, **[[#R6]]** and **[[#R7]]**, the HTTP response contains the following extra headers that describe the outcome of the Rhizome payload operation and the bundle operation:<code>
Serval-Rhizome-Result-Payload-Status-Code: -1|0|1|2|3|4|5|6|7
Serval-Rhizome-Result-Payload-Status-Message: <text>
Serval-Rhizome-Result-Bundle-Status-Code: -1|0|1|2|3|4|5|6|7|8
Serval-Rhizome-Result-Bundle-Status-Code: <text>
Serval-Rhizome-Bundle-Id: <BundleIDHex>
Serval-Rhizome-Bundle-Version: <ASCIIDecimal>
Serval-Rhizome-Bundle-Filesize: <ASCIIDecimal>
Serval-Rhizome-Bundle-Filehash: <FilehashHex>
Serval-Rhizome-Bundle-BK: <BundleKeyHex>
Serval-Rhizome-Bundle-Crypt: 0|1
Serval-Rhizome-Bundle-Tail: <ASCIIDecimal>
Serval-Rhizome-Bundle-Date: <ASCIIDecimal>
Serval-Rhizome-Bundle-Name: <NameQuotedString>
Serval-Rhizome-Bundle-Service: <token>
Serval-Rhizome-Bundle-Author: <AuthorSIDHex>
Serval-Rhizome-Bundle-Secret: <BundleSecretHex>
Serval-Rhizome-Bundle-Rowid: <ASCIIDecimal>
Serval-Rhizome-Bundle-Inserttime: <ASCIIDecimal>
</code>
    * The ''Serval-Rhizome-Result-Payload-Status-Code'' values are encapsulated in the new Java class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/rhizome/RhizomePayloadStatus.java|org.servalproject.servaldna.rhizome.RhizomePayloadStatus]]
    * The ''Serval-Rhizome-Result-Bundle-Status-Code'' values are encapsulated in the new Java class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/rhizome/RhizomeBundleStatus.java|org.servalproject.servaldna.rhizome.RhizomeBundleStatus]]
    * The ''Serval-Rhizome-Bundle-Filehash'' header is only present if the bundle filesize is nonzero.
    * The ''Serval-Rhizome-Bundle-BK'' header is only present if the manifest contains a Bundle Key (''BK'') field
    * The ''Serval-Rhizome-Bundle-Crypt'' header is only present if the manifest contains a ''crypt'' field
    * The ''Serval-Rhizome-Bundle-Tail'' header is only present if the manifest contains a ''tail'' field, ie, is a journal
    * The ''Serval-Rhizome-Bundle-Date'' header is only present if the manifest contains a ''date'' field
    * The ''Serval-Rhizome-Bundle-Name'' header is only present if the manifest contains a ''name'' field
    * The ''Serval-Rhizome-Bundle-Service'' header is only present if the manifest contains a ''service'' field
    * The ''Serval-Rhizome-Bundle-Author'' header is only present if the identity that created the bundle is present and currently unlocked in the keyring.
    * The ''Serval-Rhizome-Bundle-Secret'' header ditto.
    * The ''Serval-Rhizome-Bundle-Rowid'' header exposes an implementation detail of the Rhizome storage database, and as such, is not guaranteed to be present.
  * <BOOKMARK:N7>**N7**.  In **[[#R7]]** the ''POST /restful/rhizome/insert'' request cannot be used to inject a [[:content:tech:rhizome_journal|Journal Bundle]], and produces a 501 "Not Implemented" HTTP response.  A future "journal append" operation will have to be implemented to support Rhizome journals.
  * <BOOKMARK:N8>**N8**.  The HTTP status code 403 "Forbidden" is returned if a request is well-formed by the rules of HTTP and obeys the RESTful API, but provokes a failure response that is not the fault of the caller nor a defect in the interface, for example running out of space in the Rhizome store, or missing a necessary bundle
  * <BOOKMARK:N9>**N9**.  Except where otherwise specified, all RESTful operations return a JSON response, ie, ''Content-Type: application/json''.  Error results return a single JSON object that contains the HTTP status code and a textual message:<code>
{
   "http_status_code": 403,
   "http_status_message": "Missing form parameter"
}
</code>
  * <BOOKMARK:N9a>**N9(a)**. The JSON object returned from Rhizome operations **[[#R4]]**, **[[#R5]]**, **[[#R6]]** and **[[#R7]]** contains four extra fields describing the outcome of the Rhizome payload operation and the bundle operation:<code>
{
   "http_status_code": 200,
   "http_status_message": "OK",
   "rhizome_payload_status_code": 2,
   "rhizome_payload_status_message": "Payload already in store"
   "rhizome_bundle_status_code": 3,
   "rhizome_bundle_status_message": "Newer bundle already in store"
}
</code>
  * <BOOKMARK:N9b>**N9(b)**. The JSON object returned from MeshMS operations **[[#R10]]**, **[[#R11]]** and **[[#R12]]** contains two extra fields describing the outcome of the MeshMS operation:<code>
{
   "http_status_code": 201,
   "http_status_message": "Created",
   "meshms_status_code": 1,
   "meshms_status_message": "Bundle updated"
}
</code>
  * <BOOKMARK:N10>**N10**.  In **[[#R10]]** and **[[#R11]]**, the JSON output of ''GET /restful/meshms/<SID>/<SID>/messagelist.json'' and ''GET /restful/meshms/<SID>/<SID>/newsince/token/messagelist.json'' has the following structure:<code>
{
"read_offset":29,
"latest_ack_my_offset":76,
"header":["type","my_sid","their_sid","offset","token","text","delivered","read","ack_offset"],
"rows":[
[">","EEBF3AC19E7EE58722A0F6D4A4D5894A72F5C71030C3399FE75808DCF6C6254B","C10C91D24BF210DD6733ED2424B4509E6CC4402D34055B6D29F7A778701AA542",105,"yytLhh7a4lraha6uj5E1e3c3itq1yIoxwPpWP6c3JYJpAAAAAAAAAA==","Text of fourth message",false,false,null],
["<","EEBF3AC19E7EE58722A0F6D4A4D5894A72F5C71030C3399FE75808DCF6C6254B","C10C91D24BF210DD6733ED2424B4509E6CC4402D34055B6D29F7A778701AA542",56,"Wn7oquN__5k0tI317ZcH9DszrdpEqnzjn3nZbvaK3fY4AAAAAAAAAA==","Text of second reply",true,false,null],
["ACK","EEBF3AC19E7EE58722A0F6D4A4D5894A72F5C71030C3399FE75808DCF6C6254B","C10C91D24BF210DD6733ED2424B4509E6CC4402D34055B6D29F7A778701AA542",33,"Wn7oquN__5k0tI317ZcH9DszrdpEqnzjn3nZbvaK3fYhAAAAAAAAAA==",null,true,false,76],
[">","EEBF3AC19E7EE58722A0F6D4A4D5894A72F5C71030C3399FE75808DCF6C6254B","C10C91D24BF210DD6733ED2424B4509E6CC4402D34055B6D29F7A778701AA542",76,"yytLhh7a4lraha6uj5E1e3c3itq1yIoxwPpWP6c3JYJMAAAAAAAAAA==","Text of third message",true,false,null],
["<","EEBF3AC19E7EE58722A0F6D4A4D5894A72F5C71030C3399FE75808DCF6C6254B","C10C91D24BF210DD6733ED2424B4509E6CC4402D34055B6D29F7A778701AA542",29,"Wn7oquN__5k0tI317ZcH9DszrdpEqnzjn3nZbvaK3fYdAAAAAAAAAA==","Text of first reply",true,true,null],
[">","EEBF3AC19E7EE58722A0F6D4A4D5894A72F5C71030C3399FE75808DCF6C6254B","C10C91D24BF210DD6733ED2424B4509E6CC4402D34055B6D29F7A778701AA542",49,"yytLhh7a4lraha6uj5E1e3c3itq1yIoxwPpWP6c3JYIxAAAAAAAAAA==","Text of second message",true,false,null],
[">","EEBF3AC19E7EE58722A0F6D4A4D5894A72F5C71030C3399FE75808DCF6C6254B","C10C91D24BF210DD6733ED2424B4509E6CC4402D34055B6D29F7A778701AA542",24,"yytLhh7a4lraha6uj5E1e3c3itq1yIoxwPpWP6c3JYIYAAAAAAAAAA==","Text of first message",true,false,null]
]
}
</code>**[[#N2]]** describes why this structure is used instead of an array of JSON objects.  The per-message fields are:
    * ''type'' -- ''<nowiki>">"</nowiki>'' for a sent message, ''<nowiki>"<"</nowiki>'' for a received message, ''<nowiki>"ACK"</nowiki>'' for a received ACK
    * ''my_sid'' -- the Serval ID (identity) of the sender (the first SID in the request URL)
    * ''their_sid'' -- the Serval ID (identity) of the recipient (the second SID in the request URL)
    * ''offset'' -- the byte position within the respective Journal Bundle (ply) where the message is stored -- there are two plys: one for sent messages, the other for received, and it is only meaningful to compare offsets within the same ply
    * ''token'' -- described in **[[#N11]]**
    * ''text'' -- the content of the message (''null'' for ACKs)
    * ''delivered'' -- equals ''<nowiki>(offset <= latest_ack_offset)</nowiki>'' for sent messages, and is always ''true'' for received messages and ACKs
    * ''read'' -- equals ''<nowiki>(offset <= read_offset)</nowiki>'' for received messages and ACKs, and is always ''false'' for sent messages
    * ''ack_offset'' -- the byte offset within the sender's ply of the sent message that the recipient is acknowledging as delivered -- equal to ''<nowiki>(latest_ack_offset)</nowiki>'' -- ''null'' for non-ACKs
  * <BOOKMARK:N11>**N11**.  The token implemented in **[[#R10]]** and **[[#R11]]** is encoded using [[http://en.wikipedia.org/wiki/Base64#URL_applications|Base64 for URL]], and comprises the Bundle ID of the respective ply (sender or recipient) and the message's byte position (offset) within that bundle
  * <BOOKMARK:N12>**N12**.  The JSON described in **[[#N10]]** contains ''my_sid'' and ''their_sid'' fields per-row so that the same JSON structure can be used in future for new HTTP RESTful requests that could return messages from more than one conversation
  * <BOOKMARK:N13>**N13**.  In **[[#R10]]**, the ''GET /restful/meshms/<SID>/<SID>/messagelist.json'' response contains at most one ACK row, which is the latest (most recent) ACK from the recipient.  Its ''ack_offset'' field will be equal to the ''<nowiki>"latest_ack_offset"</nowiki>'' field in the top-level object.
  * <BOOKMARK:N14>**N14**.  In **[[#R11]]**, the ''GET /restful/meshms/<SID>/<SID>/newsince/<TOKEN>/messagelist.json'' response can contain more than one ACK row, as ACKs are output as they are received.  The ''<nowiki>"latest_ack_offset"</nowiki>'' top-level object field is therefore omitted from this response.
  * <BOOKMARK:N15>**N15**.  In **[[#R11]]**, the ''GET /restful/meshms/<SID>/<SID>/newsince/<TOKEN>/messagelist.json'' response may return sent messages whose ''read'' field is ''true'' even though prior (lower offset) sent messages have a ''false'' value.  This occurs because the ''read_offset'' marker can advance during the course of the request.  Once a sent message is marked as read, all prior sent messages are also implicitly read (there is no per-message read/unread flag).  The ''<nowiki>"read_offset"</nowiki>'' top-level object field is therefore omitted from this response.
  * <BOOKMARK:N16>**N16**.  Three additional RESTful HTTP requests were implemented to make the MeshMS API complete:
    * ''POST /restful/meshms/<SID>/readall'' marks all the messages in all conversations as read
    * ''POST /restful/meshms/<SID>/<SID>/readall'' marks all the messages in a single conversation as read
    * ''POST /restful/meshms/<SID>/<SID>/recv/<offset>/read'' advances the read-offset in a single conversation, to mark a single message (and all its predecessors) as read

===== COMPLETION REPORT =====

The work was completed on 14 July 2014, having been performed concurrently with three other contracts:
  * [[naf5]]
  * [[naf6]]
  * [[openitp2]]

==== Preliminary work ====

This contract drove several significant architectural improvements to [[:content:servaldna:]] and [[:content:servalmesh:]].

== Security: SQL injections ==
One issue that must be addressed when building any external interface is security.
  * A class of potential SQL injection vulnerabilities in the existing HTTP server was fixed (see [[https://github.com/servalproject/serval-dna/issues/69|Serval DNA issue #69]]).
  * This prompted a re-factor of the Serval DNA source code to regularise the use of internal types for representing fundamental types like SID, Bundle ID, MDP port number, and so forth (see [[https://github.com/servalproject/serval-dna/issues/11|Serval DNA issue #11]] and Serval DNA Git commits [[https://github.com/servalproject/serval-dna/commit/3758b038abea00af385fc44ad50156ff7a3daa8a|3758b03]], and [[https://github.com/servalproject/serval-dna/commit/a95ef791391e6b1c08779846fa0b3efee28d42fa|a95ef79]] and [[https://github.com/servalproject/serval-dna/commit/221fc4a4fc517a2b987bc8162f262f7885f58260|221fc4a]]).
  * As a result, many poor coding patterns that were susceptible to classic vulnerability bugs (eg, interpreted code injection and buffer overruns) were replaced with safe and far more rigorous coding patterns.
  * This provided a sound code base upon which to embark on implementation of a new HTTP REST API.

== New HTTP server ==
The next issue to be addressed was the existing Rhizome HTTP server code.
  * The only HTTP clients to date had been other Android apps by the Serval Project and Serval DNA itself, which sent small, known header blocks.
  * HTTP requests were being parsed by reading the entire header block into memory and parsing it using a combination of pointer work and sscanf(3).  Much of this code was duplicated between [[https://github.com/servalproject/serval-dna/blob/00cf61721d661f649705d93873722192f588e8a0/rhizome_http.c|rhizome_http.c]] and [[https://github.com/servalproject/serval-dna/blob/00cf61721d661f649705d93873722192f588e8a0/rhizome_direct_http.c|rhizome_direct_http.c]] (from commit [[https://github.com/servalproject/serval-dna/tree/00cf61721d661f649705d93873722192f588e8a0|00cf617]]).
  * The HTTP request parsing code needed improvement in order to support all potential HTTP clients, and to provide a sound basis for parsing more HTTP headers such as [[http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.8|Authorization]] and [[http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17|Content-Type]].
  * A completely new HTTP request parser was written ([[https://github.com/servalproject/serval-dna/blob/9d54c629b2205b2bb702a2d0871c363e527f7a3d/http_server.c|http_server.c]]) which:
    * conserves memory by using a single buffer for reading and disassembling an HTTP request header and ''multipart/form-data'' body;
    * adheres more closely to published standards (eg, now parses tokens using the proper lexical character set instead of any non-space sequence, and corrects an off-by-one mistake in the existing interpretation of the [[http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35|Range]] header);
    * uses progressive parsing that does not require the entire header block to fit within the buffer, just each header line;
    * makes extensive use of [[http://linux.die.net/man/3/assert|assert(3)]] and [[http://linux.die.net/man/3/abort|abort(3)]] to ensure that buffer overruns and other logical errors do not go undetected (if there is any buffer overrun vulnerability, it is likely to cause Serval DNA to terminate, so the only potential attack is likely to be denial of service rather than information disclosure or remote code execution;
    * formats and sends an HTTP response using the same buffer used for receiving the request, allowing the content (body) to either be provided statically (already rendered into a temporary buffer) or generated dynamically (rendered progressively into a single buffer as the response is sent);
  * The existing Rhizome HTTP interfaces (Rhizome HTTP transport and [[:content:tech:Rhizome Direct]]) were rewritten to use the new HTTP request parser.
  * The new HTTP request parser is modular and is not tied to the existing Rhizome implementation, so it may easily be re-used to implement other HTTP interfaces in future.

== Re-organisation around RESTful API ==
Henceforward the RESTful API will be the preferred (and eventually only) method for applications to use Rhizome and MeshMS.
  * The existing JNI API for Rhizome and MeshMS was inferior:
    * the JNI API provided function-call entry points into the Serval DNA shared library
    * application processes opened and accessed the Rhizome database themselves, not via requests to the daemon
    * this sometimes caused database locking to block the daemon, which severely degraded responsiveness when starting a freshly-installed Batphone app in the presence of other devices holding lots of Rhizome content -- see [[https://github.com/servalproject/serval-dna/issues/1|Serval DNA issue #1]]
    * all Android applications using MeshMS and Rhizome had to be bundled together in a single APK, so that Android would run them all with the same User ID, so they could all access and modify the Rhizome database files
  * Commit [[https://github.com/servalproject/serval-dna/commit/68dbaef38d03535271eed14a6da35323a0dd7a4f|68dbaef]] refactored the Java API to add a stateful object that holds an open connection to the daemon, and keep the Serval DNA RESTful interface independent from the JNI command interface
  * Commit [[https://github.com/servalproject/serval-dna/commit/82b13caac4c2386a096acc5c72ebde2b4871c245|82b13ca]] deprecated the existing MeshMS JNI API
  * Commit [[https://github.com/servalproject/serval-dna/commit/11e9d382995da9fc2c0342abcc52623cc8a80781|11e9d38]] renamed the Serval DNA configuration ''rhizome.api.restful.'' options to ''api.restful.'' because the RESTful interface will be used for services other than Rhizome (eg, MeshMS)

== Serval DNA power consumption ==
The final issue to be addressed was the CPU activity of the [[:content:servaldna:]] daemon whilst idle.
  * The previous implementation of the [[:content:servalmesh:|Serval Mesh app for Android]] did not have a continuously running daemon:
    * the daemon only ran while the "Enable Services" checkbox was ticked on the "Connect" menu
    * the daemon awoke periodically on *alarm* events (once per second) while running, even while the network was idle or no network interfaces were up
    * this caused rapid battery drain while Serval Mesh networking was active, which was advised in the [[https://github.com/servalproject/batphone/blob/0.91/doc/RELEASE-0.91.md|Release Notes for 0.91]]
    * the JNI API (see above) accessed the Rhizome database directly in-process; it did not use the daemon, so the Rhizome and MeshMS screens functioned at all times
  * The new implementation (**[[#R15]]**) implements MeshMS operations using the RESTful HTTP interface:
    * this requires a continuously running daemon, or at least a daemon running while any Rhizome or MeshMS screen are active
    * this could quickly drain the battery simply by opening the app
  * This necessitated improvements to the event scheduling sub-system within [[:content:servaldna:]]:
    * only use CPU in response to real external events (user and network)
    * only schedule timed wake-ups for externally imposed timing requirements such as network protocol time-outs
  * Commit [[https://github.com/servalproject/serval-dna/commit/6d4ad0e1505d8826cda024159892b82699e392c7|6d4ad0e]] improved the event alarm scheduler [[https://github.com/servalproject/serval-dna/blob/development/fdqueue.c|fdqueue.c]]:
    * a new kind of "lazy" alarm does not force the daemon process to wake
    * whenever the daemon does wake due to network or user activity, overdue lazy alarms are run in time order
    * existing periodic events, such as the configuration file poll and shutdown file poll, were re-implemented to use lazy alarms
    * this allowed **[[#R15]]** to be implemented without any impact on battery life
  * Commit [[https://github.com/servalproject/batphone/commit/f0948b2bb0cc8b9ca3bf89dd743df30dacbc3620|f0948b2]] improved the [[:content:servalmesh:]] app:
    * starts the daemon as needed and keeps it running
    * changing the "Enable Services" checkbox simply toggles a Serval DNA configuration option
  * This improvement has significant positive repercussions for other Serval products like [[:content:meshextender:]], by extending battery life towards the maximum practical limit

==== Implementation ====

**[[#R1a|R1(a)]]** -- **HTTP Basic Authorization** and **loopback interface** restriction
  * commit [[https://github.com/servalproject/serval-dna/commit/3d3e900e727fac20f1984ced519de948d073b9ca|3d3e900]] expanded the [[:content:tech:configuration|Serval DNA configuration schema]] to include Rhizome Restful API user names and passwords (in the clear)
  * commits [[https://github.com/servalproject/serval-dna/commit/21fe12859fe95b5b1812dc85ef3911519ed83149|21fe128]] and [[https://github.com/servalproject/serval-dna/commit/2ac1bc32403ecbf3deb54073bb02d8effa4e6264|2ac1bc3]] implemented HTTP Basic Authorization:
    * the [[#new HTTP server]] was extended to parse the [[http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.8|Authorization]] header
    * the [[#new HTTP server]] was extended to support the [[http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.47|WWW-Authenticate]] response header
    * two test cases were added for Basic Authentication
    * Basic Authentication was implemented in an empty ''/restful/rhizome/bundlelist.json'' resource to return a 401 Unauthorized response unless a correct [[http://en.wikipedia.org/wiki/Basic_access_authentication|Basic Authentication]] user/password pair are supplied as per [[http://www.ietf.org/rfc/rfc2617.txt|RFC2617]]
  * commit [[https://github.com/servalproject/serval-dna/commit/b44046d6121167688892def3163a53f7d84126a2|b44046d]] improved the empty ''/restful/rhizome/bundlelist.json'' resource to return a 403 Forbidden response unless the request arrives over the local loopback interface
  * all the checks in the ''/restful/rhizome/bundlelist.json'' resource were factored into a function that was re-used to implement the other resources
  * commit [[https://github.com/servalproject/serval-dna/commit/067340bbba734c8b505a94815798f5a1ddd5283a|067340b]] renamed the HTTP Authorisation realm from "Serval Rhizome" to "Serval RESTful API"

**[[#R2]]** -- **''GET /restful/rhizome/bundlelist.json''** request
  * commits [[https://github.com/servalproject/serval-dna/commit/1b906f3f118cde97598a36be2ae9c210c7320ac8|1b906f3]] and [[https://github.com/servalproject/serval-dna/commit/6b961c56ce24a46ad58ebe744ca87e47c49adadd|6b961c5]]:
    * added JSON content (see **[[#N1]]**) to the empty ''/restful/rhizome/bundlelist.json'' that had already been implemented by the Basic Authentication work (see above)
    * did not implement the token (yet)
    * added a new test case that:
      * injects 100 files into Rhizome
      * fetches the JSON bundle list using the [[http://curl.haxx.se/|curl(1)]] command-line HTTP client
      * performs assertions on the returned JSON using the [[http://stedolan.github.io/jq/|jq(1)]] utility
  * commit [[https://github.com/servalproject/serval-dna/commit/50bbb722d0ec0581cc5790e5a42c18cbd0f3531c|50bbb72]] added a new [[http://en.wikipedia.org/wiki/Universally_unique_identifier|UUID]] data type and primitive functions to Serval DNA
  * commit [[https://github.com/servalproject/serval-dna/commit/64db53a092874ff63ad1a0c9ad70e4bac71aab47|64db53a]] upgraded the Rhizome database schema to include a single, unique UUID per database
  * commit [[https://github.com/servalproject/serval-dna/commit/4380fdcccd5111648092b5262d9d80244216832d|4380fdc]] altered the JSON returned to include the specified token.  The token includes the Rhizome database's UUID to ensure that it cannot be used erroneously in a ''/restful/rhizome/newsince/<token>/bundlelist.json'' request to another database.

**[[#R3]]** -- **''GET /restful/rhizome/newsince/<token>/bundlelist.json''** request
  * the structure of the JSON returned by ''GET /restful/rhizome/bundlelist.json'' (**[[#R2]]**) was improved as described in **[[#N2]]**, **[[#N3]]** and **[[#N4]]**
  * the ''GET /restful/rhizome/newsince/<token>/bundlelist.json'' request was implemented using the same JSON content generation code as ''GET /restful/rhizome/bundlelist.json'' modified as follows:
    * commit [[https://github.com/servalproject/serval-dna/commit/c1f0c0cb074be826ce60cb5754a3c2edc9dec56b|c1f0c0c]] adds a new dispatched path function which decodes the ''<token>'' and invokes the same JSON content generator with different parameters
    * commit [[https://github.com/servalproject/serval-dna/commit/21b10b2cb7b7c16de6d36f9790cb1b0d92175fb6|21b10b2]] alters the underlying database iteration logic to iterate in chronological order (oldest first) if fetching bundles since a given ROWID (see **[[#N5]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/1acfff6ab5a7c4997612116f47728759574297a3|1acfff6]] adds support for pausing HTTP response transmission for a specified time interval
    * commit [[https://github.com/servalproject/serval-dna/commit/fce089317321a67fe223da43ccb889c1e24e8e2a|fce0893]] deals with iterator end by pausing the transmission and polling (re-querying) every 2 seconds until a configurable timeout (default 60 seconds) is reached, instead of terminating the response immediately -- **[[#V1]]** for the variation that replaces a 204 No Data response with an empty JSON response
    * commit [[https://github.com/servalproject/serval-dna/commit/29fab6dcb76d0a17352a0ec292d78f1ba1a5554a|29fab6d]] makes the poll interval configurable (default 2 seconds)
  * commit [[https://github.com/servalproject/serval-dna/commit/6e4acb6ab9fe381c8683f2c0d0822497411d2e64|6e4acb6]] ensures that the token matches the Rhizome database UUID, otherwise the ROWID embedded in the token is meaningless and the request fails
  * commit [[https://github.com/servalproject/serval-dna/commit/513aa159687d00e1d6782490a5b1cb67103e47a7|513aa15]] put the finishing touches on a new test case that runs three HTTP requests simultaneously while adding new bundles

**[[#R4]]** -- **''GET /restful/rhizome/<BundleID>.rhm''** request (see **[[#V2]]**) (completed 13 December)
  * commit [[https://github.com/servalproject/serval-dna/commit/40698b14e00803e43222fc184ae2d3748c838d0a|40698b1]] fixed a minor bug in the HTTP request parser, that incorrectly detected a Content-Type header present in a GET request
  * commit [[https://github.com/servalproject/serval-dna/commit/26e01208981a6c1480a76d7d21f317cff328de78|26e0120]] implemented the request and a test case that fetches three manifests in succession
  * commit [[https://github.com/servalproject/serval-dna/commit/183cb46000f9ba69302391421fcc0fc3bc2f27b1|183cb46]] added the extra HTTP response headers described in **[[#N6]]**

**[[#R5]]** -- **''GET /restful/rhizome/<BundleID>/raw.bin''** request (completed 13 December)
  * commit [[https://github.com/servalproject/serval-dna/commit/6361bfd757eaea8442bcdb6c9d327649a2a26f7b|6361bfd]] implemented the request and a test case that fetches four payloads in succession and checks the HTTP response headers described in **[[#N6]]**
  * commit [[https://github.com/servalproject/serval-dna/commit/8c9ac6c1a5ac3d720bf7b232ff7099a3c9ca3ed6|8c9ac6c]] improved the test case to use a mixture of encrypted and plain payloads

**[[#R6]]** -- **''GET /restful/rhizome/<BundleID>/decrypted.bin''** request (completed 16 December)
  * commit [[https://github.com/servalproject/serval-dna/commit/6798e943ec1d910255d5a2dac852840ce7b3ba21|6798e94]] implemented the request and a test case that fetches four decrypted payloads in succession and checks the HTTP response headers described in **[[#N6]]**

**[[#R7]]** -- **''POST /restful/rhizome/insert''** request (completed 30 December)
  * commits [[https://github.com/servalproject/serval-dna/commit/250309f4d8baa22ab34f0bb78a44b7b5f91feaa3|250309f]], [[https://github.com/servalproject/serval-dna/commit/b5f7a088b27d4c9cf93d50ca9dbfd78f8fd536d5|b5f7a08]], [[https://github.com/servalproject/serval-dna/commit/21328e28888d775c88c81b74c06928eff30240a4|21328e2]], [[https://github.com/servalproject/serval-dna/commit/d52ba4c8712ae76eec336e5d0e8d28bfe0b9af1e|d52ba4c]], [[https://github.com/servalproject/serval-dna/commit/8a1ce7f373cc4c5a6a0821c22d919c4db24c5092|8a1ce7f]], [[https://github.com/servalproject/serval-dna/commit/9ebef81a49e308690406cf02313984c34d8be96d|9ebef81]], [[https://github.com/servalproject/serval-dna/commit/b37e27f5dab5acfb5d8f932b2af9396bd6fecc15|b37e27f]], [[https://github.com/servalproject/serval-dna/commit/72040517e1879d0d1b0e50ed605e18ed186e7d90|7204051]], [[https://github.com/servalproject/serval-dna/commit/34188fa48941fd784cf5031738e46c1bde5a44e0|34188fa]], [[https://github.com/servalproject/serval-dna/commit/7cecdf7a3a971e21ee041e07606d9dd6644f184d|7cecdf7]], [[https://github.com/servalproject/serval-dna/commit/f81883758a035245f3d5d1386bbe7795206c11db|f818837]], [[https://github.com/servalproject/serval-dna/commit/dd5048bd43d31381609f46b9d46d20a93154c660|dd5048b]], and [[https://github.com/servalproject/serval-dna/commit/42e6168998e624c85375b9da4720283512a77a8f|42e6168]] refactored and improved the existing Rhizome and HTTP server code to support the new requirements of an HTTP "add bundle" operation
  * commit [[https://github.com/servalproject/serval-dna/commit/ee9c96bb8cf6eb5442161d8947bc917686171dea|ee9c96b]] implemented the request and test cases (see **[[#R14]]**)

**[[#R8]]** -- MeshMS in Serval DNA (initial work completed 5 August, finalised on 30 December 2013)
  * commit [[https://github.com/servalproject/serval-dna/commit/8a1c0a39e31e038cc120aea7bb824367b389adea|8a1c0a3]] implemented the MeshMS2 service in ServalDNA, based on a student implementation from earlier in 2013
  * commits [[https://github.com/servalproject/serval-dna/commit/bacba19dc17a94d09f9967f5d7d296d5e96de5c2|bacba19]] and [[https://github.com/servalproject/serval-dna/commit/42ab9aec4caf377f93fe76d091d2e24e987798a9|42ab9ae]] fixed some significant bugs
  * the commits [[https://github.com/servalproject/serval-dna/commit/9ebef81a49e308690406cf02313984c34d8be96d|9ebef81]] and [[https://github.com/servalproject/serval-dna/commit/72040517e1879d0d1b0e50ed605e18ed186e7d90|7204051]] (also listed above under **[[#R7]]**) improved the implementation by refactoring payload and manifest handling functions

**Merge** into mainline development branch (completed 30 December 2013)
  * commit [[https://github.com/servalproject/serval-dna/commit/d4320f20daf5bbad1627d30335041fc5fb1a4f7a|d4320f2]] merged all work up to **[[#R8]]** into the mainline development branch, to make the Rhizome code improvements available for merge into other feature branches

**JSON responses** (completed 20 January 2014)
  * commit [[https://github.com/servalproject/serval-dna/commit/6a1c8bcf5ac34962d7dff363d507e285e829d9a6|6a1c8bc]] implemented **[[#N8]]** and **[[#N9]]** so that all responses are in JSON format except where otherwise specified

**[[#R9]]** -- **''GET /restful/meshms/<SID>/conversationlist.json''** request (completed 22 January 2014, fixed 26 June 2014)
  * commits [[https://github.com/servalproject/serval-dna/commit/5c1ebc71a9efdd41e29eb2cdfd30414965fe8e86|5c1ebc7]], [[https://github.com/servalproject/serval-dna/commit/e3e3e1e046c24a55ec8e19237b4190e27791d6a5|e3e3e1e]], [[https://github.com/servalproject/serval-dna/commit/cb420c61e385e543b5895992037df729181f5fe0|cb420c6]] and [[https://github.com/servalproject/serval-dna/commit/94274ba5fab5a2ebce2a5e006a8589363bacd2ec|94274ba]] refactored MeshMS and HTTP ''GET /restful/rhizome/bundlelist.json'' code for re-use
  * commit [[https://github.com/servalproject/serval-dna/commit/8897563d09963c0f39ec15e3e5bfe25b41ab870a|8897563]] implemented the request and a test case, based on the existing MeshMS CLI test, that creates three conversations, runs the HTTP request, then marks the conversations as read and runs the HTTP request again
  * commit [[https://github.com/servalproject/serval-dna/commit/6c16b95a384852de3da2b80bf15521385b550bba|6c16b95]] fixed a SEGV bug when requesting an empty conversation list

**[[#R10]]** -- **''GET /restful/meshms/<fromSID>/<toSID>/messagelist.json''** request (see **[[#V3]]**) (completed 24 January 2014)
  * commit [[https://github.com/servalproject/serval-dna/commit/fb74dc66491ac3a50cea9e7e586180f13f1f93a1|fb74dc6]] fixed a minor bug in the Base64 encoding (see **[[#N11]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/879395b121e804852a9e8503ad8dbf528834a8c9|879395b]] refactored the existing ''meshms list messages'' code to use an iterator
  * commit [[https://github.com/servalproject/serval-dna/commit/7b5752a1116f77342a7a08e0ce1832908b1bb619|7b5752a]] implemented the request (see **[[#N10]]**, **[[#N11]]** and **[[#N13]]**) and added a test case

**[[#R11]]** -- **''GET /restful/meshms/<fromSID>/<toSID>/newsince/<TOKEN>/messagelist.json''** request (see **[[#V3]]**) (completed 31 January 2014)
  * commit [[https://github.com/servalproject/serval-dna/commit/b1992b3905547c101866702a9464e4b2466bd0f5|b1992b3]] refactored some MeshMS source code to make it a bit clearer
  * commit [[https://github.com/servalproject/serval-dna/commit/c73bc49cf067f05b6589953c8bf04b39797b56ea|c73bc49]] added sender SID and recipient SID to the JSON structure produced by **[[#R10]]** (see **[[#N12]]**), also refactored the new MeshMS iterator code to use more consistent naming
  * commit [[https://github.com/servalproject/serval-dna/commit/ebe444ffe32beae8b0ca3597f36f243b268b2c64|ebe444f]] refactored the RESTful MeshMS test script for re-usability in more RESTful MeshMS test cases
  * commit [[https://github.com/servalproject/serval-dna/commit/f4249707a4f984e0fa273ecce73f7c9272bc9f42|f424970]] implemented the request (see **[[#N10]]**, **[[#N11]]**, **[[#N14]]** and **[[#N15]]**) and added two test cases

**[[#R12]]** -- **''POST /restful/meshms/<fromSID>/<toSID>/sendmessage''** request (see **[[#V4]]** and **[[#V5]]**) (completed 7 February 2014)
  * commits [[https://github.com/servalproject/serval-dna/commit/015ed0b181b2bd3346cbf443733728b199111285|015ed0b]], [[https://github.com/servalproject/serval-dna/commit/051066af8609a6725dcef5ddbfd957eb5b7131f9|051066a]] and [[https://github.com/servalproject/serval-dna/commit/fb2709d10c30eb9b1e0727b88276f385ca977059|fb2709d]] reorganised the source code and made naming of HTTPD components more logical
  * commit [[https://github.com/servalproject/serval-dna/commit/fd86a3d17f8aa1035c26dfdf948cb59602d7992f|fd86a3d]] implemented the request with one test case
  * commit [[https://github.com/servalproject/serval-dna/commit/5dd9ea7a6b9f0ce9eb740424a4282af60769b4ce|5dd9ea7]] refactored MeshMS code to improve handling and reporting of failures
  * commits [[https://github.com/servalproject/serval-dna/commit/116389b589c5a81eb45a15a495d508aa6709255c|116389b]] and [[https://github.com/servalproject/serval-dna/commit/0769fa54e8d60217e5fe8e9a55365e1f7c6382b6|0769fa5]] improved the MeshMS RESTful JSON response to include the MeshMS status code, adding two more test cases

**Merge** into mainline development branch (completed 7 February 2014)
  * commit [[https://github.com/servalproject/serval-dna/commit/0727fb3b62ad6ff6c23154ef2853314ae28eeabf|0727fb3]] merged all work up to **[[#R12]]** into the mainline development branch

**[[#R13]]** -- Java API (completed 14 July 2014)
  * commit [[https://github.com/servalproject/serval-dna/commit/99fb8b6108021842ddaa82d95e17fe1f2f1347b8|99fb8b6]] moved the Java API foundation classes from the Batphone source repository to the Serval DNA repository
  * commit [[https://github.com/servalproject/serval-dna/commit/d879189299b2978b5cb419f7eb91f42a89479407|d879189]] fixed a bug in Serval DNA's HTTP server code (from **[[#R1a]]**) that produced a 500 Server Error if the Basic Authorization string was long
  * commit [[https://github.com/servalproject/serval-dna/commit/9cbd7c365cc1a9c1f94ff4503d06b520a4e06f49|9cbd7c3]] implemented a Java API to list MeshMS conversations:
    * uses the RESTful HTTP interface developed in **[[#R9]]**
    * makes an HTTP connection using the standard [[http://docs.oracle.com/javase/1.5.0/docs/api/java/net/HttpURLConnection.html|java.net.HttpURLConnection]]
    * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/ServalDClient.java|org.servalproject.servaldna.ServalDClient]] establishes an HTTP connection to a running [[:content:servaldna:]] daemon
    * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/meshms/MeshMSConversationList.java|org.servalproject.servaldna.meshmes.MeshMSConversationList]] decodes the JSON response
  * commit [[https://github.com/servalproject/serval-dna/commit/0a5441474482d0cc5cdda1546136b10adbc21e9b|0a54414]] refactored the JSON parsing code (developed in the previous commit):
    * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/json/JSONTokeniser.java|org.servalproject.json.JSONTokeniser]] implements a simple, stateful JSON tokeniser wrapped around an [[http://docs.oracle.com/javase/1.5.0/docs/api/java/io/InputStream.html|java.io.InputStream]] object
  * commit [[https://github.com/servalproject/serval-dna/commit/e9437e9a61ad6248c6ae0084521ffafc51c09b68|e9437e9]] implemented a Java API to list MeshMS messages from a single conversation:
    * uses the RESTful HTTP interface developed in **[[#R10]]**
    * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/meshms/MeshMSMessageList.java|org.servalproject.servaldna.meshmes.MeshMSMessageList]]
  * commit [[https://github.com/servalproject/serval-dna/commit/819b8dc9e73b8b9199fbe949af195aa9a12cde96|819b8dc]] improved the Serval DNA HTTP server to be strict about the Content-Disposition and Content-Type form-part headers supplied in POST requests
  * commit [[https://github.com/servalproject/serval-dna/commit/b14c58bdf68cc51c8fbf32e22b84da2db8584a6a|b14c58b]] merged the MeshMS Java API for listing conversations and messages into the development branch for use in Batphone (**[[#R15]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/eba7f6555f22aa4be8ba1fe99cd25ba632826349|eba7f65]] implemented the Java API to send a MeshMS message:
    * uses the RESTful HTTP interface developed in **[[#R12]]**
    * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/meshms/MeshMSCommon.java|org.servalproject.servaldna.meshms.MeshMSCommon]] contains methods for parsing a MeshMS status report JSON (**[[#N9b]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/7736a4ceb1beefeee4adc2e427b73164feeb6c59|7736a4c]] implemented three new MeshMS RESTful HTTP requests (**[[#N16]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/c79a382a27252c6fa201d68a24a244c52c0685c0|c79a382]] implemented the MeshMS Java API to mark messages as read
  * commit [[https://github.com/servalproject/serval-dna/commit/660daf04382a8f78bda935cff4cb03b03ddc0fa4|660daf0]] merged the completed MeshMS Java API into the development branch for use in Batphone (**[[#R15]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/1ac67de0e9407cd827f87535c66f8144f34bcec3|1ac67de]] fixed a bug in Serval DNA's HTTP form-data MIME part parsing that was revealed by the new Java API
  * commit [[https://github.com/servalproject/serval-dna/commit/61023287b16e25d7a4e504ddc0815ed2125f1943|6102328]] improved Serval DNA's Rhizome manifest validation code to provide a textual reason for failure, which can be passed back to API clients to assist diagnosis
  * commit [[https://github.com/servalproject/serval-dna/commit/db8ee79a4ee11bc0ddf48e9c451fa813b9844851|db8ee79]] implemented a Java API to list Rhizome bundles:
    * uses the RESTful HTTP interface developed in **[[#R2]]**
    * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/rhizome/RhizomeCommon.java|org.servalproject.servaldna.rhizome.RhizomeCommon]] contains methods for parsing a Rhizome status report JSON (**[[#N9a]]**)
    * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/json/JSONTableScanner.java|org.servalproject.json.JSONTableScanner]] generalises the parsing of the JSON table structure returned by Rhizome and MeshMS lists (**[[#N1]]** and **[[#N10]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/93e67ede635930e678ac2714b0436de733a53c3d|93e67ed]] implemented a Java API to fetch a Rhizome manifest given its Bundle ID:
    * uses the RESTful HTTP interface developed in **[[#R4]]**
  * commit [[https://github.com/servalproject/serval-dna/commit/34b6ff48bf9c44808867c3d37e2c4fc9677dcdd7|34b6ff4]] implemented a Java API to fetch a "raw" Rhizome payload (not decrypted) given its Bundle ID
    * uses the RESTful HTTP interface developed in **[[#R5]]**
  * commit [[https://github.com/servalproject/serval-dna/commit/cf4363578988cafa23301d8b0c1cb8a3db3c08b9|cf43635]] improved Serval DNA's internal handling of Rhizome payload and bundle operation outcome enums, and reports these status enum codes and messages via the RESTful API (**[[#N6]]** and **[[#N9a]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/d16be8f42d75fe4715ed6197927a01fcde2f39dc|d16be8f]] added ''meshms_status_message'' to the result returned by MeshMS RESTful operations (**[[#N9b]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/2aec8f31a4ea71196d6041a3a59421884caaa0f0|2aec8f3]] implemented a Java API to fetch a decrypted Rhizome payload given its Bundle ID
    * uses the RESTful HTTP interface developed in **[[#R6]]**
  * commit [[https://github.com/servalproject/serval-dna/commit/a81d05b4f6db84c732e7c2c59a7288e4d16fe29d|a81d05b]] implemented a Java API to insert a new Rhizome bundle
    * uses the RESTful HTTP interface developed in **[[#R7]]**
  * commit [[https://github.com/servalproject/serval-dna/commit/57cce64b6cdf988a3437ce2f98edc16492ae4e35|57cce64]] narrowed the classes of exceptions thrown by Rhizome Java API methods, to make the API more self-documenting
  * commit [[https://github.com/servalproject/serval-dna/commit/04b2a20e54fc8d55dc6187ad21cf3c533d385b3c|04b2a20]] improved the Content-Type header and added Content-Transfer-Encoding headers sent by the Java API *insert bundle* operation, to help future-proof the protocol
  * commit [[https://github.com/servalproject/serval-dna/commit/8842f32b193876723dce5807fcaadd791ed75a5e|8842f32]] ensured that inserting a journal bundle returns the 501 "Not Implemented" HTTP code (**[[#N7]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/606f087dd5fe3cc3a22569ae2050dea447b2d2bf|606f087]] merged the almost-complete Rhizome Java API into the development branch for use in Batphone (**[[#R15]]**)
  * commit [[https://github.com/servalproject/serval-dna/commit/4c6612a61e2e5ce46d840bf6f0ae3771a6583446|4c6612a]] implemented a Java API to list Rhizome bundles that have arrived since a given token:
    * uses the RESTful HTTP interface developed in **[[#R3]]**
  * commit [[https://github.com/servalproject/serval-dna/commit/89623015fcfb75a6b8adccb6bf785a9e46cb26e5|8962301]] implemented a Java API to list MeshMS messages that have arrived since a given token from a single conversation:
    * uses the RESTful HTTP interface developed in **[[#R11]]**

**[[#R14]]** -- Automated tests (completed 11 July 2014)

  * The **[[https://github.com/servalproject/serval-dna/blob/development/tests/meshms|tests/meshms]]** script contains all the tests that were created when the MeshMS protocol was moved from [[:content:servalmesh:]] (Java) into [[:content:servaldna:]] (C) (**[[#R8]]**).

  * Created the new **[[https://github.com/servalproject/serval-dna/blob/5df5a337215c33d79240025566186266901d318c/tests/rhizomehttp|rhizomehttp]]** test script containing tests for the new RESTful HTTP API (later split into **[[https://github.com/servalproject/serval-dna/blob/development/tests/rhizomerestful|rhizomerestful]]** and **[[https://github.com/servalproject/serval-dna/blob/development/tests/meshmsrestful|meshmsrestful]]**):
    * commits [[https://github.com/servalproject/serval-dna/commit/3d3e900e727fac20f1984ced519de948d073b9ca#diff-1|3d3e900]], [[https://github.com/servalproject/serval-dna/commit/21fe12859fe95b5b1812dc85ef3911519ed83149#diff-3|21fe128]] and [[https://github.com/servalproject/serval-dna/commit/2ac1bc32403ecbf3deb54073bb02d8effa4e6264#diff-1|2ac1bc3]] added test cases for Basic Authentication (**[[#R1a]]**) and **bundlelist.json** (**[[#R2]]**)
    * commits [[https://github.com/servalproject/serval-dna/commit/6b961c56ce24a46ad58ebe744ca87e47c49adadd#diff-3|6b961c5]], [[https://github.com/servalproject/serval-dna/commit/13634f8748997b05f1002bd026bcbd398d194fe2#diff-7|13634f8]], [[https://github.com/servalproject/serval-dna/commit/4380fdcccd5111648092b5262d9d80244216832d#diff-3|4380fdc]] and [[https://github.com/servalproject/serval-dna/commit/f8f7716544ceb0abdbe3a3b41c90304c018e9ad9#diff-2|f8f7716]] improved the test case for **bundlelist.json** (**[[#R2]]**)
    * commits [[https://github.com/servalproject/serval-dna/commit/833466411c3fefba19e7f933a8bdae4cda54d631|8334664]], [[https://github.com/servalproject/serval-dna/commit/29fab6dcb76d0a17352a0ec292d78f1ba1a5554a#diff-2|29fab6d]] and [[https://github.com/servalproject/serval-dna/commit/513aa159687d00e1d6782490a5b1cb67103e47a7|513aa15]] added a test case for **newsince bundlelist.json** (**[[#R3]]**)
    * commits [[https://github.com/servalproject/serval-dna/commit/26e01208981a6c1480a76d7d21f317cff328de78#diff-1|26e0120]] and [[https://github.com/servalproject/serval-dna/commit/183cb46000f9ba69302391421fcc0fc3bc2f27b1#diff-3|183cb46]] added a test case for **<BID>.rhm** (**[[#R4]]**)
    * commits [[https://github.com/servalproject/serval-dna/commit/6361bfd757eaea8442bcdb6c9d327649a2a26f7b#diff-2|6361bfd]] and [[https://github.com/servalproject/serval-dna/commit/8c9ac6c1a5ac3d720bf7b232ff7099a3c9ca3ed6#diff-1|8c9ac6c]] added a test case for **<BID>/raw.bin** (**[[#R5]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/6798e943ec1d910255d5a2dac852840ce7b3ba21#diff-1|6798e94]] added a test case for **<BID>/decrypted.bin** (**[[#R6]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/39b2f3a6f5614e7f7b493fb4badb7af4e5869ea4|39b2f3a]] improved the test case titles (descriptions)
    * commit [[https://github.com/servalproject/serval-dna/commit/ee9c96bb8cf6eb5442161d8947bc917686171dea|ee9c96b]] added 14 test cases for **insert** (**[[#R7]]**):
      - insert four bundles with different authors (**bundle-author** form parameter) and //name//, //service// and //crypt// manifest fields, then update them all and check that only the anonymous bundle cannot be updated because the Bundle secret is unknown
      - update an anonymous (authorless) bundle by passing the **bundle-secret** form parameter
      - insert an empty bundle (nil payload)
      - insert a large bundle (50 MiB payload)
      - omit the **manifest** form parameter
      - pass incorrect **manifest** form parameter Content-Type
      - pass duplicate **manifest** form parameters
      - insert a Journal bundle, fails
      - omit **payload** form parameter
      - pass duplicate **payload** form parameter
      - pass **payload** form parameter before **manifest** parameter
      - pass unsupported form parameter
      - manifest //filesize// contradicts supplied **payload** size
      - manifest //filehash// contradicts supplied **payload** hash
    * commit [[https://github.com/servalproject/serval-dna/commit/669080edc8aeb15ecc3d1a09765f21659e22e397|669080e]] included the **rhizomehttp** script in the [[https://github.com/servalproject/serval-dna/blob/development/tests/all|tests/all]] script
    * commit [[https://github.com/servalproject/serval-dna/commit/6a1c8bcf5ac34962d7dff363d507e285e829d9a6#diff-2|6a1c8bc]] made all RESTful HTTP responses have a Content-Type of JSON instead of HTML (specifically, failure results)
    * commit [[https://github.com/servalproject/serval-dna/commit/ebe444ffe32beae8b0ca3597f36f243b268b2c64|ebe444f]] refactored the MeshMS "add message" test fixture into a function for re-use in other tests
    * commit [[https://github.com/servalproject/serval-dna/commit/8897563d09963c0f39ec15e3e5bfe25b41ab870a#diff-4|8897563]] added a test case for **conversationlist.json** (**[[#R9]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/7b5752a1116f77342a7a08e0ce1832908b1bb619#diff-2|7b5752a]] added a test case for **messagelist.json** (**[[#R10]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/f4249707a4f984e0fa273ecce73f7c9272bc9f42#diff-5|f424970]] added test cases for **newsince messagelist.json** (**[[#R11]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/fd86a3d17f8aa1035c26dfdf948cb59602d7992f#diff-8|fd86a3d]] added a test case for **sendmessage** (**[[#R12]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/5dd9ea7a6b9f0ce9eb740424a4282af60769b4ce#diff-4|5dd9ea7]] added three new test cases for **sendmessage** (**[[#R12]]**):
      - missing **message** form parameter
      - duplicate **message** form parameters
      - send from an unknown (locked) identity
    * commit [[https://github.com/servalproject/serval-dna/commit/0769fa54e8d60217e5fe8e9a55365e1f7c6382b6#diff-8|0769fa5]] added a new test case for **messagelist.json** (**[[#R10]]**):
      - list messages from an unknown (locked) identity
    * commit [[https://github.com/servalproject/serval-dna/commit/f2772b0ce8ac92e8f45b0110a2b7fed38400cae8#diff-5|f2772b0]] refactored some test fixture code to be shared with the new [[https://github.com/servalproject/serval-dna/blob/development/tests/meshmsjava|meshmsjava]] test script
    * commit [[https://github.com/servalproject/serval-dna/commit/819b8dc9e73b8b9199fbe949af195aa9a12cde96#diff-4|819b8dc]] improved existing tests and added four new test cases for **sendmessage** (**[[#R12]]**):
      - missing Content-Type header from **message** form parameter
      - unsupported Content-Type in **message** form parameter
      - missing Content-Type charset from **message** form parameter
      - unsupported Content-Type charset in **message** form parameter
      - added proper Content-Type headers to form parts (''plain/text; charset=utf-8'') in other tests
    * commit [[https://github.com/servalproject/serval-dna/commit/7736a4ceb1beefeee4adc2e427b73164feeb6c59#diff-6|7736a4c]] added four new tests for the read-message POST requests (**[[#N16]]**):
      - mark all conversations as read
      - mark all messages as read in a single conversation
      - mark a single message as read in a single conversation
      - supply spurious content body to a POST request
    * commit [[https://github.com/servalproject/serval-dna/commit/776d4de894aa7543beef1fa03f63b3b4793ca630|776d4de]] split the [[https://github.com/servalproject/serval-dna/blob/5df5a337215c33d79240025566186266901d318c/tests/rhizomehttp|rhizomehttp]] test script into [[https://github.com/servalproject/serval-dna/blob/development/tests/rhizomerestful|rhizomerestful]] and [[https://github.com/servalproject/serval-dna/blob/development/tests/meshmsrestful|meshmsrestful]]

  * Created the new **[[https://github.com/servalproject/serval-dna/blob/development/tests/meshmsjava|meshmsjava]]** test script containing tests for the new MeshMS Java API:
    * commit [[https://github.com/servalproject/serval-dna/commit/9cbd7c365cc1a9c1f94ff4503d06b520a4e06f49#diff-8|9cbd7c3]] added a new test case for "Java API MeshMS list conversations" (**[[#R13]]**, **[[#R9]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/e9437e9a61ad6248c6ae0084521ffafc51c09b68#diff-10|e9437e9]] added two new test cases for "Java API MeshMS list messages" (**[[#R13]]**, **[[#R10]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/89623015fcfb75a6b8adccb6bf785a9e46cb26e5#diff-0371afbd127921209ea37daba1e3ef47|8962301]] added two new test cases for "Java API MeshMS list new messages since token" (**[[#R13]]**, **[[#R11]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/eba7f6555f22aa4be8ba1fe99cd25ba632826349#diff-0371afbd127921209ea37daba1e3ef47|eba7f65]] added two new test cases for "Java API MeshMS send message" (**[[#R13]]**, **[[#R12]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/c79a382a27252c6fa201d68a24a244c52c0685c0#diff-0371afbd127921209ea37daba1e3ef47|c79a382]] added three new test cases for "Java API MeshMS mark messages as read" (**[[#R13]]**, **[[#N16]]**)

  * Created the new **[[https://github.com/servalproject/serval-dna/blob/development/tests/rhizomejava|rhizomejava]]** test script containing tests for the new Rhizome Java API:
    * commit [[https://github.com/servalproject/serval-dna/commit/a9ec5dcf695ece8363f838fc71cd25f92a1ed0bc|a9ec5dc]] refactored some shell functions from the existing [[https://github.com/servalproject/serval-dna/blob/development/tests/rhizomerestful|rhizomerestful]] script into [[https://github.com/servalproject/serval-dna/blob/development/testdefs_rhizome.sh|testdefs_rhizome.sh]] ready for re-use
    * commit [[https://github.com/servalproject/serval-dna/commit/db8ee79a4ee11bc0ddf48e9c451fa813b9844851#diff-4601ee6fcb99859fee9b658744fa9024|db8ee79]] created the new script and added a test case for "Java API Rhizome list bundles" (**[[#R13]]**, **[[#R2]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/4c6612a61e2e5ce46d840bf6f0ae3771a6583446#diff-4601ee6fcb99859fee9b658744fa9024|4c6612a]] added a test case for "Java API Rhizome list new bundles since a token" (**[[#R13]]**, **[[#R3]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/93e67ede635930e678ac2714b0436de733a53c3d#diff-4601ee6fcb99859fee9b658744fa9024|93e67ed]] added a test case for "Java API Rhizome fetch manifest" (**[[#R13]]**, **[[#R4]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/34b6ff48bf9c44808867c3d37e2c4fc9677dcdd7#diff-4601ee6fcb99859fee9b658744fa9024|34b6ff4]] added a test case for "Java API Rhizome fetch raw payload" (**[[#R13]]**, **[[#R5]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/2aec8f31a4ea71196d6041a3a59421884caaa0f0#diff-4601ee6fcb99859fee9b658744fa9024|2aec8f3]] added a test case for "Java API Rhizome fetch decrypted payload" (**[[#R13]]**, **[[#R6]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/3715c5bf0b2be95bc773eb7e8225079cb55d7c81#diff-4601ee6fcb99859fee9b658744fa9024|3715c5b]] added four negative test cases for "Java API Rhizome fetch" (**[[#R13]]**, **[[#R4]]**, **[[#R5]]**, **[[#R6]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/a81d05b4f6db84c732e7c2c59a7288e4d16fe29d#diff-4601ee6fcb99859fee9b658744fa9024|a81d05b]] added a test case for "Java API Rhizome insert bundle" (**[[#R13]]**, **[[#R7]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/a87133d4d361edd602617092f54040e21f5335a3#diff-4601ee6fcb99859fee9b658744fa9024|a87133d]] added a negative test case for "Java API Rhizome insert bundle" (**[[#R13]]**, **[[#R7]]**) -- can update an anonymous bundle if secret is provided
    * commit [[https://github.com/servalproject/serval-dna/commit/e35bf77938a036b059f926e3d00d23dd3d2f027c#diff-4601ee6fcb99859fee9b658744fa9024|e35bf77]] added two test cases for inserting empty payloads (**[[#R13]]**, **[[#R7]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/8842f32b193876723dce5807fcaadd791ed75a5e#diff-4601ee6fcb99859fee9b658744fa9024|8842f32]] added a test case for inserting journal bundle not implemented (**[[#R13]]**, **[[#N7]]**)
    * commit [[https://github.com/servalproject/serval-dna/commit/2757a08b87d3614f48f6e7c528280de25332dbff#diff-4601ee6fcb99859fee9b658744fa9024|2757a08]] added two negative cases for inserting an inconsistent bundle (**[[#R13]]**, **[[#N7]]**)

  * The Java tests were added to the [[https://github.com/servalproject/serval-dna/blob/development/tests/all|tests/all]] script:
    * commit [[https://github.com/servalproject/serval-dna/commit/f948f8e12e0439a239e7362b446fe37e9af3ce40|f948f8e]] added the JNI and MeshMS Java API tests, only run if the JDK is present
    * commit [[https://github.com/servalproject/serval-dna/commit/db8ee79a4ee11bc0ddf48e9c451fa813b9844851#diff-7|db8ee79]] added the new Rhizome Java API test script
    * commit [[https://github.com/servalproject/serval-dna/commit/c5d3069d823f52494ed5095529ab60fbdab0a8fd|c5d3069]] ensured that the JNI API library is always built by the ''make all'' command

The test code coverage for the new C source code created by this contract exceeds the average:

^                        ^ Lines                ^^ Functions          ^^
| ''http_server.c''      |    70% |     927/1325 |  95.5% |      64/67 |
| ''meshms_restful.c''   |    89% |      316/355 |   100% |      21/21 |
| ''rhizome_restful.c''  |    82% |      410/500 |   100% |      24/24 |
^ Average for all Serval DNA source code                           ^^^^^
| ''*.c *.h''            |  72.2% |  17904/24796 |  81.9% |  1516/1851 |

There are no code coverage statistics for the Java source code.

**[[#R15]]** -- upgrade Serval Mesh app for Android (completed 25 June 2014)

  * The [[:content:servalmesh:|Serval Mesh app for Android]] was improved to use the Java APIs:
    * commit [[https://github.com/servalproject/batphone/commit/73e1a70ba4e343defe8f7fcdc90efe5afb03653e|73e1a70]] caused MeshMS UI to start the daemon on demand and use the Java API developed in **[[#R13]]** to list conversations (**[[#R9]]**) and messages (**[[#R10]]**)
    * commit [[https://github.com/servalproject/batphone/commit/8c71c3418f7f97f72a4686d7c0d362cfd0ecaa85|8c71c34]] caused MeshMS UI to use the Java API developed in **[[#R13]]** for sending (**[[#R12]]**) and marking messages as read (**[[#N16]]**)

**Merge** into mainline development branch (completed 11 July 2014)
  * commit [[https://github.com/servalproject/serval-dna/commit/606f087dd5fe3cc3a22569ae2050dea447b2d2bf|606f087]] merged all work into the mainline development branch
The Serval Project Wiki

User Tools

Site Tools

Page Tools
