User Tools

Site Tools


content:activity:naf4

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
content:activity:naf4 [11/07/2014 00:06]
Andrew Bettison [PROGRESS REPORT] complete R13 Java API
content:activity:naf4 [14/07/2014 19:17]
Andrew Bettison [COMPLETION REPORT] add test code coverage information to R14
Line 141: Line 141:
     * ''POST /restful/meshms/<SID>/<SID>/recv/<offset>/read'' advances the read-offset in a single conversation, to mark a single message (and all its predecessors) as read     * ''POST /restful/meshms/<SID>/<SID>/recv/<offset>/read'' advances the read-offset in a single conversation, to mark a single message (and all its predecessors) as read
  
-===== PROGRESS REPORT =====+===== COMPLETION REPORT ===== 
 + 
 +The work was completed on 14 July 2014, having been performed concurrently with three other contracts: 
 +  * [[naf5]] 
 +  * [[naf6]] 
 +  * [[openitp2]]
  
 ==== Preliminary work ==== ==== Preliminary work ====
 +
 +This contract drove several significant architectural improvements to [[:content:servaldna:]] and [[:content:servalmesh:]].
  
 == Security: SQL injections == == Security: SQL injections ==
Line 168: Line 175:
 == Re-organisation around RESTful API == == Re-organisation around RESTful API ==
 Henceforward the RESTful API will be the preferred (and eventually only) method for applications to use Rhizome and MeshMS. Henceforward the RESTful API will be the preferred (and eventually only) method for applications to use Rhizome and MeshMS.
-  * The existing Rhizome and MeshMS JNI API provided function-call entry points into the Serval DNA shared library +  * The existing JNI API for Rhizome and MeshMS was inferior: 
-  That meant that application processes opened and accessed the Rhizome database themselves, not via requests to the daemon +    * the JNI API provided function-call entry points into the Serval DNA shared library 
-  This sometimes caused database locking to block the daemon, which severely degraded responsiveness when starting a freshly-installed Batphone app in the presence of other devices holding lots of Rhizome content -- see [[https://github.com/servalproject/serval-dna/issues/1|Serval DNA issue #1]] +    * application processes opened and accessed the Rhizome database themselves, not via requests to the daemon 
-  The JNI API also required applications to be part of the same Android APK file, so that Android would run them all with the same User ID, so they could all access and modify the Rhizome database files+    this sometimes caused database locking to block the daemon, which severely degraded responsiveness when starting a freshly-installed Batphone app in the presence of other devices holding lots of Rhizome content -- see [[https://github.com/servalproject/serval-dna/issues/1|Serval DNA issue #1]] 
 +    all Android applications using MeshMS and Rhizome had to be bundled together in a single APK, so that Android would run them all with the same User ID, so they could all access and modify the Rhizome database files
   * Commit [[https://github.com/servalproject/serval-dna/commit/68dbaef38d03535271eed14a6da35323a0dd7a4f|68dbaef]] refactored the Java API to add a stateful object that holds an open connection to the daemon, and keep the Serval DNA RESTful interface independent from the JNI command interface   * Commit [[https://github.com/servalproject/serval-dna/commit/68dbaef38d03535271eed14a6da35323a0dd7a4f|68dbaef]] refactored the Java API to add a stateful object that holds an open connection to the daemon, and keep the Serval DNA RESTful interface independent from the JNI command interface
   * Commit [[https://github.com/servalproject/serval-dna/commit/82b13caac4c2386a096acc5c72ebde2b4871c245|82b13ca]] deprecated the existing MeshMS JNI API   * Commit [[https://github.com/servalproject/serval-dna/commit/82b13caac4c2386a096acc5c72ebde2b4871c245|82b13ca]] deprecated the existing MeshMS JNI API
Line 178: Line 186:
 == Serval DNA power consumption == == Serval DNA power consumption ==
 The final issue to be addressed was the CPU activity of the [[:content:servaldna:]] daemon whilst idle. The final issue to be addressed was the CPU activity of the [[:content:servaldna:]] daemon whilst idle.
-  * The Rhizome and MeshMS screens must function even when Serval Mesh services are not enabled +  * The previous implementation of the [[:content:servalmesh:|Serval Mesh app for Android]] did not have a continuously running daemon: 
-  * The previous implementation of the [[:content:servalmesh:|Serval Mesh app for Android]] did not require a continuously running daemon: +    * the daemon only ran while the "Enable Services" checkbox was ticked on the "Connect" menu 
-    * the daemon was only run while the "Enable Services" checkbox was ticked on the "Connect" menu +    * the daemon awoke periodically on *alarm* events (once per second) while running, even while the network was idle or no network interfaces were up 
-    * the daemon used timed *alarm* events to wake it at short intervals (once per second) while running, even while the network was idle or no network interfaces were up +    * this caused rapid battery drain while Serval Mesh networking was active, which was advised in the [[https://github.com/servalproject/batphone/blob/0.91/doc/RELEASE-0.91.md|Release Notes for 0.91]] 
-  * The new implementation implements all MeshMS operations using the RESTful HTTP interface (**[[#R15]]**), which requires a continuously running daemon +    * the JNI API (see above) accessed the Rhizome database directly in-process; it did not use the daemon, so the Rhizome and MeshMS screens functioned at all times 
-  This would mean that the new implementation would quickly drain the battery unless the [[:content:servaldna:]] CPU usage were close to zero while idle+  * The new implementation (**[[#R15]]**) implements MeshMS operations using the RESTful HTTP interface: 
 +    * this requires a continuously running daemon, or at least a daemon running while any Rhizome or MeshMS screen are active 
 +    this could quickly drain the battery simply by opening the app 
 +  * This necessitated improvements to the event scheduling sub-system within [[:content:servaldna:]]
 +    * only use CPU in response to real external events (user and network) 
 +    * only schedule timed wake-ups for externally imposed timing requirements such as network protocol time-outs
   * Commit [[https://github.com/servalproject/serval-dna/commit/6d4ad0e1505d8826cda024159892b82699e392c7|6d4ad0e]] improved the event alarm scheduler [[https://github.com/servalproject/serval-dna/blob/development/fdqueue.c|fdqueue.c]]:   * Commit [[https://github.com/servalproject/serval-dna/commit/6d4ad0e1505d8826cda024159892b82699e392c7|6d4ad0e]] improved the event alarm scheduler [[https://github.com/servalproject/serval-dna/blob/development/fdqueue.c|fdqueue.c]]:
     * a new kind of "lazy" alarm does not force the daemon process to wake     * a new kind of "lazy" alarm does not force the daemon process to wake
Line 192: Line 205:
     * starts the daemon as needed and keeps it running     * starts the daemon as needed and keeps it running
     * changing the "Enable Services" checkbox simply toggles a Serval DNA configuration option     * changing the "Enable Services" checkbox simply toggles a Serval DNA configuration option
 +  * This improvement has significant positive repercussions for other Serval products like [[:content:meshextender:]], by extending battery life towards the maximum practical limit
  
 ==== Implementation ==== ==== Implementation ====
Line 281: Line 295:
   * commit [[https://github.com/servalproject/serval-dna/commit/0727fb3b62ad6ff6c23154ef2853314ae28eeabf|0727fb3]] merged all work up to **[[#R12]]** into the mainline development branch   * commit [[https://github.com/servalproject/serval-dna/commit/0727fb3b62ad6ff6c23154ef2853314ae28eeabf|0727fb3]] merged all work up to **[[#R12]]** into the mainline development branch
  
-**[[#R13]]** -- Java API (completed 11 July 2014)+**[[#R13]]** -- Java API (completed 14 July 2014)
   * commit [[https://github.com/servalproject/serval-dna/commit/99fb8b6108021842ddaa82d95e17fe1f2f1347b8|99fb8b6]] moved the Java API foundation classes from the Batphone source repository to the Serval DNA repository   * commit [[https://github.com/servalproject/serval-dna/commit/99fb8b6108021842ddaa82d95e17fe1f2f1347b8|99fb8b6]] moved the Java API foundation classes from the Batphone source repository to the Serval DNA repository
   * commit [[https://github.com/servalproject/serval-dna/commit/d879189299b2978b5cb419f7eb91f42a89479407|d879189]] fixed a bug in Serval DNA's HTTP server code (from **[[#R1a]]**) that produced a 500 Server Error if the Basic Authorization string was long   * commit [[https://github.com/servalproject/serval-dna/commit/d879189299b2978b5cb419f7eb91f42a89479407|d879189]] fixed a bug in Serval DNA's HTTP server code (from **[[#R1a]]**) that produced a 500 Server Error if the Basic Authorization string was long
Line 305: Line 319:
   * commit [[https://github.com/servalproject/serval-dna/commit/61023287b16e25d7a4e504ddc0815ed2125f1943|6102328]] improved Serval DNA's Rhizome manifest validation code to provide a textual reason for failure, which can be passed back to API clients to assist diagnosis   * commit [[https://github.com/servalproject/serval-dna/commit/61023287b16e25d7a4e504ddc0815ed2125f1943|6102328]] improved Serval DNA's Rhizome manifest validation code to provide a textual reason for failure, which can be passed back to API clients to assist diagnosis
   * commit [[https://github.com/servalproject/serval-dna/commit/db8ee79a4ee11bc0ddf48e9c451fa813b9844851|db8ee79]] implemented a Java API to list Rhizome bundles:   * commit [[https://github.com/servalproject/serval-dna/commit/db8ee79a4ee11bc0ddf48e9c451fa813b9844851|db8ee79]] implemented a Java API to list Rhizome bundles:
-    * uses the RESTful HTTP interface developed in **[[#R3]]**+    * uses the RESTful HTTP interface developed in **[[#R2]]**
     * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/rhizome/RhizomeCommon.java|org.servalproject.servaldna.rhizome.RhizomeCommon]] contains methods for parsing a Rhizome status report JSON (**[[#N9a]]**)     * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/servaldna/rhizome/RhizomeCommon.java|org.servalproject.servaldna.rhizome.RhizomeCommon]] contains methods for parsing a Rhizome status report JSON (**[[#N9a]]**)
     * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/json/JSONTableScanner.java|org.servalproject.json.JSONTableScanner]] generalises the parsing of the JSON table structure returned by Rhizome and MeshMS lists (**[[#N1]]** and **[[#N10]]**)     * new class [[https://github.com/servalproject/serval-dna/blob/development/java/org/servalproject/json/JSONTableScanner.java|org.servalproject.json.JSONTableScanner]] generalises the parsing of the JSON table structure returned by Rhizome and MeshMS lists (**[[#N1]]** and **[[#N10]]**)
Line 321: Line 335:
   * commit [[https://github.com/servalproject/serval-dna/commit/04b2a20e54fc8d55dc6187ad21cf3c533d385b3c|04b2a20]] improved the Content-Type header and added Content-Transfer-Encoding headers sent by the Java API *insert bundle* operation, to help future-proof the protocol   * commit [[https://github.com/servalproject/serval-dna/commit/04b2a20e54fc8d55dc6187ad21cf3c533d385b3c|04b2a20]] improved the Content-Type header and added Content-Transfer-Encoding headers sent by the Java API *insert bundle* operation, to help future-proof the protocol
   * commit [[https://github.com/servalproject/serval-dna/commit/8842f32b193876723dce5807fcaadd791ed75a5e|8842f32]] ensured that inserting a journal bundle returns the 501 "Not Implemented" HTTP code (**[[#N7]]**)   * commit [[https://github.com/servalproject/serval-dna/commit/8842f32b193876723dce5807fcaadd791ed75a5e|8842f32]] ensured that inserting a journal bundle returns the 501 "Not Implemented" HTTP code (**[[#N7]]**)
-  * commit [[https://github.com/servalproject/serval-dna/commit/606f087dd5fe3cc3a22569ae2050dea447b2d2bf|606f087]] merged the completed Rhizome Java API into the development branch for use in Batphone (**[[#R15]]**)+  * commit [[https://github.com/servalproject/serval-dna/commit/606f087dd5fe3cc3a22569ae2050dea447b2d2bf|606f087]] merged the almost-complete Rhizome Java API into the development branch for use in Batphone (**[[#R15]]**) 
 +  * commit [[https://github.com/servalproject/serval-dna/commit/4c6612a61e2e5ce46d840bf6f0ae3771a6583446|4c6612a]] implemented a Java API to list Rhizome bundles that have arrived since a given token: 
 +    * uses the RESTful HTTP interface developed in **[[#R3]]** 
 +  * commit [[https://github.com/servalproject/serval-dna/commit/89623015fcfb75a6b8adccb6bf785a9e46cb26e5|8962301]] implemented a Java API to list MeshMS messages that have arrived since a given token from a single conversation: 
 +    * uses the RESTful HTTP interface developed in **[[#R11]]**
  
-**[[#R14]]** -- Automated tests (in progress, 25 June 2014)+**[[#R14]]** -- Automated tests (completed 11 July 2014)
  
   * The **[[https://github.com/servalproject/serval-dna/blob/development/tests/meshms|tests/meshms]]** script contains all the tests that were created when the MeshMS protocol was moved from [[:content:servalmesh:]] (Java) into [[:content:servaldna:]] (C) (**[[#R8]]**).   * The **[[https://github.com/servalproject/serval-dna/blob/development/tests/meshms|tests/meshms]]** script contains all the tests that were created when the MeshMS protocol was moved from [[:content:servalmesh:]] (Java) into [[:content:servaldna:]] (C) (**[[#R8]]**).
Line 378: Line 396:
  
   * Created the new **[[https://github.com/servalproject/serval-dna/blob/development/tests/meshmsjava|meshmsjava]]** test script containing tests for the new MeshMS Java API:   * Created the new **[[https://github.com/servalproject/serval-dna/blob/development/tests/meshmsjava|meshmsjava]]** test script containing tests for the new MeshMS Java API:
-    * commit [[https://github.com/servalproject/serval-dna/commit/9cbd7c365cc1a9c1f94ff4503d06b520a4e06f49#diff-8|9cbd7c3]] added a new test case for "Java API MeshMS list conversations" (**[[#R13]]**) +    * commit [[https://github.com/servalproject/serval-dna/commit/9cbd7c365cc1a9c1f94ff4503d06b520a4e06f49#diff-8|9cbd7c3]] added a new test case for "Java API MeshMS list conversations" (**[[#R13]]**, **[[#R9]]**) 
-    * commit [[https://github.com/servalproject/serval-dna/commit/e9437e9a61ad6248c6ae0084521ffafc51c09b68#diff-10|e9437e9]] added two new test cases for "Java API MeshMS list messages" (**[[#R13]]**)+    * commit [[https://github.com/servalproject/serval-dna/commit/e9437e9a61ad6248c6ae0084521ffafc51c09b68#diff-10|e9437e9]] added two new test cases for "Java API MeshMS list messages" (**[[#R13]]**, **[[#R10]]**) 
 +    * commit [[https://github.com/servalproject/serval-dna/commit/89623015fcfb75a6b8adccb6bf785a9e46cb26e5#diff-0371afbd127921209ea37daba1e3ef47|8962301]] added two new test cases for "Java API MeshMS list new messages since token" (**[[#R13]]**, **[[#R11]]**) 
 +    * commit [[https://github.com/servalproject/serval-dna/commit/eba7f6555f22aa4be8ba1fe99cd25ba632826349#diff-0371afbd127921209ea37daba1e3ef47|eba7f65]] added two new test cases for "Java API MeshMS send message" (**[[#R13]]**, **[[#R12]]**) 
 +    * commit [[https://github.com/servalproject/serval-dna/commit/c79a382a27252c6fa201d68a24a244c52c0685c0#diff-0371afbd127921209ea37daba1e3ef47|c79a382]] added three new test cases for "Java API MeshMS mark messages as read" (**[[#R13]]**, **[[#N16]]**)
  
   * Created the new **[[https://github.com/servalproject/serval-dna/blob/development/tests/rhizomejava|rhizomejava]]** test script containing tests for the new Rhizome Java API:   * Created the new **[[https://github.com/servalproject/serval-dna/blob/development/tests/rhizomejava|rhizomejava]]** test script containing tests for the new Rhizome Java API:
     * commit [[https://github.com/servalproject/serval-dna/commit/a9ec5dcf695ece8363f838fc71cd25f92a1ed0bc|a9ec5dc]] refactored some shell functions from the existing [[https://github.com/servalproject/serval-dna/blob/development/tests/rhizomerestful|rhizomerestful]] script into [[https://github.com/servalproject/serval-dna/blob/development/testdefs_rhizome.sh|testdefs_rhizome.sh]] ready for re-use     * commit [[https://github.com/servalproject/serval-dna/commit/a9ec5dcf695ece8363f838fc71cd25f92a1ed0bc|a9ec5dc]] refactored some shell functions from the existing [[https://github.com/servalproject/serval-dna/blob/development/tests/rhizomerestful|rhizomerestful]] script into [[https://github.com/servalproject/serval-dna/blob/development/testdefs_rhizome.sh|testdefs_rhizome.sh]] ready for re-use
 +    * commit [[https://github.com/servalproject/serval-dna/commit/db8ee79a4ee11bc0ddf48e9c451fa813b9844851#diff-4601ee6fcb99859fee9b658744fa9024|db8ee79]] created the new script and added a test case for "Java API Rhizome list bundles" (**[[#R13]]**, **[[#R2]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/4c6612a61e2e5ce46d840bf6f0ae3771a6583446#diff-4601ee6fcb99859fee9b658744fa9024|4c6612a]] added a test case for "Java API Rhizome list new bundles since a token" (**[[#R13]]**, **[[#R3]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/93e67ede635930e678ac2714b0436de733a53c3d#diff-4601ee6fcb99859fee9b658744fa9024|93e67ed]] added a test case for "Java API Rhizome fetch manifest" (**[[#R13]]**, **[[#R4]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/34b6ff48bf9c44808867c3d37e2c4fc9677dcdd7#diff-4601ee6fcb99859fee9b658744fa9024|34b6ff4]] added a test case for "Java API Rhizome fetch raw payload" (**[[#R13]]**, **[[#R5]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/2aec8f31a4ea71196d6041a3a59421884caaa0f0#diff-4601ee6fcb99859fee9b658744fa9024|2aec8f3]] added a test case for "Java API Rhizome fetch decrypted payload" (**[[#R13]]**, **[[#R6]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/3715c5bf0b2be95bc773eb7e8225079cb55d7c81#diff-4601ee6fcb99859fee9b658744fa9024|3715c5b]] added four negative test cases for "Java API Rhizome fetch" (**[[#R13]]**, **[[#R4]]**, **[[#R5]]**, **[[#R6]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/a81d05b4f6db84c732e7c2c59a7288e4d16fe29d#diff-4601ee6fcb99859fee9b658744fa9024|a81d05b]] added a test case for "Java API Rhizome insert bundle" (**[[#R13]]**, **[[#R7]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/a87133d4d361edd602617092f54040e21f5335a3#diff-4601ee6fcb99859fee9b658744fa9024|a87133d]] added a negative test case for "Java API Rhizome insert bundle" (**[[#R13]]**, **[[#R7]]**) -- can update an anonymous bundle if secret is provided
 +    * commit [[https://github.com/servalproject/serval-dna/commit/e35bf77938a036b059f926e3d00d23dd3d2f027c#diff-4601ee6fcb99859fee9b658744fa9024|e35bf77]] added two test cases for inserting empty payloads (**[[#R13]]**, **[[#R7]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/8842f32b193876723dce5807fcaadd791ed75a5e#diff-4601ee6fcb99859fee9b658744fa9024|8842f32]] added a test case for inserting journal bundle not implemented (**[[#R13]]**, **[[#N7]]**)
 +    * commit [[https://github.com/servalproject/serval-dna/commit/2757a08b87d3614f48f6e7c528280de25332dbff#diff-4601ee6fcb99859fee9b658744fa9024|2757a08]] added two negative cases for inserting an inconsistent bundle (**[[#R13]]**, **[[#N7]]**)
  
   * The Java tests were added to the [[https://github.com/servalproject/serval-dna/blob/development/tests/all|tests/all]] script:   * The Java tests were added to the [[https://github.com/servalproject/serval-dna/blob/development/tests/all|tests/all]] script:
-    * commit [[https://github.com/servalproject/serval-dna/commit/f948f8e12e0439a239e7362b446fe37e9af3ce40|f948f8e]] runs the Java tests only if the JDK is present+    * commit [[https://github.com/servalproject/serval-dna/commit/f948f8e12e0439a239e7362b446fe37e9af3ce40|f948f8e]] added the JNI and MeshMS Java API testsonly run if the JDK is present 
 +    * commit [[https://github.com/servalproject/serval-dna/commit/db8ee79a4ee11bc0ddf48e9c451fa813b9844851#diff-7|db8ee79]] added the new Rhizome Java API test script
     * commit [[https://github.com/servalproject/serval-dna/commit/c5d3069d823f52494ed5095529ab60fbdab0a8fd|c5d3069]] ensured that the JNI API library is always built by the ''make all'' command     * commit [[https://github.com/servalproject/serval-dna/commit/c5d3069d823f52494ed5095529ab60fbdab0a8fd|c5d3069]] ensured that the JNI API library is always built by the ''make all'' command
 +
 +The test code coverage for the new C source code created by this contract exceeds the average:
 +
 +^                        ^ Lines                ^^ Functions          ^^
 +| ''http_server.c''      |    70% |     927/1325 |  95.5% |      64/67 |
 +| ''meshms_restful.c''      89% |      316/355 |   100% |      21/21 |
 +| ''rhizome_restful.c''  |    82% |      410/500 |   100% |      24/24 |
 +^ Average for all Serval DNA source code                           ^^^^^
 +| ''*.c *.h''            |  72.2% |  17904/24796 |  81.9% |  1516/1851 |
 +
 +There are no code coverage statistics for the Java source code.
  
 **[[#R15]]** -- upgrade Serval Mesh app for Android (completed 25 June 2014) **[[#R15]]** -- upgrade Serval Mesh app for Android (completed 25 June 2014)
Line 393: Line 437:
     * commit [[https://github.com/servalproject/batphone/commit/73e1a70ba4e343defe8f7fcdc90efe5afb03653e|73e1a70]] caused MeshMS UI to start the daemon on demand and use the Java API developed in **[[#R13]]** to list conversations (**[[#R9]]**) and messages (**[[#R10]]**)     * commit [[https://github.com/servalproject/batphone/commit/73e1a70ba4e343defe8f7fcdc90efe5afb03653e|73e1a70]] caused MeshMS UI to start the daemon on demand and use the Java API developed in **[[#R13]]** to list conversations (**[[#R9]]**) and messages (**[[#R10]]**)
     * commit [[https://github.com/servalproject/batphone/commit/8c71c3418f7f97f72a4686d7c0d362cfd0ecaa85|8c71c34]] caused MeshMS UI to use the Java API developed in **[[#R13]]** for sending (**[[#R12]]**) and marking messages as read (**[[#N16]]**)     * commit [[https://github.com/servalproject/batphone/commit/8c71c3418f7f97f72a4686d7c0d362cfd0ecaa85|8c71c34]] caused MeshMS UI to use the Java API developed in **[[#R13]]** for sending (**[[#R12]]**) and marking messages as read (**[[#N16]]**)
 +
 +**Merge** into mainline development branch (completed 11 July 2014)
 +  * commit [[https://github.com/servalproject/serval-dna/commit/606f087dd5fe3cc3a22569ae2050dea447b2d2bf|606f087]] merged all work into the mainline development branch
  
content/activity/naf4.txt · Last modified: 21/02/2016 22:40 by Andrew Bettison