The Serval Project Wiki

User Tools

Site Tools

You are here: The Serval Project Wiki » content » Technology Roadmap » Rhizome
Trace:
content:tech:rhizome

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Last revision Both sides next revision
content:tech:rhizome [20/03/2013 00:14]
Andrew Bettison update a few details 		content:tech:rhizome [14/12/2015 08:22]
Paul Gardner-Stephen [Technical documentation]
Line 1: Line 1:
 ====== Rhizome ====== ====== Rhizome ======
  
-**Rhizome** is a file distribution system developed by [[:|The Serval Project]] that enables files to be propagated over different network topologies. Rhizome can distribute files via mesh, direct connect and client-server connections.+**Rhizome** is a resilient content distribution system developed by [[:|The Serval Project]].  It forms the basis of all non-[[http://en.wikipedia.org/wiki/Connection_oriented|connection-oriented]] services provided by [[:content:servalmesh:|Serval Mesh]], such as [[meshms|text messaging]], file sharing, voice mail, and automatic software upgrades.  
 + 
 +==== Technical documentation==== 
 + 
 +Authoritative documentation of Rhizome now exists in the source code of the Serval Mesh software, and should be used in preference to the older documents on this wiki. 
 + 
 +  * Rhizome REST API [[https://github.com/servalproject/serval-dna/blob/development/doc/Servald-REST-API.md]]  
 + 
 +===== Older Documentation 
 + 
 +See note above, as there exists important updates to much of this documentation, and which is documented directly in the source code repositories. 
 + 
 +  * [[Rhizome API]] describes the APIs for applications that use Rhizome 
 +  * [[Security Framework]] describes the cryptographic features of Rhizome 
 +  * [[:content:activity:naf4]] describes the original development of the Rhizome HTTP REST API (beware: some details will be out of date)
  
 ==== Purpose and features ==== ==== Purpose and features ====
  
-  * Rhizome is a system for transporting units of data called **payloads** ("files") between **applications** running on nodes in a digital [[http://en.wikipedia.org/wiki/Telecommunications_network|telecommunications network]] +  * Rhizome is a **[[http://en.wikipedia.org/wiki/Store_and_forward|store and forward]]** service for transporting units of data called **payloads** ("files") between **applications** running on nodes in a digital [[http://en.wikipedia.org/wiki/Telecommunications_network|telecommunications network]] 
-  * Rhizome uses **[[http://en.wikipedia.org/wiki/Store_and_forward|store and forward]]** to propagate payloads +  * Rhizome is designed for [[http://en.wikipedia.org/wiki/Wireless_mesh_network|wireless mesh networks]] such as [[MDP]] in the [[:content:servalmesh:|Serval Mesh]], but can also be used within a fixed network infrastructure
-  * Rhizome makes no guarantees about delivery time or eventual arrival+
   * Rhizome guarantees that payloads will remain intact wherever they are transported   * Rhizome guarantees that payloads will remain intact wherever they are transported
-  * Rhizome aims to make the best effort to transport payloads to wherever they are wanted, subject to the constraints of the network (topology, bandwidth) and its nodes (storage, battery), without favouring any payloads above others of equal kind +  * Rhizome makes no guarantees about delivery time or eventual arrival 
-  * Rhizome is neutral with respect to originating and receiving nodes and applications +  * Rhizome makes its best effort to transport payloads to wherever they are wanted, subject to the constraints of the network (topology, bandwidth) and its nodes (storage, battery), without favouring any payloads above others of equal kind 
-  * Rhizome uses cryptography to guarantee the integrity of payloads by preventing any party except the originator from modifying a payload or its meta data +  * Rhizome is neutral with respect to originating and receiving nodes, applications, and payload content 
 +  * Rhizome uses cryptography to preserve the integrity and privacy of payloads 
 +  * Rhizome protects anonymous senders by not storing or transmitting information that may allow senders to be traced, even on the originating device 
 +  * Rhizome functions without centralised infrastructure such as well-known servers or a persistent network connection 
 +  * Rhizome can be configured to use central servers to improve efficiency and reach
 ==== Rhizome applications ==== ==== Rhizome applications ====
  
-  * Users do not use Rhizome directly, but //indirectly via [[http://en.wikipedia.org/wiki/Application_software|applications]]// which use Rhizome as their data transport, eg, [[:content:technologies:meshms|MeshMS]] and [[:content:servalmaps:|Serval Maps]]+  * Users do not use Rhizome directly, but //indirectly via [[http://en.wikipedia.org/wiki/Application_software|applications]]// which use Rhizome as their data transport 
 +  * Current Rhizome applications: 
 +    * [[MeshMS]], the simple text messaging service of the [[:content:servalmesh:|Serval Mesh]] app for Android 
 +    * [[:content:servalmaps:|Serval Maps]], an Android app produced by the [[:|Serval Project]], uses Rhizome to share geo-tagged data like photos and documents 
 +    * the [[:content:servalmesh:|Serval Mesh]] app for Android uses Rhizome to distribute software upgrades 
 +    * the file sharing service of the [[:content:servalmesh:|Serval Mesh]] app for Android is a simple UI for sharing, browsing and saving/viewing non-MeshMS content in Rhizome 
 +    * [[Serval SAM]] is a crowd survey app for Android that allows the user to fill in an [[http://opendatakit.org/|Open Data Kit]] form and uses Rhizome to transport completed forms to a central collection server 
 +    * [[Sensor Logger]], an Android app produced by [[:content:about]], uses Rhizome to send continuous accelerometer logs to the [[:content:seniordevs:servalhq|Serval HQ]] central collection server  
 +  * Future applications may include: 
 +    * voice mail 
 +    * disaster response situational awareness 
 +    * medical or environmental sensor data collection 
 +    * collaborative text editing and whiteboarding 
 +    * distributed document management 
 +    * citizen journalism 
 +    * classroom and remote teaching tools 
 +    * group chat 
 +    * Wikipedia caching
   * Applications **inject** data into Rhizome by supplying a payload and some meta data   * Applications **inject** data into Rhizome by supplying a payload and some meta data
   * Applications **extract** data from Rhizome, receiving the payload and all meta data   * Applications **extract** data from Rhizome, receiving the payload and all meta data
Line 21: Line 54:
 ==== Rhizome's data model ==== ==== Rhizome's data model ====
  
-  * Every Rhizome payload is accompanied by some structured meta data called a **manifest**: +Every file distributed by Rhizome ("payload") is accompanied by some structured meta-data called a **[[Rhizome manifest|manifest]]**. 
-    * some meta data is supplied by the originating application, eg, ''name'', ''mimetype'' + 
-    * most meta data is created by Rhizome to identify and prioritise the payload, eg, ''id'', ''filesize'', ''filehash'', ''version'', etc. +A payload plus a [[Rhizome manifest|manifest]] together form a **[[Rhizome bundle]]**. 
-  * A manifest together with its payload is called a **bundle** + 
-  * A manifest may be present on many nodes at a time +==== Rhizome's data transport ==== 
-  * A payload may be present on many nodes at a time+ 
 +  * A **Rhizome node** is any network node (device) that is running the Rhizome software 
 +  * A Rhizome node exchanges manifests and payloads directly with adjacent nodes on the network 
 +  * Rhizome bundles are transferred between nodes automatically and autonomously, ie, without user intervention, using any available network connection 
 +  * A manifest may be stored on many nodes at a time 
 +  * A payload may be stored on many nodes at a time
   * A manifest may persist for any length of time on any node   * A manifest may persist for any length of time on any node
   * A payload may persist for any length of time on any node   * A payload may persist for any length of time on any node
-  * Rhizome does not modify any manifest once it is created +  * Each Rhizome node advertises its recently acquired bundles to its neighbours
-  * Rhizome does not modify any payload +
-  * A bundle may indicate the [[:content:technologies:sid|Serval Identity]] of its intended recipient, otherwise it is for general consumption (broadcast) +
-  * A payload may indicate the [[:content:technologies:sid|Serval Identity]] of its sender, otherwise it is of unknown origin (anonymous) +
-  * Rhizome enforces a set of logical conditions on every bundle it receives from any source: +
-    * certain fields must always be present +
-    * each field must have a certain length and/or format+
  
-== Rhizome manifest ==+== Rhizome store ==
  
-The Rhizome manifest is a set of key-value pairs called **"fields"**.+  * A Rhizome node contains a **Rhizome store** 
 +  * A Rhizome store is a database containing payloads indexed by PH (Payload Hash) and manifests indexed by BID (Bundle ID) 
 +  * A Rhizome store may reside in volatile (RAM) or non-volatile (Flash, disk) local storage 
 +  * The original implementation of the Rhizome store used [[http://www.sqlite.org/|SQLite]] for the database
  
-The Rhizome manifest is designed for forward and backward compatibility between different versions of the Rhizome software: +== Rhizome Rank ==
-  * Every Rhizome node uses the manifest fields it recognises and silently ignores those it does not +
-  * Every Rhizome node preserves all manifest fields intact when it stores the manifest and when it transmits the manifest to other nodes+
  
-==== Rhizome nodes ====+  * Every Rhizome node maintains its own ordered list of all the manifests in its store, called the **Rank** 
 +  * Every Rhizome node attempts to fetch missing payloads from neighbouring nodes, in order of rank 
 +  * Every Rhizome node recovers local storage space by evicting its lowest ranked payloads 
 +  * A bundle's position in the rank depends on meta data in the manifest and local node state, which may include: 
 +    * own Serval Identities 
 +    * Serval Identities of recently reachable nodes 
 +    * current time 
 +    * current or recent geographic location
  
-  * A **Rhizome node** is any network node (device) that is running the Rhizome software +== Network connections == 
-  * Payloads and manifests are copied directly from node to node using network transfers + 
-  * Payloads and manifests are stored on nodes using volatile (RAM) or non-volatile (Flash, disk) local storage +  * Rhizome nodes may transfer bundles using **[[http://en.wikipedia.org/wiki/Network_layer|network layer]]** connections over established networks, eg, [[http://en.wikipedia.org/wiki/Transmission_Control_Protocol|TCP]] or [[http://en.wikipedia.org/wiki/User_Datagram_Protocol|UDP]] or [[http://en.wikipedia.org/wiki/Http|HTTP]] over public Internet or cellular GSM 
-  * Each Rhizome node actively transmits its recently acquired manifests to its neighbours +  * Adjacent Rhizome nodes may transfer bundles over **[[http://en.wikipedia.org/wiki/Data_link_layer|data link layer]]** connections, eg, [[MDP]] to [[:content:servalmesh:|Serval Mesh]] devices within Fi-Fi range
-  * Each Rhizome node records all received manifests in a sorted list called the **rank** +
-  * Each Rhizome node attempts to acquire the payloads of its highest ranked manifests from neighbouring nodes +
-  * Each Rhizome node recovers local storage space by evicting its lowest ranked payloads and manifests+
  
 ==== Neutrality ==== ==== Neutrality ====
Line 63: Line 100:
   * Rhizome does not rank (prioritise) based on the content of payloads   * Rhizome does not rank (prioritise) based on the content of payloads
  
-==== Bundle integrity ==== 
  
-  * Rhizome creates a random 256-bit Elliptic Curve private key for every bundle, called the **Bundle Secret** or **BS** 
-  * The 256-bit public key derived from the Bundle Secret is called the **Bundle ID** or **BID** 
-  * Rhizome does not reveal the Bundle Secret to any other application or node 
-  * Rhizome computes the 512-bit SHA512 hash of the payload, called the **Payload Hash** or **PH** (also **File Hash** or **FH**) 
-  * At injection, Rhizome inserts the Payload Hash into the bundle's manifest 
-  * Rhizome signs every complete manifest using the Bundle Secret 
-  * Rhizome nodes will only store and transmit manifests with valid signatures 
-  * Only an application possessing the original Bundle Secret can alter the manifest or payload 
- 
-The net result of Rhizome's crytpo system is that: 
-  * any party may create a new bundle (with a new Bundle ID) 
-  * only the originating party may modify the manifest or payload of an existing bundle (keeping the same Bundle ID) 
-  * thus, once a bundle is created, all new payloads with the same Bundle ID are guaranteed to be from the same original sender 
- 
-==== Payload security ==== 
- 
-  * Rhizome can optionally **encrypt** a payload during injection with a one-way asymmetric cipher using the recipient's [[:content:technologies:sid|Serval ID]] public key 
-  * Rhizome can optionally **encrypt** a payload during injection with a two-way session cipher constructed using the sender's and recipient's [[:content:technologies:sid|Serval ID]] public keys 
-  * Rhizome will **decrypt** an encrypted payload during extraction if the node possesses the recipient's [[:content:technologies:sid|Serval ID]] secret key 
-  * The Payload Hash of an encrypted payload is the SHA512 hash of the encrypted form, not the clear text form 
-  
 ==== User intervention ==== ==== User intervention ====
  
   * Users may improve the local capacity of Rhizome by operating nodes with high storage capacity   * Users may improve the local capacity of Rhizome by operating nodes with high storage capacity
   * Users may improve the reach of Rhizome by periodically moving a node between distant locations   * Users may improve the reach of Rhizome by periodically moving a node between distant locations
- 
-==== Network connections ==== 
- 
-  * Rhizome nodes may communicate using **[[http://en.wikipedia.org/wiki/Network_layer|network layer]]** connections over established networks, eg, [[http://en.wikipedia.org/wiki/Transmission_Control_Protocol|TCP]] or [[http://en.wikipedia.org/wiki/User_Datagram_Protocol|UDP]] or [[http://en.wikipedia.org/wiki/Http|HTTP]] over public Internet or cellular GSM 
-  * Adjacent Rhizome nodes may communicate over **[[http://en.wikipedia.org/wiki/Data_link_layer|data link layer]]** connections, eg, [[:content:technologies:mdp|MDP]] to [[:content:technologies:servalmesh|Serval Mesh]] devices within Fi-Fi range 
  
 ==== Architecture ==== ==== Architecture ====
  
-Rhizome is currently implemented as part of the [[:content:technologies:servaldna|Serval DNA]] daemon.+Rhizome is currently implemented as part of the [[:content:servaldna:|Serval DNA]] daemon.
  
 ==== The name "Rhizome" ==== ==== The name "Rhizome" ====
  
-Once a file is inserted into Rhizome, it can be notoriously difficult to eradicate.  This suggested an analogy to a biological process of vegetative reproduction: a [[http://en.wikipedia.org/wiki/Rhizome|rhizome]] is an underground plant stem with the ability to send out shoots which develop into new plants. If a rhizome is cut into smaller pieces, each piece can grow into a new organism.+Once a file is inserted into Rhizome, it can be notoriously difficult to eradicate.  This is analogous to a biological process of vegetative reproduction: a [[http://en.wikipedia.org/wiki/Rhizome|rhizome]] is an underground plant stem with the ability to send out shoots which develop into new plants. If a rhizome is cut into smaller pieces, each piece can grow into a new organism.  Anybody who has battled the incursion of [[http://en.wikipedia.org/wiki/Elymus_repens|couch grass]] into their garden knows how obstinate a rhizome can be. 
 + 
 +==== Prototypes ====
  
-Curiously, but not by coincidence, the principles of the [[http://en.wikipedia.org/wiki/Rhizome_(philosophy)|rhizomatic]] philosophy of thought also resemble the high-level requirements of the Rhizome file distribution system.+The [[Rhizome Retriever]] was the first prototype of the manifest-payload distribution system.
  
content/tech/rhizome.txt · Last modified: 21/02/2016 22:52 by Andrew Bettison

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution 4.0 International
CC Attribution 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki