The Serval Project Wiki

User Tools

Site Tools

You are here: The Serval Project Wiki » content » Technology Roadmap » Rhizome
Trace:
content:tech:rhizome

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
content:tech:rhizome [07/04/2013 19:26]
Andrew Bettison [Purpose and features] neutral to payload content, encryption for privacy, decentralised 		content:tech:rhizome [21/02/2016 22:52] (current)
Andrew Bettison Re-word [Technical documentation] section; remove link to GitHub (maintenance burden)
Line 1: Line 1:
 ====== Rhizome ====== ====== Rhizome ======
  
-**Rhizome** is the resilient file distribution system developed by [[:|The Serval Project]].  It forms the basis of all non-[[http://en.wikipedia.org/wiki/Connection_oriented|connection-oriented]] services provided by [[:content:technologies:servalmesh|Serval Mesh]], such as [[:content:technologies:meshms|text messaging]], file sharing, voice mail, and automatic software upgrades. +**Rhizome** is a resilient content distribution system developed by [[:|The Serval Project]].  It forms the basis of all non-[[http://en.wikipedia.org/wiki/Connection_oriented|connection-oriented]] services provided by [[:content:servalmesh:|Serval Mesh]], such as [[meshms|text messaging]], file sharing, voice mail, and automatic software upgrades. 
  
 +==== Technical documentation====
 +
 +**[[Rhizome API]]** contains links to the authoritative [[:content:dev:techdoc|technical documentation]] of the Rhizome HTTP REST API, which resides in the source code of the [[:content:servaldna:development|Serval DNA]] software, and should be used in preference to the older documents on this wiki.
 +
 +Other relevant information:
 +
 +  * [[Security Framework]] describes the cryptographic features of Rhizome
 +  * [[:content:activity:naf4]] describes the original development of the Rhizome HTTP REST API (beware: some technical details are out of date; refer to [[Rhizome API]] for current documentation)
  
 ==== Purpose and features ==== ==== Purpose and features ====
  
   * Rhizome is a **[[http://en.wikipedia.org/wiki/Store_and_forward|store and forward]]** service for transporting units of data called **payloads** ("files") between **applications** running on nodes in a digital [[http://en.wikipedia.org/wiki/Telecommunications_network|telecommunications network]]   * Rhizome is a **[[http://en.wikipedia.org/wiki/Store_and_forward|store and forward]]** service for transporting units of data called **payloads** ("files") between **applications** running on nodes in a digital [[http://en.wikipedia.org/wiki/Telecommunications_network|telecommunications network]]
-  * Rhizome is designed for [[http://en.wikipedia.org/wiki/Wireless_mesh_network|wireless mesh networks]] such as [[:content:technologies:mdp|MDP]] in the [[:content:technologies:servalmesh|Serval Mesh]], but can also be used within a fixed network infrastructure+  * Rhizome is designed for [[http://en.wikipedia.org/wiki/Wireless_mesh_network|wireless mesh networks]] such as [[MDP]] in the [[:content:servalmesh:|Serval Mesh]], but can also be used within a fixed network infrastructure
   * Rhizome guarantees that payloads will remain intact wherever they are transported   * Rhizome guarantees that payloads will remain intact wherever they are transported
   * Rhizome makes no guarantees about delivery time or eventual arrival   * Rhizome makes no guarantees about delivery time or eventual arrival
Line 20: Line 28:
   * Users do not use Rhizome directly, but //indirectly via [[http://en.wikipedia.org/wiki/Application_software|applications]]// which use Rhizome as their data transport   * Users do not use Rhizome directly, but //indirectly via [[http://en.wikipedia.org/wiki/Application_software|applications]]// which use Rhizome as their data transport
   * Current Rhizome applications:   * Current Rhizome applications:
-    * [[:content:technologies:meshms|MeshMS]], the simple text messaging service of the [[:content:technologies:servalmesh|Serval Mesh]] app for Android+    * [[MeshMS]], the simple text messaging service of the [[:content:servalmesh:|Serval Mesh]] app for Android
     * [[:content:servalmaps:|Serval Maps]], an Android app produced by the [[:|Serval Project]], uses Rhizome to share geo-tagged data like photos and documents     * [[:content:servalmaps:|Serval Maps]], an Android app produced by the [[:|Serval Project]], uses Rhizome to share geo-tagged data like photos and documents
-    * the [[:content:technologies:servalmesh|Serval Mesh]] app for Android uses Rhizome to distribute software upgrades +    * the [[:content:servalmesh:|Serval Mesh]] app for Android uses Rhizome to distribute software upgrades 
-    * the file sharing service of the [[:content:technologies:servalmesh|Serval Mesh]] app for Android is a simple UI for sharing, browsing and saving/viewing non-MeshMS content in Rhizome +    * the file sharing service of the [[:content:servalmesh:|Serval Mesh]] app for Android is a simple UI for sharing, browsing and saving/viewing non-MeshMS content in Rhizome 
-    * [[:content:technologies:servalsam|Serval SAM]] is a crowd survey app for Android that allows the user to fill in an [[http://opendatakit.org/|Open Data Kit]] form and uses Rhizome to transport completed forms to a central collection server +    * [[Serval SAM]] is a crowd survey app for Android that allows the user to fill in an [[http://opendatakit.org/|Open Data Kit]] form and uses Rhizome to transport completed forms to a central collection server 
-    * [[:content:technologies:sensorlogger|Sensor Logger]], an Android app produced by the [[:|Serval Project]], uses Rhizome to send continuous accelerometer logs to the [[:content:seniordevs:servalhq|Serval HQ]] central collection server +    * [[Sensor Logger]], an Android app produced by [[:content:about]], uses Rhizome to send continuous accelerometer logs to the [[:content:seniordevs:servalhq|Serval HQ]] central collection server 
   * Future applications may include:   * Future applications may include:
     * voice mail     * voice mail
Line 41: Line 49:
 ==== Rhizome's data model ==== ==== Rhizome's data model ====
  
-  * Every Rhizome payload is accompanied by some structured meta data called a **manifest**: +Every file distributed by Rhizome ("payload") is accompanied by some structured meta-data called a **[[Rhizome manifest|manifest]]**. 
-    * some meta data is supplied by the originating application, eg, ''name'', ''mimetype'' + 
-    * most meta data is created by Rhizome to identify and prioritise the payload, eg, ''id'', ''filesize'', ''filehash'', ''version'', etc. +A payload plus a [[Rhizome manifest|manifest]] together form a **[[Rhizome bundle]]**. 
-  * A manifest together with its payload is called a **bundle** + 
-  * A manifest may be present on many nodes at a time +==== Rhizome's data transport ==== 
-  * A payload may be present on many nodes at a time+ 
 +  * A **Rhizome node** is any network node (device) that is running the Rhizome software 
 +  * A Rhizome node exchanges manifests and payloads directly with adjacent nodes on the network 
 +  * Rhizome bundles are transferred between nodes automatically and autonomously, ie, without user intervention, using any available network connection 
 +  * A manifest may be stored on many nodes at a time 
 +  * A payload may be stored on many nodes at a time
   * A manifest may persist for any length of time on any node   * A manifest may persist for any length of time on any node
   * A payload may persist for any length of time on any node   * A payload may persist for any length of time on any node
-  * Rhizome does not modify any manifest once it is created +  * Each Rhizome node advertises its recently acquired bundles to its neighbours
-  * Rhizome does not modify any payload +
-  * A bundle may indicate the [[:content:technologies:sid|Serval Identity]] of its intended recipient, otherwise it is for general consumption (broadcast) +
-  * A payload may indicate the [[:content:technologies:sid|Serval Identity]] of its sender, otherwise it is of unknown origin (anonymous) +
-  * Rhizome enforces a set of logical conditions on every bundle it receives from any source: +
-    * certain fields must always be present +
-    * each field must have a certain length and/or format+
  
-== Rhizome manifest ==+== Rhizome store ==
  
-The Rhizome manifest is a set of key-value pairs called **"fields"**.+  * A Rhizome node contains a **Rhizome store** 
 +  * A Rhizome store is a database containing payloads indexed by PH (Payload Hash) and manifests indexed by BID (Bundle ID) 
 +  * A Rhizome store may reside in volatile (RAM) or non-volatile (Flash, disk) local storage 
 +  * The original implementation of the Rhizome store used [[http://www.sqlite.org/|SQLite]] for the database
  
-The Rhizome manifest is designed for forward and backward compatibility between different versions of the Rhizome software: +== Rhizome Rank ==
-  * Every Rhizome node uses the manifest fields it recognises and silently ignores those it does not +
-  * Every Rhizome node preserves all manifest fields intact when it stores the manifest and when it transmits the manifest to other nodes+
  
-==== Rhizome nodes ====+  * Every Rhizome node maintains its own ordered list of all the manifests in its store, called the **Rank** 
 +  * Every Rhizome node attempts to fetch missing payloads from neighbouring nodes, in order of rank 
 +  * Every Rhizome node recovers local storage space by evicting its lowest ranked payloads 
 +  * A bundle's position in the rank depends on meta data in the manifest and local node state, which may include: 
 +    * own Serval Identities 
 +    * Serval Identities of recently reachable nodes 
 +    * current time 
 +    * current or recent geographic location
  
-  * A **Rhizome node** is any network node (device) that is running the Rhizome software +== Network connections == 
-  * Payloads and manifests are copied directly from node to node using network transfers + 
-  * Payloads and manifests are stored on nodes using volatile (RAM) or non-volatile (Flash, disk) local storage +  * Rhizome nodes may transfer bundles using **[[http://en.wikipedia.org/wiki/Network_layer|network layer]]** connections over established networks, eg, [[http://en.wikipedia.org/wiki/Transmission_Control_Protocol|TCP]] or [[http://en.wikipedia.org/wiki/User_Datagram_Protocol|UDP]] or [[http://en.wikipedia.org/wiki/Http|HTTP]] over public Internet or cellular GSM 
-  * Each Rhizome node actively transmits its recently acquired manifests to its neighbours +  * Adjacent Rhizome nodes may transfer bundles over **[[http://en.wikipedia.org/wiki/Data_link_layer|data link layer]]** connections, eg, [[MDP]] to [[:content:servalmesh:|Serval Mesh]] devices within Fi-Fi range
-  * Each Rhizome node records all received manifests in a sorted list called the **rank** +
-  * Each Rhizome node attempts to acquire the payloads of its highest ranked manifests from neighbouring nodes +
-  * Each Rhizome node recovers local storage space by evicting its lowest ranked payloads and manifests+
  
 ==== Neutrality ==== ==== Neutrality ====
Line 83: Line 95:
   * Rhizome does not rank (prioritise) based on the content of payloads   * Rhizome does not rank (prioritise) based on the content of payloads
  
-==== Bundle integrity ==== 
  
-  * Rhizome creates a random 256-bit Elliptic Curve private key for every bundle, called the **Bundle Secret** or **BS** 
-  * The 256-bit public key derived from the Bundle Secret is called the **Bundle ID** or **BID** 
-  * Rhizome does not reveal the Bundle Secret to any other application or node 
-  * Rhizome computes the 512-bit SHA512 hash of the payload, called the **Payload Hash** or **PH** (also **File Hash** or **FH**) 
-  * At injection, Rhizome inserts the Payload Hash into the bundle's manifest 
-  * Rhizome signs every complete manifest using the Bundle Secret 
-  * Rhizome nodes will only store and transmit manifests with valid signatures 
-  * Only an application possessing the original Bundle Secret can alter the manifest or payload 
- 
-The net result of Rhizome's crytpo system is that: 
-  * any party may create a new bundle (with a new Bundle ID) 
-  * only the originating party may modify the manifest or payload of an existing bundle (keeping the same Bundle ID) 
-  * thus, once a bundle is created, all new payloads with the same Bundle ID are guaranteed to be from the same original sender 
- 
-==== Payload security ==== 
- 
-  * Rhizome can optionally **encrypt** a payload during injection with a one-way asymmetric cipher using the recipient's [[:content:technologies:sid|Serval ID]] public key 
-  * Rhizome can optionally **encrypt** a payload during injection with a two-way session cipher constructed using the sender's and recipient's [[:content:technologies:sid|Serval ID]] public keys 
-  * Rhizome will **decrypt** an encrypted payload during extraction if the node possesses the recipient's [[:content:technologies:sid|Serval ID]] secret key 
-  * The Payload Hash of an encrypted payload is the SHA512 hash of the encrypted form, not the clear text form 
-  
 ==== User intervention ==== ==== User intervention ====
  
   * Users may improve the local capacity of Rhizome by operating nodes with high storage capacity   * Users may improve the local capacity of Rhizome by operating nodes with high storage capacity
   * Users may improve the reach of Rhizome by periodically moving a node between distant locations   * Users may improve the reach of Rhizome by periodically moving a node between distant locations
- 
-==== Network connections ==== 
- 
-  * Rhizome nodes may communicate using **[[http://en.wikipedia.org/wiki/Network_layer|network layer]]** connections over established networks, eg, [[http://en.wikipedia.org/wiki/Transmission_Control_Protocol|TCP]] or [[http://en.wikipedia.org/wiki/User_Datagram_Protocol|UDP]] or [[http://en.wikipedia.org/wiki/Http|HTTP]] over public Internet or cellular GSM 
-  * Adjacent Rhizome nodes may communicate over **[[http://en.wikipedia.org/wiki/Data_link_layer|data link layer]]** connections, eg, [[:content:technologies:mdp|MDP]] to [[:content:technologies:servalmesh|Serval Mesh]] devices within Fi-Fi range 
  
 ==== Architecture ==== ==== Architecture ====
  
-Rhizome is currently implemented as part of the [[:content:technologies:servaldna|Serval DNA]] daemon.+Rhizome is currently implemented as part of the [[:content:servaldna:|Serval DNA]] daemon.
  
 ==== The name "Rhizome" ==== ==== The name "Rhizome" ====
Line 124: Line 109:
 Once a file is inserted into Rhizome, it can be notoriously difficult to eradicate.  This is analogous to a biological process of vegetative reproduction: a [[http://en.wikipedia.org/wiki/Rhizome|rhizome]] is an underground plant stem with the ability to send out shoots which develop into new plants. If a rhizome is cut into smaller pieces, each piece can grow into a new organism.  Anybody who has battled the incursion of [[http://en.wikipedia.org/wiki/Elymus_repens|couch grass]] into their garden knows how obstinate a rhizome can be. Once a file is inserted into Rhizome, it can be notoriously difficult to eradicate.  This is analogous to a biological process of vegetative reproduction: a [[http://en.wikipedia.org/wiki/Rhizome|rhizome]] is an underground plant stem with the ability to send out shoots which develop into new plants. If a rhizome is cut into smaller pieces, each piece can grow into a new organism.  Anybody who has battled the incursion of [[http://en.wikipedia.org/wiki/Elymus_repens|couch grass]] into their garden knows how obstinate a rhizome can be.
  
-Curiously (but not coincidentally) some of the principles of the [[http://en.wikipedia.org/wiki/Rhizome_(philosophy)|rhizomatic]] philosophy of thought also describe the behaviour of the Rhizome file distribution system.+==== Prototypes ==== 
 + 
 +The [[Rhizome Retriever]] was the first prototype of the manifest-payload distribution system.
  
content/tech/rhizome.1365388000.txt.gz · Last modified: 07/04/2013 19:26 by Andrew Bettison

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution 4.0 International
CC Attribution 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki