User:Scott Parlane/DesignStudy

From CSSEMediaWiki
(Difference between revisions)
Jump to: navigation, search
Line 7: Line 7:
 
* Support extremely slow, high capacity links. (80+GiB with 1 week latency)
 
* Support extremely slow, high capacity links. (80+GiB with 1 week latency)
 
* Support multiple transfer agents.
 
* Support multiple transfer agents.
 
+
=== Constraints ===
 +
* WORM data assumed (allowing for rename/move/etc)
 
== Initial Design ==
 
== Initial Design ==
 
* Make use of [http://www.4underscore.net/] infrastructure (espically networking code).
 
* Make use of [http://www.4underscore.net/] infrastructure (espically networking code).
 
* Use the code generator git://git.scottnz.com/generator.git/ to generate OO code from a design spec.
 
* Use the code generator git://git.scottnz.com/generator.git/ to generate OO code from a design spec.
 
* Split system into 3 parts: metadata store, data servers, and transfer agents.
 
* Split system into 3 parts: metadata store, data servers, and transfer agents.
 +
Warning: Notation used below is similar, but not the same as, the notation used by generator, this is intentional, until more graphical representation can be created.
 +
=== MetaData Store ===
 +
Tracks the entire tree of each data storage server and
 +
* Location (Pointer(Char):Name, ArrayObjectPaired(Server,Directory):Servers,ArrayObject(Transporter):Agents)
 +
* Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name)
 +
* File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)
 +
* Server (Pointer(Char):Name, ArrayObject(Location):Locations)
 +
* Transporter(Pointer(Char):UUID, ArrayObject(File):Carrying)
 +
=== Data Store ===
 +
Provides long term storage for all of the tracked files
 +
* Location (Pointer(Char):Name, Array(Directory):Root, Pointer(Char):Path)
 +
* Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name)
 +
* File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)
 +
=== Transporter ===
 +
Provides a temporary storage area for transporting data between data stores
 +
* Path (Pointer(char):Name, ArrayObject(File):Contents)
 +
* File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)

Revision as of 04:31, 2 August 2010

Contents

Project (Out of Band Synchronisation)

Description

Provide a solution for performing offsite back ups, where directly transferring the data is impossible or incredibly expensive, however regular transport between locations is avaliable and at least 1 operation computer is transferred between sites.

Goals/Requirements

  • Synchronise data using out of band data transfer.
  • Support extremely slow, high capacity links. (80+GiB with 1 week latency)
  • Support multiple transfer agents.

Constraints

  • WORM data assumed (allowing for rename/move/etc)

Initial Design

  • Make use of [1] infrastructure (espically networking code).
  • Use the code generator git://git.scottnz.com/generator.git/ to generate OO code from a design spec.
  • Split system into 3 parts: metadata store, data servers, and transfer agents.

Warning: Notation used below is similar, but not the same as, the notation used by generator, this is intentional, until more graphical representation can be created.

MetaData Store

Tracks the entire tree of each data storage server and

  • Location (Pointer(Char):Name, ArrayObjectPaired(Server,Directory):Servers,ArrayObject(Transporter):Agents)
  • Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name)
  • File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)
  • Server (Pointer(Char):Name, ArrayObject(Location):Locations)
  • Transporter(Pointer(Char):UUID, ArrayObject(File):Carrying)

Data Store

Provides long term storage for all of the tracked files

  • Location (Pointer(Char):Name, Array(Directory):Root, Pointer(Char):Path)
  • Directory (ArrayObject(Directory):SubDirs, ArraryObject(File):Contents, Pointer(Char):Name)
  • File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)

Transporter

Provides a temporary storage area for transporting data between data stores

  • Path (Pointer(char):Name, ArrayObject(File):Contents)
  • File (Pointer(char):Name, Pointer(char):md5, Pointer(char):sha1)
Personal tools