On this page:
6.1 Database protocol
6.1.1 Implementations of the database protocol
6.1.1.1 Filesystem database
6.1.1.2 Caching
6.1.2 Vague object storage
6.1.2.1 Presentable set storage
6.1.2.2 Side effect storage
6.1.3 Other interesting set
6.1.4 Dependency graph

6 Netfarm server

The Netfarm server is a decentralise2 system, which retrieves a partition of all the objects available to it, runs scripts for them, and makes the objects and computed values available to its connections.

In the introduction to decentralise2, we argued that the terms “client” and “server” do not have enough meaning to be useful when discussing Netfarm software; but we ended up calling this module a “server”.

netfarm-system <: standard-systemClass

The class of a Netfarm server.

6.1 Database protocol

6.1.1 Implementations of the database protocol

A diagram of Netfarm server classes, including system and database mixin classes. Shaded classes are instantiable, and not shaded classes are not instantiable. Dashed classes are decentralise2 classes included for completion.

memory-system <: netfarm-systemClass

A system that stores everything in memory. Compare this to decentralise2’s memory-database-mixin.

simple-memory-system <: netfarm-systemClass

A system that stores everything in memory, using the simplest feasible implementations of the database protocol.

6.1.1.1 Filesystem database

filesystem-mixin Class

A mixin that stores objects in a filesystem.

Objects are stored in a “trie” constructed of directories, to avoid slowdowns that occur on some file systems, should one directory contain many files. For each object, three files are stored:
  • A .object file that contains the object, and computed values appended to the end of the object,

  • A .affected-by file that contains a list of object names that this object has been affected by, and

  • A .affects file that contains a list of object names that this object affects.

Some files are also stored in the storage directory:
  • A dependency-graph file that stores the dependency graph as an association list,

  • A interesting-set file that stores the other interesting set,

  • A unpresentable-set file that stores the inverse of the presentable set (as it would be smaller than the presentable set),

  • TODO: A fulfilled-dependencies file that stores object names that have had all dependencies fulfilled, but have not been further processed, and

  • TODO: A script-machines file that stores information about the script machines that need to be run.

:directory Initarg

The directory that objects should be stored in.

6.1.1.2 Caching

caching-mixin Class

A mixin that caches some objects in memory, but defers to another database implementation when it does not have an object stored.

:cache-size Initarg

The maximum rendered size of objects that can be cached at once, in bytes. This defaults to 100 megabytes (10^8 bytes).

:fallback-vague-object-size Initarg

The size that vague objects should be assumed to take when rendered, should the real database not provide a size.

6.1.2 Vague object storage

put-vague-object system vague-objectGeneric Function

get-vague-object system nameGeneric Function

map-vague-objects function systemGeneric Function

vague-object-stored-p system nameGeneric Function

count-vague-objects systemGeneric Function

6.1.2.1 Presentable set storage

presentable-name-p system nameAccessor

count-presentable-objects systemGeneric Function

6.1.2.2 Side effect storage

apply-side-effect system name cause side-effect-type &keyGeneric Function

map-computed-values-caused-by function cause-name systemGeneric Function

objects-affected-by system nameGeneric Function
Return a list of the object hashes affected by a given object.

objects-affecting system nameGeneric Function
Return a list of the object hashes which affect a given object.

objects-affecting-hash system name &key testGeneric Function
Compute a hash value for the objects-affecting list, by \prod_{a \in \text{affecting}} a \pmod{2^{256}}

If a test function is provided, only object names which satisfy the function will be hashed.

When a test is not provided, hashing can be implemented incrementally, but the default method computes it from the entire objects-affecting list. (Recall multiplication and modular multiplication are commutative - a database could store the last hash, and then compute h := h \times a \pmod{2^{256}} when adding a.)

(We were initially going to reduce logxor over the set, but bitwise exclusive-or, among all additive groups, can be attacked by  (Wagner 2002). It is unclear what such an attack could achieve, but we are going to avoid it by using a multiplicative group instead.)

6.1.3 Other interesting set

add-other-interesting-block system nameGeneric Function

remove-other-interesting-block system nameGeneric Function

other-interesting-block-p system nameGeneric Function

6.1.4 Dependency graph

A Netfarm server maintains a dependency graph, consisting of a set of dependency edges, which each store the name of a dependent (a string), the name of a dependency (another string), and a type (a keyword).

For the avoidance of doubt (because we frequently got confused by the names): when a relevant edge exists, storing a dependent object is prevented until a dependency object is retrieved.

All functions that take edge types except for add-dependency-edge also accept the type :all to retrieve all edges, regardless of type.

add-dependency-edge system type dependent dependencyGeneric Function

remove-dependents system dependencyGeneric Function

map-dependents function system type dependencyGeneric Function

map-dependencies function system type dependenciesGeneric Function

map-dependency-edges function systemGeneric Function

no-dependencies-p system type nameGeneric Function

dependency-list-mixin Class

A mixin that implements a dependency graph by storing a list of its edges. This mixin may be slower than dependency-table-mixin, but it is easier to read about, as it does not do any tricky indexing.

dependency-table-mixin Class

A mixin that implements a dependency graph by storing the edges in hash tables. This may look up dependencies and dependents more efficiently than dependency-table-mixin, especially with many dependencies to manage.