HOWTO: Infocalypse 2.0 hg extension


updated: 20110918
Note: Contains Freenet only links

Table of Contents


Introduction

The Infocalypse 2.0 hg extension is an extension for Mercurial that allows you to create, publish and maintain incrementally updateable repositories in Freenet.

It works better than the other DVCS currently available for Freenet.

Most of the information you will find in this document can also be found in the extension's online help. i.e.:

hg help infocalypse

[TOC]


Requirements

The extension has the following dependencies:

[TOC]


Installation

You checked the requirements and understand the risks right?

Here are step-by-step instructions on how to install the extension.

[TOC]


Updating

This extension is under active development. You should periodically update to get the latest bug fixes and new features.

Once you've installed the extension and pulled it for the first time, you can get updates by cd'ing into the initial INSTALL_DIR and typing:

hg fn-fmsread -v
hg fn-pull --aggressive
hg update
If you're not running FMS you can skip the fn-fmsread step. You may have trouble getting the top key. Just keep retrying.

If you're having trouble updating and you know the index has increased, use the full URI with the new index as above.

[TOC]


Background

Here's background information that's useful when using the extension. See the Infocalypse 2.0 hg extension page on my freesite for a more detailed description of how the extension works.

Repositories are collections of hg bundle files

An Infocalypse repository is just a collection of hg bundle files which have been inserted into Freenet as CHKs and some metadata describing how to pull the bundles to reconstruct the repository that they represent. When you 'push' to an infocalypse repository a new bundle CHK is inserted with the changes since the last update. When you 'pull', only the CHKs for bundles for changesets not already in the local repository need to be fetched.

Repository USKs

The latest version of the repository's metadata is stored on a Freenet Updateable Subspace Key (USK) as a small binary file.

You'll notice that repository USKs end with a number without a trailing '/'. This is an important distinction. A repository USK is not a freesite. If you try to view one with fproxy you'll just get a 'Potentially Dangerous Content' warning. This is harmless, and ugly but unavoidable at the current time because of limitation in fproxy/FCP.

Repository top key redundancy

Repository USKs that end in *.R1/<number> are inserted redundantly, with a second USK insert done on *.R0/<number>. Top key redundancy makes it easier for other people to fetch your repository.

Inserting to a redundant repository USK makes the inserter more vulnerable to correlation attacks. Don't use '.R1' USKs if you're worried about this.

Repository Hashes

Repository USKs can be long and cumbersome. A repository hash is the first 12 bytes of the SHA1 hash of the zero index version of a repository USK. e.g.:
SHA1( USK@kRM~jJVREwnN2qnA8R0Vt8HmpfRzBZ0j4rHC2cQ-0hw,2xcoQVdQLyqfTpF2DpkdUIbHFCeL4W~2X1phUYymnhM,AQACAAE/wiki_hacking.R1/0 )
  == 'c856b1653f0b'
You can get the repository hash for a repository USK using:
hg fn-info
from a directory the repository USK has been fn-pull'd into.

You can get the hashes of repositories that other people have announced via fms with:

hg fn-fmsread --listall

Repository hashes are used in the fms update trust map.

The default private key

When you run fn-setup, it creates a default SSK private key, which it stores in the default_private_key parameter in your .infocalypse/infocalypse.ini file.

You can edit the config file to substitute any valid SSK private key you want.

If you specify an Insert URI without the key part for an infocalypse command the default private key is filled in for you. i.e

hg fn-create --uri USK@/test.R1/0
Inserts the local hg repository into a new USK in Freenet, using the private key in your config file.

USK <--> Directory mappings

The extension's commands 'remember' the insert and request repository USKs they were last run with when run again from the same directory.

This makes it unnecessary to retype cumbersome repository USK values once a repository has been successfully pulled or pushed from a directory.

Aggressive topkey searching

fn-pull and fn-push have an --aggressive command line argument which causes them to search harder for the latest request URI.

This can be slow, especially if the USK index is much lower than the latest index in Freenet.

You will need to use it if you're not using FMS update notifications.

[TOC]


Basic Usage

Here are examples of basic commands.

Generating a new private key

You can generate an new private key with:
hg fn-genkey
This has no effect on the stored default private key.

Make sure to change the 'SSK' in the InsertURI to 'USK' when supplying the insert URI on the command line.

Creating a new repository

hg fn-create --uri USK@/test.R1/0
Inserts the local hg repository into a new USK in Freenet, using the private key in your config file. You can use a full insert URI value if you want.

If you see an "update -- Bundle too big to salt!" warning message when you run this command you should consider running fn-reinsert --level 4.

Pushing to a repository

hg fn-push --uri USK@/test.R1/0
Pushes incremental changes from the local directory into an existing Infocalypse repository.

The <keypart>/test.R1/0 repository must already exist in Freenet. In the example above the default private key is used. You could have specified a full Insert URI. The URI must end in a number but the value doesn't matter because fn-push searches for the latest unused index.

You can ommit the --uri argument when you run from the same directory the fn-create (or a previous fn-push) was run from.

Pulling from a repository

hg fn-pull --uri <request uri>
pulls from an Infocalypse repository in Freenet into the local repository. Here's an example with a fully specified uri.

You can ommit the --uri argument when you run from the same directory a previous fn-pull was successfully run from.

For maximum reliability use the --aggressive argument.

[TOC]


Using FMS to send and receive update notifications

The extension can send and receive repository update notifications via FMS. It is highly recommended that you setup this feature.

The update trust map

There's a trust map in the .infocalypse/infocalypse.ini config file which determines which fms ids can update the index values for which repositories. It is purely local and completely separate from the trust values which appear in the FMS web of trust.

The format is:
<number> = <fms_id>|<usk_hash0>|<usk_hash1>| ... |<usk_hashn>

The number value must be unique, but is ignored.

The fms_id values are the full FMS ids that you are trusting to update the repositories with the listed hashes.

The usk_hash* values are repository hashes.

Here's an example trust map config entry:

   # Example .infocalypse snippet
   [fmsread_trust_map]
   1 = test0@adnT6a9yUSEWe5p8J-O1i8rJCDPqccY~dVvAmtMuC9Q|55833b3e6419
   0 = djk@isFiaD04zgAgnrEC5XJt1i4IE7AkNPqhBG5bONi6Yks|be68e8feccdd|5582404a9124
   2 = test1@SH1BCHw-47oD9~B56SkijxfE35M9XUvqXLX1aYyZNyA|fab7c8bd2fc3

You must update the trust map to enable index updating for repos other than the one this code lives in (c856b1653f0b). You can edit the config file directly if you want.

However, the easiest way to update the trust map is by using the --trust and --untrust options on fn-fmsread.

For example to trust falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITw to notify you about changes to the repository with repo hash 2220b02cf7ee, type:

hg fn-fmsread --trust --hash 2220b02cf7ee --fmsid falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITw

And to stop trusting that FMS id for updates to 2220b02cf7ee, you would type:

hg fn-fmsread --untrust --hash 2220b02cf7ee --fmsid falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITw

To show the trust map type:

hg fn-fmsread --showtrust

Reading other people's notifications

hg fn-fmsread -v
Will read update notifications for all the repos in the trust map and locally cache the new latest index values. If you run with -v it prints a message when updates are available which weren't used because the sender(s) weren't in the trust map.
hg fn-fmsread --list
Displays announced repositories from fms ids that appear in the trust map.
hg fn-fmsread --listall
Displays all announced repositories including ones from unknown fms ids.

Pulling an announced repository

You can use the --hash option with fn-pull to pull any repository you see in the fn-read --list or fn-read --listall lists.

For example to pull the latest version of the infocalypse extension code, cd to an empty directory and type:

hg init
hg fn-pull --hash c856b1653f0b --aggressive

Posting your own notifications

hg fn-fmsnotify -v
Posts an update notification for the current repository to fms.

You MUST set the fms_id value in the config file to your fms id for this to work.

Use --dryrun to double check before sending the actual fms message.

Use --announce at least once if you want your USK to show up in the fmsread --listall list.

By default notifications are written to and read from the infocalypse.notify fms group.

The read and write groups can be changed by editing the following variables in the config file:

fmsnotify_group = <group>
fmsread_groups = <group0>[|<group1>|...]

fms can have pretty high latency. Be patient. It may take hours (sometimes a day!) for your notification to appear. Don't send lots of redundant notifications.

[TOC]


Reinserting and 'sponsoring' repositories

hg fn-reinsert
will re-insert the bundles for the repository that was last pulled into the directory.

The exact behavior is determined by the level argument.

level:

Levels 1 and 4 require that you have the private key for the repository. For other levels, the top key insert is skipped if you don't have the private key.

DO NOT use fn-reinsert if you're concerned about correlation attacks. The risk is on the order of re-inserting a freesite, but may be worse if you use redundant (i.e. USK@<line noise>/name.R1/0) top keys.

[TOC]


Forking a repository onto a new USK

hg fn-copy --inserturi USK@/name_for_my_copy.R1/0
copies the Infocalypse repository which was fn-pull'd into the local directory onto a new repository USK under your default private key. You can use a full insert URI if you want.

This only requires copying the top key data (a maximum of 2 SSK inserts).

[TOC]


Sharing private keys

It is possible for multiple people to collaborate anonymously over Freenet by sharing the private key to a single Infocalypse repository.

The FreeFAQ is an example of this technique.

Here are some things to keep in mind when sharing private keys.

[TOC]


Inserting a freesite

hg fn-putsite --index <n>
inserts a freesite based on the configuration in the freesite.cfg file in the root of the repository. Use:
hg fn-putsite --createconfig
to create a basic freesite.cfg file that you can modify. Look at the comments in it for an explanation of the supported parameters.

The default freesite.cfg file inserts using the same private key as the repo and a site name of 'default'. Editing the name is highly recommended.

You can use --key CHK@ to insert a test version of the site to a CHK key before writing to the USK.

Limitations:

[TOC]


Risks

I don't believe that using this extension is significantly more dangerous that using any other piece of Freenet client code, but here is a list of the risks which come to mind:

[TOC]


Advocacy

Here are some reasons why I think the Infocalypse 2.0 hg extension is better than pyFreenetHg and egit-freenet:

[TOC]


Source Code

The authoritative repository for the extension's code is hosted in Freenet:
hg init
hg fn-fmsread -v
hg fn-pull --aggressive --debug --uri USK@kRM~jJVREwnN2qnA8R0Vt8HmpfRzBZ0j4rHC2cQ-0hw,2xcoQVdQLyqfTpF2DpkdUIbHFCeL4W~2X1phUYymnhM,AQACAAE/wiki_hacking.R1/20
hg update

NOTE:
This repository has some other unmaintained and abandoned stuff in it.
e.g. the pre jfniki python server based wiki code, python incremental archive stuff.

It is also mirrored on bitbucket.org:

hg clone http://bitbucket.org/dkarbott/wiki_hacking/

[TOC]

Fixes and version information

[TOC]

Freenet-only links

This document is meant to inserted into Freenet.

It contains links (starting with 'CHK@' and 'USK@') to Freenet keys that will only work from within fproxy [HTTP link!].

You can find reasonably up to date version of this document on my freesite:

USK@-bk9znYylSCOEDuSWAvo5m72nUeMxKkDmH3nIqAeI-0,qfu5H3FZsZ-5rfNBY-jQHS5Ke7AT2PtJWd13IrPZjcg,AQACAAE/feral_codewright/17/infocalypse_howto.html

[TOC]

Contact

FMS:
djk@isFiaD04zgAgnrEC5XJt1i4IE7AkNPqhBG5bONi6Yks

I lurk on the freenet and fms boards.

If you really need to you can email me at d kar bott at com cast dot net but I prefer FMS.

freesite:
USK@-bk9znYylSCOEDuSWAvo5m72nUeMxKkDmH3nIqAeI-0,qfu5H3FZsZ-5rfNBY-jQHS5Ke7AT2PtJWd13IrPZjcg,AQACAAE/feral_codewright/17/

[TOC]