.HTML "Shifs - a simple way to transform and build dynamic file servers
.TL
Shifs
A simple way to transform and build dynamic file servers
.AU
Uriel M.
uriel @ cat-v.org
.AB
.PP
.I Shifs
is
.I Shifs
is a way to transform and generate simple dynamic file systems that can be used directly
from the command line and can take advantage of the standard tools used to manipulate text
streams.
.I Shifs
descrived here is only a prototype but it is already functional enough to be useful and replace some
specialized components.
.AE
\" .2C
.NH
Introduction
.PP
.I Shifs
uses a very simple language to deffine a set of
.I paterns
that trigger
.I actions .
In that sense it is similar to
.I awk
but instead of handling text streams, it works with file trees.
.I Shifs
was born out of the desire to have a simple way to transform
and exten the data expossed by file server in a generic and extensible way.
Editing the source of existing file servers is too intrusive and makes it harder
to maintain backwards compatibility for applications that might expect the old
interface. One could avoid that in some cases by adding new synthetic files to
the file server that would export the data in the desired format, but that cluters
the file server with files that might have a very specialized purpose that is not
related to the function of the file server. In other cases access to the file server
might not be available, or one might want to transform a whole tree that might
include files exported by different servers.
.PP
Other goal was to make it convenient to leverage the power of the standard application toolset of the
operating system and in that way bring the
.I software-tools
philosphy to the world of dynamic file server creation, and do so in a secure and convenient way.
.PP
The more inmediate task that prompted this interest was the need to overhaul the
HTML generation code in
.I wikifs
which consits of almost a thousand lines of C code used to first parse the internal
text format of wiki pages and then generate the HTML output. Considering that the
same task could be acomplished by a dozen lines of awk and other standard tools,
it appeared worthwhile to take advantage of that approach without discarding the
convenience of a file system interface.
.NH
suckfs
.PP
A first attempt was made to implement in C a whole language called
.I suckfs
(because it
.i sucks files) in the spirit of awk
where rules would match path names and actions would dynamically generate
or transform the contents of the matched files.
But soon it became clear that the language would need to be too complex to even
perform the most simple tasks, and it failed to acomplish the main goal that was to take
advantage of existing tools and languages without adding extra complexity.
Also it was apreciated that it would be convenient if the facility could be available to
as many platforms as possible. Inferno provided this and was a good oportunity to experiment
with Limbo.
In hindsight this was a very good choice as it simplified some problems considerably. In particular
using the
.I sh
module made everything much easier.
.NH
The shifs language
.PP
The language was simplified to the bare basics, providing a fundation that could be extended
in the future as more complex interactions become feasible.
The language has a syntax somewhat similar to
.I mk
and consists of
.I rules ,
each composed of a
.I pattern
and a set of
.I actions .
.NH 2
.PP
A pattern starts at the begining of line and consists of an operation specifier (
.I r
for read,
.I w
for write, etc.), followed by the regexp match operator
.I ~
(other operators might be added in the future), the rest of the pattern up to the end of line is the argument
to the regexp operator which is a regular expresion that is matched against file paths.
.PP
\" .1C
.KF
.P1
r ~[0-9]+/[a-z]+$
.P2
.ce
.I "Figure 1. Sample shifs pattern"
.KE
.PP
.NH 2
Actions
.PP
To complete a rule a set of
.I action
lines follow the pattern line, action lines should
begin with a
.I tab
character and use the Inferno sh syntax.
\" .1C
.KF
.P1
r ~^today$
date|sed 's/ .*//'
.P2
.ce
.I "Figure 2. A complete shifs rule"
.KE
.PP
The environment in which actions are run includes some convenient variables set by shifs before the action is run, of special interest is
.I $FPATH
which contains the full path that matched this rule.
.NH
Execution
.PP
It is easy to run
.I shifs
from a terminal window, the arguments that the following format:
.B shifs
[
.B -po
]
[
.B -r
.I /source/tree
]
[
.B -m
.I /mount/point
]
[
.I rules
]
.PP
The
.I -p
option allows to 'pass-thru' files that don't match any rule so they are retrived unchanged from
the underlying file server. The
.I -o
makes the exported file system read only. The path passed to
.I -r
is used as starting point for the exported file tree, and actions use it as their working directory, if the
.I -p
options was used all the files under that tree that don't match any rule will also be accessible without any modification. If
.I rules
is not specified the rules will be read from stdin.
.NH
Walking, opening and reading.
.PP
.I Shifs
keeps track of walk operations, when a
.I fid
is opened the full path is matched against each rule, if a rule matches
a new process is spawn that sets its stdout to a pipe from where the subsequent read operations
will feed and then executes the contents of the associated action.
.NH
Uses
.PP
Some of the tasks where shifs can be helpful are: performing simple transformations
to the contents of arbitrary file trees, for example when importing a file system
from a windows system one can easily tranform all files to use the Plan 9/Unix new line
format by simply using
.I tr(1) .
Other useful task is to format data from its canonical format into formats convenient for export,
a good example of this is adding html formating before exporting a file tree over httpd.
Also it can be convenient to create synthetic files that collect and sumarice the contents of whole file
trees.
Finally one can even create completely synthetic file trees, for example here is a file server that returns the square of any given number:
\" .1C
.KF
.P1
% shifs '~^[0-9]+$~ {load expr; echo ${expr $FNAME $FNAME x}}'
% cat /mnt/shifs/4
16
%
.P2
.ce
.I "Figure 1. Sample shifs pattern"
.KE
.PP
.NH
Conclusions and future directions
.PP
The current version of
.I shifs
is still very much a prototype with many rough edges, but so far it has demonstrated to be a good tool to experiment with 'one off' file servers. And that is probably its greatest power, just like shells allow you to easily write small throw-away programs that when found useful can be codified either as shell scripts or in some cases rewriten in a "
.I real"
language. It also manages to at least partially suceed in its other goal of making the more traditional Unix/Plan 9 set of text manipulation tools more easily accessible when writing simple file servers.
.I Shifs
is still somewhat fragile, but still can be useful in experimenting with changes to the interface of existing file servers and in working as an adaptor between file system interfaces.
The main areas that still need work is handling of write operations and transformation of directory listings. The ruleset language is probably somewhat simplistic and naive and might need some redesign.
Perhaps for better integration with a Plan 9 environment a rewrite in C could be considered, but maybe it would be best to instead try to take advantage of the "native" Dis.
.NH
Aknowledgements
.PP
Awk is clearly the main source of inspiration, but the more general tool philosophy underlines the goals of the project. Also one must aknowledge how both Limbo and the sh module in Inferno made implementation almost trivial.
.PP
I am not aware at the moment of any similar projects, but it is very likely that someone has thought of a similar thng before, my apologies if I have not noticed it.
.NH
References
.LP
[Inferno] Inferno Manual page for Sh(1)
.LP
[Aho79] A. V. Aho and B. W. Kernighan and P. J. Weinberger
``AWK - A Pattern Scanning and Processing Language'',
.LP
[Kernighan76] B. W. Kernighan and P. J. Plauger
``Software Tools''
|
