Introspector/LanguageTools/SilverScheme/Doc

-- Objects --

Everything in SilverScheme is an object. An object consists of two parts: a code section and an field list. The code section may contain a lambda expression. The field list is basically a hash with unique keys, the values refer to other objects.

-- Scope --

Scope is complicated. SilverScheme, like any modern LISP has lexical scope (block scope). To understand scope in SilverScheme a little knowledge of the syntax is required.

Every definition is basically a (define), lambda list is (name value . forms). The forms are evaluated in the scope of the (newly created) object. For example (define a 10 (define b 20)) will create a new object a with value 10 and with an attribute b with value 20. Thus a.b = 20.

When you say (define a.b 30) you define in the current scope a new binding for symbol b as field of object a. An example demonstrates some things: (define a 10 (define b 20)) (define-ref c a) ;c is an alias for a (define d a) ;d is a copy of a (define a.b 30) a.b = 30 c.b = 30 d.b = 20

Thus (define a.b value) doesn't set a.b, but it sets the binding of symbol b to object a.

Wow, I've finally managed to get scope working decently :-).

For your understanding here are a few simple facts:
 * An object is in a box, a memory location.
 * A field points to a box.
 * The object id is actually the box id.
 * The set! function changes the object in a box.
 * The define function makes a field point to a different box.

-- Pass-by-reference, pass-by-value --

The general rule of SilverScheme is that function arguments are passed by reference. However two basic functions are an exception on this. Arguments to (define) are passed by value, (define-ref) is the normal reference one, thus (define) can be seen as (define-ref), but with arg.copy done for each argument.

The let construct also needs pass-by-value to keep consistent with define (and to avoid easily made hard to find bugs). To indicate reference one must put
 * ref before the clause. For example:

(let ((a 10) ;a is by-value :ref (b 20)) ;b is by-ref     __some_code__)

Granted, assigning by reference to a literal is stupid (literals are constant), but this demonstrates the principle.

Copies in SilverScheme (pass-by-value creates copies) use copy-on-write. I guess you already know how copy-on-write works.

-- Metadata --

SilverScheme is designed around the concept that: a) code should be free, in all it's forms and b) that if you don't do that somebody else will ;-). All information about an object is accessible through it's metadata interface. The metadata interface consists of roughly two parts: documentation contributed by the user and information gathered by the interpreter.

The meta field is also where all attributes are stored. An attribute in SilverScheme is simply an object. The user can thus easily define attributes.

There are thus roughly three types of objects of metadata: the documentation, the reflection info and custom attributes. The documentation is placed into the obj.documentation field in hash form. Both reflection and custom attributes go into obj.meta in normal object form.

SilverScheme supports documentation through the Argentum [1] scheme. Argentum is the POD of SilverScheme. Documentation can be put in doc comments of the form ;...\n and #!! ... !# (multi-line). Doc comments use a syntax similar to YAML.

Attributes can be easily created using (define meta.attrname value).

-- Classes --

SilverScheme has a weird way of implementing classes. The instance of a class is defined as a field of that class named 'prototype'. The constructor then does (create-instance), (create-instance) adds the needed class information to a copy of the prototype.

Note that classes can access private members of their prototypes and vice versa.

To access the class of an object you can use the obj.class field (or the obj.meta.class field).

An example of a simple class: (define-class Foo (Object Bar) ;;Foo inherits from Object                              ;;and promises to implement                               ;;the public interface of Bar  (define-prototype ;;(define prototype) doesn't work! (define baz 10)))

-- Inheritance --

SilverScheme does not support multiple inheritance. Instead it has interface inheritance and mixins (later more about that). Every class after the first is interface inherited. Interface inheritance means that a class only inherits from the parent in name, not in code (aka no code is copied).

Note that it's not possible to inherit from a non-class. This is to keep the class mechanism simple.

-- Mixins --

SilverScheme uses mixins to do what other languages do with multiple inheritance. A mixin is an object that has fields marked with the (mixin) attribute. The object can also be marked with the (full-mixin) attribute, essentially giving all it's fields the mixin attributes.

When an object has the (use-mixin __mixin_name__) attribute it copies the source code of the mixin fields in the mixin and evaluates them in the current scope. Kinda like how Ruby does it. This is used for example to give objects the <, <=, =, etc functions for free if they have the <=> function and say (use-mixin Comparable).

-- Domains --

SilverScheme provides some powerful tools to make programming without bugs easier. The most important one is domains. A domain is a function that takes one argument, and checks if that argument is in the domain. If so nothing happens, if not an Domain-Violation-Error is thrown. Domains have three main uses: checking variables, enforcing preconditions, enforcing postconditions.

To check a variable against a domain one simply uses the (domain) self-applying attribute (most attributes are self-applying btw). For example: (define a 5 ;domain: 0 < a < 5 (domain (\ (value) (and (> value 0) (< value 11)))))

The domain function accepts a few keywords after the lambda expression:
 * read -- Check the domain whenever the variable contents are read.
 * write -- Check the domain whenever the variable contents are written.

The keywords may be used both.

By default only :write is on. The domain-default-operation environment variable is used to set the default to a different value.

To check a preconditions replace the argument name with (argument-name domain): (\ ((a (\ (value) ;domain: a > 0 (> a 0))))   ...)

The domain may be in a variable (useful for often used domains).

Another way to do preconditions is to use the (pre-condition __arg_name__ __domain__) attribute. The advantage of the attribute is that you can use it conditionally (for example in a testing situation, but not in production use)

To check a postcondition use the (post-condition) attribute. For example: (define simple-add (\ (a b) (+ a b)) (post-domain (\ (value) (> value 0))))

The Argentum documentation extractor (agdoc) also extracts domains, so using them is a good way to show what the pre- and postconditions are.

-- owner, parent, super --

In a.b object defined as (define a 10 (define b 20)) a owns object b since b was defined by object a. Object b has full access to the private members of object a. For object b object a is known by as 'owner'. The variable 'owner' always points to the owner of an object.

The parent of an object is the prototype of the superclass.

The super variable refers to parent.__name_of_this_function__.

-- call/cc, call/tf, call/rf --

SilverScheme call/cc (call-with-current-continuation) is pretty much like the standard Scheme call/cc. However it has some more argument passing power. When you do (call/cc (\ (b) (b 10 20))) then the resulting continuation has a field values which contains a list of the values passed to it. When you invoke the continuation with the arguments the arguments are given as the result of the break function, always as a list.

The call/tf (call-with-throw-function) function is like the Guile catch function. It takes one argument, a lambda list with one parameter, the throw function. When the throw function is invoked a non-local exit is done to the call/tf, like with call/cc, however there is no return. The arguments passed to the throw function are returned by (call/tf) as a list in the second element of a list (the first element if #t if no non-local exit occurred and #f otherwise).

The call/rf function is similar. However it doesn't signal success or failure and returns the list directly (not as a sublist).

-- Exceptions --

Exceptions are build upon call/tf and work roughly like this:

(try (begin __code_to_be_tried__) (catch (__exception_class__  __code_to_be_executed_on_that_exception__) (__exception_class__  __code_to_be_executed_on_that_exception__) ...) (finally __code_that_should_always_be_executed))

Pretty simple isn't it? The (throw) function is rigged to go to the nearest exception handler at all times. It's not quite schemisch, but it looks pretty good :-).

-- conditional evaluation of parameters --

Normally parameters are evaluated left to right. SilverScheme has an easy way to create an exception on that however. For example: (\ (`a b `(c (\ (val) (> val 10))) (if (b)  a   c))

Right, stupid example, but it demonstrates my point. If you put a backquote before the argument in the lambda list the argument will not be evaluated until needed.

-- macros --

A macro is a function that returns a list that is evaluated in the scope of the caller. A macro is also an ideal way to get a virus in a SilverScheme system. A macro is made by simply creating a function and then attaching the (macro) attribute to it.

Macros can be dangerous, they can be helpful too however. It depends on the situation what they are. That's why I'm introducing set-macro-safe. The set-macro-safe function takes one argument which is either a number or a symbol and sets the macro-safe level to that argument. The possible arguments are: 0 'allow-all -- Allow normal macro behaviour for all macros. 1 'explicit-deny -- Allow normal macro behaviour for all macro's except when explicitly told not to do that. 2 'allow-base -- Allow normal macro behaviour for all macro's in the Base hierarchy (the baselib). Disallow normal macro behaviour for all other macros unless explicitly told otherwise. 3 'explicit-allow -- The same as 'allow-base, except that the Base macro's aren't automatically allowed anymore. 4 'deny-all -- Deny all macro's.

Some primitive macro's are allowed regardless of what the macro-safe level says.

Macro's that can't be called with normal macro behaviour can still be called, but they only take a single argument (a list) and produce a list that must be explicitly evaluated (using '(eval list)').

The standard macro level is 0. Note that that's not a safe level for any production code.

At the first deny command the macro level automatically gets set to 1, if it's not already at that or a lower level. Thus level 0 can be regarded as level 1, however it's faster since the macro deny checks are off.

-- Dynamic binding and environment variables --

Sometimes it's handy to pass arguments to functions without putting them in the argument list but by defining a variable and having the function read that variable. In the case (begin (define a 10) (foo)) foo can't read a, and it shouldn't. For environment variables like macro-safe it's another situation however. That's why I'm introducing environment variables.

Environment variables are dynamically bound. They can't leave the scope in which they are defined however (like any other variable). To define an environment variable you need a special function (define-environment-variable) aka (defenv). This function simply behaves like a normal define, except that the variable is automatically set read-only (you can override, but normally it's what you want). The variable can then be accessed using env.variable-name, note that 'env' is a special variable.

Almost all the special variables can be accessed using env too (env.super, env.owner, etc), they only have names without env in them because they are used so much that it's reasonable to shorten them. The macro-safe variable is however only accessible with env.macro-safe (it's a read-write variable btw).

A word of warning is due here, don't define environment variables unless you really need them, it's bad style to depend on them for stuff you should pass in a parameter list.

-- compilation --

SilverScheme is like any Scheme compilable. However SilverScheme has multiple compiling modes.

The standard mode is keep-all, which basically creates a resource section in the executable and puts the literal code files in it. The keep-all mode has the advantage that the result (obj.source) will have the same markup and comments as the original. Since the data is put in resource files the actual code can be executed pretty fast. Needless to say this actually increases file size.

A second mode is keep-source, this mode only keeps the actual source code, not comments. It helps keep the file size down.

A third mode is keep-minimum, removing all the comments and sources except those referred to in the program. Good for really small files, bad for readability. Note that since documentation is put into the objects it can't be removed by this mode.

The fourth and most drastic mode is keep-minimum-remove-docs, this mode also kicks out the documentation. Only useful if you really really want a small program.

An implementation may compress resource files with tar and gzip at any compression level. I recommend not or very weakly compressing the actual sources but strongly compressing the documentation.

The extension of a SilverScheme source file is .dgs, the extension of a SilverScheme compiled file is .dgsb (SilverScheme Bytecode).

Note that the SilverScheme bytecode is not IL, Parrot or machinecode though compiling to these is allowed as an optional feature. The SilverScheme bytecode is a special bytecode designed for SilverScheme (probably looking like other LISP bytecodes).

-- Permissions --

The permission system is aimed at providing flexible authorization for certain objects to do certain things. Permissions are given with tokens. Access tokens give an object A the power to get/set/invoke object B even if this is forbidden by the access system (access system: public SUBSET-OF {'read 'write 'invoke}, private SUBSET-OF {'read 'write 'invoke}). Here's an example of access tokens:

(define a 10 (read-only) ;;(public 'read 'invoke) (private 'read 'write 'invoke)  (define giver (\      (a.give-token-to-caller 'a.write)) (a.give-bag-access)))

In this example 'a is first declared read-only (that's the default actually), then a function a.giver is declared and made read-only implicitly. The giver function is given bag-access by 'a. Any object can give bag access during it's definition or at any time through it's code. The giver function now gives write permission for a to any object that calls it. There is no caller object by the way since I consider it a bad idea if code can manipulate things up the stack except in such special cases.

There is a second kind of tokens called custom tokens. Basically a custom token is a token that doesn't have an implicit action associated with it. An example of a custom token system:

(define nice 0 (define set ;accessor (\ (value)     (if (< value 0) (if (caller-has-token? 'system.root-access)         (set! nice value)         (throw (Invalid-Rights-Exception "You don't have root permissions"))        (set! nice value))))

Without the root-access token you can't use it.

To be honest, custom tokens don't feel right yet. I'd rather leave them out of the language for now, letting them mature first, I included them only for completeness sake.

-- Static --

The fields of an object can be divided into two parts: static and non-static. Static fields are conceptually bound to the storage box of the object and stick around when set! is used. However static and non-static fields are in the same namespace, so if a non-static field is in the new value during set! it will override the old value. Here's an example:

(define a 10 (define set (\ ((value (\ (val) (> val 10))))  (set! owner value)) (static))) (set! a 20) ;set! obj -> obj.set (a.set.source) => (\ ((value (\ (val) (> val 10)))) (set! owner value))

-- Overload and override --

When defining a subclass and overloading a field one often easily uses define. This is a bad idea however since define creates a whole new field that doesn't share any of the attributes and stuff like that with the old field. Instead SilverScheme has a method called (overload) which is used like this: (define-class Foo  (define bar (\ __some_code__) (private)))

(define-class Fred (Foo) (overload bar ((\ __some_other_code__))))

And voila, bar is copied. Here's a template for overload: (overload __name__ (__new_value__) __attributes__). If the new value part is empty the value isn't changed (that's why I have it between parenthesises).

The (define) function will do an override. In some cases it might be helpful to warn or throw an error when define is used. In such cases the (override) function (which is simply define with another name) is useful.

The override error checking can be turned on and off using the environment variable check-override-errors.

-- Temporary environment variables --

It's possible to create temporary environment variables that only exist in a certain scope. One can use the (let-environment-variable) function for this, it takes the same kinds of arguments as let, but puts them under env. The advantage over a set..body..unset construction is that let-environment-variable automatically unsets on error.

-- Primitives --

SilverScheme's version of intercalls are called primitives. Primitives are always functions (RPC functions to be exact), but they are not objects. Primitives can only be called from Base (baselib). Any attempt to do weird things with primitives may be punished with unexpected behaviour (rationale: it's internal stuff, only a few core hackers will touch it and they will know what to do). There is one primitive handler which is defined as (call-primitive name . args) and is private in the top level of the Base module. The primitive handler can take and return complex objects, however mostly it will operate on simple things.

The advantage of a single primitive handler over for example a primitive attribute is that there is just one entry point that's easily documented. After all the accepted function names will probably be visible in a switch statement.

-- Relative modification --

The basic idea behind serialization in SilverScheme is 'relative modification'. Instead of saying "send this object" the serialization mechanism says "I've got an object with this identity, do you need it". Obviously you don't need the same object twice, so the amount of data send over the network is decreased.

Another possibility with the serialization mechanism is sending in patches to serialized libraries. This has a huge advantage, namely that you only need to send the actual changes, instead of the textual changes (diff sometimes sends over half the file for one line changed, SilverScheme would never do that).

The trick in serialization is the basic SilverScheme object itself. This is how an object looks like:

object                    field ++             +---+ ++        |     +---+ ++        |     +-+-+ +--+-+        |     +-+-+ +--+-+        | +--+-+        | ++
 * identity |        +---> |  subject  |
 * prototype |       |     |  name     |
 * changed  |        |     |  object |C|
 * ivalue |C|        |
 * code   |C|        |
 * scope    | ---+

The C subfields are change flags.

The identity of an object is to the outside world an uri like 'http://myproject/1003', internally it's a number. The identity of an object is unique, no two different objects with the same identity can exist at any time.

The prototype of an object is the object that this object is a copy of. Often this is the prototype of the class of the object, in case the object is copied using for example define it's the original (the second arg to define). Objects can have prototypes that are not in the current session (like a prototype from the web).

The changed flag of an object is set on whenever the object is no longer a perfect copy of the prototype. The changed flag is a kind of warning flag, it doesn't tell what has changed only that something has changed.

The ivalue is the internal value of an object. It's were for example the value of a String object is stored in the native representation of the string (or more likely a pointer to the native representation of the string). It has a changed flag to signal when the internal value has changed.

The code property is the code section of an object. It has a changed flag to signal when the code has changed.

For each field in the scope there is a name, a subject reference and an object reference with a changed flag. If the field is bound to a new object (with a different identity) the change flag goes on.

To send over an object one first needs to ask the other party if he has the prototype. If not the whole object is send over. If he has the prototype one must look at the object's change flag, if it's off one can suffice with saying "it's a perfect copy of the prototype", otherwise one has to look up which change flags are on and send over the information in those properties.

-- The library system --

SilverScheme has two types of libraries: source libraries and static state libraries (I'll call them state libraries from now on). A source library is simply a bunch of SilverScheme source files. A state library is a serialized library. Typically a state library is a relative serialization patch against it's dependencies (to keep lib size down) and some extra files (like pictures, should it be needed).

State libraries are usually published on the WWW with an URI like this: 'http://domain/prefix/name/build/', for example 'http://somedomain/home/peter/myproject/12/'. The prefix is just the path to the project directory. The build is a number that increases when you create a new build of the state files, it runs independent of the version number and is the primary means of identifying a version of the library. Additionally there may be something like 'http://somedomain/home/peter/myproject/0.0.1/' where the version number is just an alias for a certain build number.

-- Introspection --

Static state files in SilverScheme are similar to the output of the introspector, indeed they are the output of the build-in SilverScheme introspector. Thus they contain all information about a program (at the time of serialization), except the stack and some other information needed for running (non-static state files contain these, but that stuff is out of scope here (I'm not even sure if there will ever be non-static state files)). Thus the (static) state files contain all the information needed for analyzation of the library. I'll create an ontology for the state files similar to the introspector ontology for the introspected IL files so that the state files will be processable with the introspector toolchain, given a few small adaptions to cope with the language differences. Unfortunately I can't use the introspector ontology due to the differences between SilverScheme and IL.

-- The SilverScheme Repository --

This is really wishful dreaming, but a nice dream :-). The SilverScheme Repository will be something like CPAN, but more extended toward introspection. It will keep libraries as both compressed source and as state files, it will keep programs (executable programs) as compressed source (you can't really serialize those to static state files since there is no natural idle point where the program is loaded but not executing). Due to an intelligent toolchain coupled to a webservice and web interface it's possible to see the structure of libraries, the documentation of libraries, etc, generated on demand out of the state files.