Next Previous Contents

13. Advanced Topics: 5070 Command Reference

In addition to the text based GUI the RAID configuration may also be manipulated from the husky prompt ( the : raid; prompt) of the onboard controller. This section describes commands that a user can input interactively or via a script file to the K9 kernel. Since K9 is an ANSI C Application Programming Interface (API) a shell is needed to interpret user input and form output. Only one shell is currently available and it is called husky. The K9 kernel is modelled on the Plan 9 operating system whose design is discussed in several papers from AT&T (See the "Further Reading" section for more information). K9 is a kernel targeted at embedded controllers of small to medium complexity (e.g. ISDN-ethernet bridges, RAID controllers, etc). It supports multiple lightweight processes (i.e. without memory management) on a single CPU with a non-pre-emptive scheduler. Device driver architecture is based on Plan 9 (and Unix SVR4) STREAMS. Concurrency control mechanisms include semaphores and signals. The husky shell is modelled on a scaled down Unix Bourne shell.

Using the built-in commands the user can write new scripts thus extending the functionality of the 5070. The commands (adapted from the 5070 man pages) are extensive and are described below.

13.1 AUTOBOOT - script to automatically create all raid sets and scsi monitors

13.2 AUTOFAULT - script to automatically mark a backend faulty after a drive failure

13.3 AUTOREPAIR - script to automatically allocate a spare and reconstruct a raid set

This device reconstruction will take anywhere from 10 minutes to one and a half hours depending on both the size and speed of the backends and the amount of activity the host is generating.

During device reconstruction, pairs of numbers will be printed indicating each 10% of data reconstructed. The pairs of numbers are separated by a slash character, the first number being the number of blocks reconstructed so far and the second being the number number of blocks to be reconstructed. Further status about the rebuild can be gained from running rebuild.

When the spare is allocated both the number of spares currently used on the backend and the spare device name is printed. The number of spares on a backend is referred to the depth of spares on the backend. Thus prior to re-engaging the spare after a reconstruction a check can be made to see if the depth is the same. If it is not, then the spare reconstruction failed and reconstruction using another spare is underway (or no spares are available), and hence we don't re-engage the drive.

13.4 BIND - combine elements of the namespace

13.5 BUZZER - get the state or turn on or off the buzzer

13.6 CACHE - display information about and delete cache ranges

13.7 CACHEDUMP - Dump the contents of the write cache to battery backed-up ram

13.8 CACHERESTORE - Load the cache with data from battery backed-up ram

13.9 CAT - concatenate files and print on the standard output

13.10 CMP - compare the contents of 2 files

13.11 CONS - console device for Husky

On a Unix system this is equivalent to:

bind -k unixfd /dev/cons

On a DOS system this is equivalent to:

bind -k doscon /dev/cons

On target hardware using a SCN2681 chip this is equivalent to:

bind -k scn2681 /dev/cons

13.12 DD - copy a file (disk, etc)

The number after the "+" is the number of fractional blocks (i.e. blocks that are less than the block size) involved. This number will usually be zero (and is otherwise when physical media with alignment requirements is involved).

A write failure outputting the last block on the previous example would cause the following output:

Write failed

8+0 records in

7+0 records out

13.13 DEVSCMP - Compare a file's size against a given value

13.14 DFORMAT- Perform formatting functions on a backend disk drive

13.15 DIAGS - script to run a diagnostic on a given device

13.16 DPART - edit a scsihd disk partition table

13.17 DUP - open file descriptor device

13.18 ECHO - display a line of text

\c suppress trailing newline

\f form feed

\n new line

\r carriage return

\t horizontal tab

\v vertical tab

\\ backslash

\nnn the character whose ASCII code is nnn (octal)

13.19 ENV- environment variables file system

13.20 ENVIRON - RaidRunner Global environment variables - names and effects

To over-ride the number of parity buffers for ALL raid 3's (and set only 72 parity buffers) set

: raid ; setenv RAID3_Default_PBUFS 128

If you set a default for all raid sets of a particular type, but want ONE of them to be different then set up a variable for that particular raid set as it's value will over-ride the default. In the above example, where all Raid Type 3 will have 128 parity buffers, you could set the variable

: raid ; setenv RAID3_Dbase_PBUFS 56

which will allow the raid 3 raid set named 'Dbase' to have 56 parity buffers, but all other raid 3's defined on the RaidRunner will have 128.

13.21 EXEC - cause arguments to be executed in place of this shell

13.22 EXIT - exit a K9 process

13.23 EXPR - evaluation of numeric expressions

+

Does nothing to expression/number to the right.

-

negates expression/number to the right.

!

logically negate expression/number to the right.

~

Bitwise negate expression/number to the right.

BINARY ARITHMETIC OPERATORS

*

Multiply enclosing expressions [2]

/

Integer division of enclosing expressions

%

Modulus of enclosing expressions.

+

Add enclosing expressions

-

Subtract enclosing expressions.

<<

Shift left expression _left_ by number in right expression. Equivalent to: left * (2 ** right)

>>

Shift left expression _right_ by number in right expression. Equivalent to: left / (2 ** right)

&

Bitwise AND of enclosing expressions

^

Bitwise exclusive OR of enclosing expressions. [8]

|

Bitwise OR of enclosing expressions. [9]

BINARY LOGICAL OPERATORS

These logical operators yield the number 1 for a true comparison and 0 for a false comparison. For logical ANDs and ORs their left and right expressions are assumed to be false if 0 otherwise true. Both logical ANDs and ORs evaluate both their left and right expressions in all case (cf. C's short-circuit action).

<=

true when left less than or equal to right. [5]

>=

true when left greater than or equal to right. [5]

<

true when left less than right. [5]

>

true when left greater than right. [5]

==

true when left equal to right. [6]

!=

true when left not equal to right. [6]

&&

logical AND of enclosing expressions [10]

||

logical OR of enclosing expressions [11]

ASSIGNMENT OPERATORS

In the following descriptions "n" is an environment variable while "r_exp" is an expression to the right. All assignment operators have the same precedence which is lower than all other operators. N.B. Multiple assignment operators group right-to-left (i.e. same as C language).

=

Assign right expression into environment variable on left.

*=

n *= r_exp is equivalent to: n = n * r_exp

/=

n /= r_exp is equivalent to: n = n / r_exp

%=

n %= r_exp is equivalent to: n = n % r_exp

+=

n += r_exp is equivalent to: n = n + r_exp

-=

n -= r_exp is equivalent to: n = n - r_exp

<<=

n <<= r_exp is equivalent to: n = n << r_exp

>>=

n >>= r_exp is equivalent to: n = n >> r_exp

&=

n &= r_exp is equivalent to: n = n & r_exp

|=

n |= r_exp is equivalent to: n = n | r_exp

13.26 GET - select one value from list

13.27 GETIV - get the value an internal RaidRunner variable

13.28 HELP - print a list of commands and their synopses

13.29 HUSKY - shell for K9 kernel

As the file "/env/status" is an environment variable the return status of a command is also available in the variable $status. The exit status of a pipeline is the exit status of the last command in the pipeline.

This example would create the file "/d4".

The output of another husky command or script can be made available inline by starting the sequence with "`" and finishing it with a "'". For example:

echo {ps output follows:

} `ps'

This prints the string "ps output follows:" followed on the next line by the current output from the command "ps". That output from "ps" would have its embedded newlines replaced by whitespaces.

13.30 HWCONF - print various hardware configuration details

13.31 HWMON - monitoring daemon for temperature, fans, PSUs.

Certain out-of-band occurrences can be considered to be catastrophic, meaning if the occurrence remains uncorrected, the RaidRunner's hardware is likely to be damaged. Occurrences such as total fan failure and sustained high temperature along with total or partial fan failure are considered as catastrophic. hwmon has a means of automatically placing the RaidRunner into a "shutdown" or quiescent state where minimal power is consumed (and hence less heat is generated). This is done by the execution of the shutdown command after a period of time where catastrophic out-of-band occurrences are sustained. This process is enabled, via the AutoShutdownSecs internal variable. See the internals manual for use of this variable. hwmon can be prevented from starting at boot time by creating the global environment variable NoHwmon and setting any value to it. A warning message will be stored in the syslog.

13.32 INTERNALS - Internal variables used by RaidRunner to change dynamics of running kernel

13.33 KILL - send a signal to the nominated process

hup - hangup [1]

int - interrupt (rubout) [2]

quit - quit (ASCII FS) [3]

kill - kill (cannot be caught or ignored) [4]

pipe - write on a pipe with no one to read it [5]

alrm - alarm clock [6]

term - software termination signal [7]

cld - child process has changed state [8]

nomem - could not obtain memory (from heap) [9]

You cannot kill processes whose process id is between 0 and 5 inclusive. These are considered sacrosanct - hyena, init and console reader/writers.

13.34 LED- turn on/off LED's on RaidRunner

Different models of RaidRunner have various differences in number of LED's and their functionality. Depending on the type of LED, the ok, faulty, warning and rebuild functions perform different functions. See your RaidRunner's Hardware Reference manual to see what LED's exist and what different functions do.

13.35 LFLASH- flash a led on RaidRunner

led2 - LED2 on the RaidRunner controller itself

Dc.s.l - Device on channel c, scsi id s, scsi lun l

status - the status LED on the RaidRunner

io - the io LED on the RaidRunner

13.36 LINE - copies one line of standard input to standard output

13.37 LLENGTH - return the number of elements in the given list

13.38 LOG - like zero with additional logging of accesses

13.39 LRANGE - extract a range of elements from the given list

13.40 LS - list the files in a directory

13.41 LSEARCH - find the a pattern in a list

13.42 LSUBSTR - replace a character in all elements of a list

13.43 MEM - memory mapped file (system)

13.44 MDEBUG - exercise and display statistics about memory allocation

The first line indicates the debug mode, either off, on or trace. The second line indicates the number times a request for memory is made (to Mmalloc() or Mcalloc() and related functions) and the number of times the memory allocator is called to free memory (via Mfree()). The difference between these first two numbers is the total number of currently allocated blocks of memory, with the number between the '(' and ')' being the total memory requested. Note that the amount of memory actually assign may be more than requested.

The third line indicates the amount of memory being managed. The second number is the total memory man aged (i.e. left over after loading the statically allocated text, data and bss space). The first number is that left over after various memory allocation tables have been subtracted out from that afore mention number. The fourth line is the total amount of extra memory assigned to requests in excess of the actual requested memory as compared with the totals on line 3.

The fifth line relates to the list of currently allocated memory. The first number is the number of free entries left and the second is the maximum table size. Note that the number of currently allocated blocks (third number on line 2) when added to the first number on line 5 gives the second number on line 5.

The above three options can generate copious output and require a detailed knowledge of the source to understand their meaning.

13.45 MKDIR - create directory (or directories)

13.46 MKDISKFS - script to create a disk filesystem

13.47 MKHOSTFS - script to create a host port filesystem

13.48 MKRAID - script to create a raid given a line of output of rconf

13.49 MKRAIDFS - script to create a raid filesystem

13.50 MKSMON - script to start the scsi monitor daemon smon

13.51 MKSTARGD - script to initialize a scsi target daemon for a given raid set

13.52 MSTARGD - monitor for stargd

13.53 NICE - Change the K9 run-queue priority of a K9 process

13.54 NULL- file to throw away output in

13.55 PARACC - display information about hardware parity accelerator

All other lines are only informative for debugging purposes. If there is no accelerator present, then the Parity Memory available will be 0.

13.56 PEDIT - Display/modify SCSI backend Mode Parameters Pages

0x2 DISCONNECT page

0x3 FORMAT page

0x4 GEOMETRY page

0x8 CACHE page

0xc NOTCH page

0xa CONTROL page

13.57 PIPE - two way interprocess communication

13.58 PRANKS - print or set the accessible backend ranks for the current controller

13.59 PRINTENV - print one or all GLOBAL environment variables

It is an error to write to the second level directory file "status". A signal number or signal name (see kill for a list of signal names - which cannot be abbreviated in this case) may be written to the file "signal". This action will send a signal to the associated process. If a signal number or name is written to the file "sigpgrp" then all the processes in the process group which this process belongs to will receive the signal.

13.60 PS - report process status

13.61 PSCSIRES - print SCSI-2 reservation table for all or specific monikers

13.62 PSTATUS - print the values of hardware status registers

13.63 RAIDACTION- script to gather/reset stats or stop/start a raid set's stargd

13.64 RAID0 - raid 0 device

The queueLen argument must be 1 or greater and sets the maximum number of requests that can be put in a queue associated with each backend file. A daemon is spawned for each backend file to service this queue called async_io. Each backend file first needs to be identified to the raid0 device via the "engage" string sent to bind_point/ctl. If required a file can have its association with this device terminated with a "disengage" string. Once a backend file is engaged its access level can be varied between "read-write", "read-only", "write-only" and "offline" as required. The default is "offline" so in most initialization situations an "access read-write" string needs to be sent to this device. When the file bind_point/ctl is read then a line is output for every engaged backend file indicating its access status (e.g. "drive 3: engaged, read-write"). Also backend files that have been disengaged and not "re-"engaged output a line (e.g. "drive 5: disengaged").

When the file bind_point/stats is read then a line is output which shows the cumulative number of reads and writes performed (including failures) for each backend of the raid device. The format of this line is D0 r0_cnt r0_fails w0_cnt w0_fails; D1 r1_cnt r1_fails w1_cnt w1_fails; ... which indicates that backend 0 (typically the drive0) has made r0_cnt reads, w0_cnt writes, r0_fails read failures and w0_fails write failures and that backend 1 (drive 1) has made r1_cnt reads, w1_cnt writes, r1_fails read failures and w1_fails write failures and so forth for each backend in the raid set.

If the string "zerostats" is written to the file bind_point/stats then all cumulative read and write counts for each backend of the raid set are zeroed.

This example creates the file "/raid0" as a bind point and then binds the raid0 device on it. The first echo command establishes the internal raid device name as R_0. The subsequent echo commands are shown in pairs for each backend file: one sending an "engage" string and the other sending an "access" string to the file "/raid0/ctl". Each "engage" string associates a backend file (via file descriptor 7) with a block size of 8192 bytes and a maximum queue length of 8. The following "access" string adjusts the access level of the backend file from "offline" (the default) to "read-write". This is a six disk raid set.

13.65 RAID1 - raid 1 device

The "logical" block size is currently 512 bytes and the given blockSize must be a power of 2 times 512 (i.e. 2**n * 512 bytes). If, for example, the blockSize was 8 Kb then a write of 8 Kb would cause both backend files to have that 8 Kb written to them. An 8 Kb read would cause the file calculated to have its "heads" closer to be read. If this file was marked "offline", "write-only" or reported an IO error then the other file would be read.

The queueLen argument must be 1 or greater and sets the maximum number of requests that can be put in a queue associated with each backend file. A daemon is spawned for each backend file to service this queue called async_io. The name argument allows associates the given backendname string with the appropriate backend. This string will be used in reporting errors on the running raid.

Each backend file first needs to be identified to the raid1 device via the "engage" string sent to bind_point/ctl. If required a file can have its association with this device terminated with a "disengage" string. Once a backend file is engaged its access level can be varied between "read-write", "read-only", "write-only" and "offline" as required. The default is "offline" so in most initialization situations an "access read-write" string needs to be sent to this device.

When the file bind_point/ctl is read then a line is output for every engaged backend file indicating its access status (e.g. "drive 3: engaged, read-write"). Also backend files that have been disengaged and not "re-"engaged output a line (e.g. "drive 5: disengaged").

When the file bind_point/stats is read then a line is output which shows the cumulative number of reads and writes performed (including failures) for each backend of the raid device. The format of this line is:

D0 r0_cnt r0_fails w0_cnt w0_fails; D1 r1_cnt r1_fails w1_cnt w1_fails;

which indicates that backend 0 (typically the drive0) has made r0_cnt reads, w0_cnt writes, r0_fails read failures and w0_fails write failures and that backend 1 (drive 1) has made r1_cnt reads, w1_cnt writes, r1_fails read failures and w1_fails write failures.

If the string "zerostats" is written to the file bind_point/stats then all cumulative read and write counts for each backend of the raid set are zeroed.

This example creates the file "/raid1" as a bind point and then binds the raid1 device on it. The first echo command

establishes the internal raid device name as R_1. The subsequent echo commands are shown in pairs for both backend files: one sending an "engage" string and the other sending an "access" string to the file "/raid1/ctl". Each "engage" string associates a backend file (via file descriptor 7) with a block size of 8192 bytes and a maximum queue length of 8. The following "access" string adjusts the access level of the backend file from "offline" (the default) to "read-write".

13.66 RAID3 - raid 3 device

To associate an internal name (or moniker) with the raid device, send the message "moniker name=internal_name" to the device's control file, bind_point/ctl.

This implementation of raid 3 uses at least 3 files in its backend. Read and write operations to the frontend (i.e. bind_point/data) must be in integral units of blockSize. Each write of blockSize bytes is striped (i.e. divided evenly) across (nbackends - 1) files with the "parity" on the other file. Subsequent writes will NOT rotate the file being used to store parity. [This rotation is a slight extension of the original raid 3 definition.] The backend "files" referred to here will typically be disks.

The "logical" block size is currently 512 bytes and the given blockSize must be an integral multiple of (nbackends - 1) * 512 If, for example, the blockSize was 8 Kb and there were 5 backends then a write of 8 Kb would cause 4 backend files to have 2 Kb written on them and the other backend file to have 2 Kb of parity written on it. An 8 Kb read would cause the 4 files known to hold the data (as distinct from the parity) to be read. If any one of these files was marked "offline", "write-only" or reported an IO error then the 5th file containing the parity would be read and the 8 Kb block reconstructed.

The queueLen argument must be 1 or greater and sets the maximum number of requests that can be put in a queue associated with each backend file. A daemon is spawned for each backend file to service this queue called async_io.

The name argument allows associates the given backendname string with the appropriate backend. This string will be used in reporting errors on the running raid.

Each backend file first needs to be identified to the raid3 device via the "engage" string sent to bind_point/ctl. If required a file can have its association with this device terminated with a "disengage" string. Once a backend file is engaged its access level can be varied between "read-write", "read-only", "write-only" and "offline" as required. The default is "offline" so in most initialization situations an "access read-write" string needs to be sent to this device. When the file bind_point/ctl is read then a line is output for every engaged backend file indicating its access status (e.g. "drive 3: engaged, read-write"). Also backend files that have been disengaged and not "re-"engaged output a line (e.g. "drive 5: disengaged").

When the file bind_point/stats is read then a line is output which shows the cumulative number of reads and writes performed (including failures) for each backend of the raid device. The format of this line is

D0 r0_cnt r0_fails w0_cnt w0_fails; D1 r1_cnt r1_fails w1_cnt w1_fails;
 ...

which indicates that backend 0 (typically the drive0) has made r0_cnt reads, w0_cnt writes, r0_fails read failures and w0_fails write failures and that backend 1 (drive 1) has made r1_cnt reads, w1_cnt writes, r1_fails read failures and w1_fails write failures and so forth for each backend in the raid set.

If the string "zerostats" is written to the file bind_point/stats then all cumulative read and write counts for each backend of the raid set are zeroed.

This example creates the file "/raid3" as a bind point and then binds the raid3 device on it. The first echo command establishes the internal raid device name as R_3. The subsequent echo commands are shown in pairs for each backend file: one sending an "engage" string and the other sending an "access" string to the file "/raid3/ctl". Each "engage" string associates a backend file (via file descriptor 7) with a block size of 8192 bytes and a maximum queue length of 8. The following "access" string adjusts the access level of the backend file from "offline" (the default) to "read-write". This is a 6 disk raid set.

13.67 RAID4 - raid 4 device

The "logical" block size is currently 512 bytes and the given blockSize must be an integral multiple of (nbackends - 1) * 512 If, for example, the blockSize was 8 Kb then a write of 8 Kb would cause 1 backend file to have that 8 Kb written to it with the other (nbackends - 2) non-parity files in that slice having 8 Kb read from them in order to generate a new 8 Kb parity block which is then written to the parity file in this slice. An 8 Kb read would cause the file known to hold the data (as distinct from the parity) to be read. If this file was marked "offline", "write-only" or reported an IO error then the other ((nbackends - 1) files in the slice (i.e. (nbackends - 2) data and 1 parity) would be read and the 8 Kb block reconstructed.

The queueLen argument must be 1 or greater and sets the maximum number of requests that can be put in a queue associated with each backend file. A daemon is spawned for each backend file to service this queue called async_io.

The name argument allows associates the given backendname string with the appropriate backend. This string will be used in reporting errors on the running raid.

Each backend file first needs to be identified to the raid5 device via the "engage" string sent to bind_point/ctl. If required a file can have its association with this device terminated with a "disengage" string. Once a backend file is engaged its access level can be varied between "read-write", "read-only", "write-only" and "offline" as required. The default is "offline" so in most initialization situations an "access read-write" string needs to be sent to this device.

When the file bind_point/ctl is read then a line is output for every engaged backend file indicating its access status (e.g. "drive 3: engaged, read-write"). Also backend files that have been disengaged and not "re-"engaged output a line (e.g. "drive 5: disengaged").

When the file bind_point/stats is read then a line is output which shows the cumulative number of reads and writes performed (including failures) for each backend of the raid device. The format of this line is

D0 r0_cnt r0_fails w0_cnt w0_fails; D1 r1_cnt r1_fails w1_cnt w1_fails;
 ...

which indicates that backend 0 (typically the drive0) has made r0_cnt reads, w0_cnt writes, r0_fails read failures and w0_fails write failures and that backend 1 (drive 1) has made r1_cnt reads, w1_cnt writes, r1_fails read failures and w1_fails write failures and so forth for each backend in the raid set.

If the string "zerostats" is written to the file bind_point/stats then all cumulative read and write counts for each backend of the raid set are zeroed.

This example creates the file "/raid4" as a bind point and then binds the raid4 device on it. The first echo command establishes the internal raid device name as R_4. The subsequent echo commands are shown in pairs for each backend file: one sending an "engage" string and the other sending an "access" string to the file "/raid4/ctl". Each "engage" string associates a backend file (via file descriptor 7) with a block size of 8192 bytes and a maximum queue length of 8. The following "access" string adjusts the access level of the backend file from "offline" (the default) to "read-write". This is a six disk raid set.

13.68 RAID5 - raid 5 device

To associate an internal name (or moniker) with the raid device, send the message "moniker name=internal_name" to the device's control file, bind_point/ctl.

This implementation of raid 5 uses at least 3 files in its backend. Read and write operations to the frontend (i.e. bind_point/data) must be in integral units of blockSize. Each write of blockSize bytes is written on 1 backend file. Its neighboring (nbackends - 2) files need to be read at the same offset to calculate a new parity block which needs to be re-written. The nbackends blocks at the same offset on the nbackends backend files are called a slice. The parity block is rotated from one slice to the next. A read of blockSize bytes needs only to read 1 backend file (unless there is a problem). The backend "files" referred to here will typically be disks.

The "logical" block size is currently 512 bytes and the given blockSize must be an integral multiple of (nbackends - 1) * 512 If, for example, the blockSize was 8 Kb then a write of 8 Kb would cause 1 backend file to have that 8 Kb written to it with the other (nbackends - 2) non-parity files in that slice having 8 Kb read from them in order to generate a new 8 Kb parity block which is then written to the parity file in this slice. An 8 Kb read would cause the file known to hold the data (as distinct from the parity) to be read. If this file was marked "offline", "write-only" or reported an IO error then the other ((nbackends - 1) files in the slice (i.e. (nbackends - 2) data and 1 parity) would be read and the 8 Kb block reconstructed.

The queueLen argument must be 1 or greater and sets the maximum number of requests that can be put in a queue associated with each backend file. A daemon is spawned for each backend file to service this queue called async_io.

The name argument allows associates the given backendname string with the appropriate backend. This string will be used in reporting errors on the running raid.

Each backend file first needs to be identified to the raid5 device via the "engage" string sent to bind_point/ctl. If required a file can have its association with this device terminated with a "disengage" string. Once a backend file is engaged its access level can be varied between "read-write", "read- only", "write-only" and "offline" as required. The default is "offline" so in most initialization situations an "access read-write" string needs to be sent to this device.

When the file bind_point/ctl is read then a line is output for every engaged backend file indicating its access status (e.g. "drive 3: engaged, read-write"). Also backend files that have been disengaged and not "re-"engaged output a line (e.g. "drive 5: disengaged").

When the file bind_point/stats is read then a line is output which shows the cumulative number of reads and writes performed (including failures) for each backend of the raid device. The format of this line is

D0 r0_cnt r0_fails w0_cnt w0_fails; D1 r1_cnt r1_fails w1_cnt w1_fails;
 ...

which indicates that backend 0 (typically the drive0) has made r0_cnt reads, w0_cnt writes, r0_fails read failures and w0_fails write failures and that backend 1 (drive 1) has made r1_cnt reads, w1_cnt writes, r1_fails read failures and w1_fails write failures and so forth for each backend in the raid set.

If the string "zerostats" is written to the file bind_point/stats then all cumulative read and write counts for each backend of the raid set are zeroed.

This example creates the file "/raid5" as a bind point and then binds the raid5 device on it. The first echo command establishes the internal raid device name as R_5. The subsequent echo commands are shown in pairs for each backend file: one sending an "engage" string and the other sending an "access" string to the file "/raid5/ctl". Each "engage" string associates a backend file (via file descriptor 7) with a block size of 8192 bytes and a maximum queue length of 8. The following "access" string adjusts the access level of the backend file from "offline" (the default) to "read-write". This is a six disk raid set.

13.69 RAM - ram based file system

13.70 RANDIO - simulate random reads and writes

13.71 RCONF, SPOOL, HCONF, MCONF, CORRUPT-CONFIG - raid configuration and spares management


D0.2.0-,D1s2l0,D2.2.0,D3.2.0,D4.2.0

means that the backend D0.2.0 has failed and does not have a spare and the other backends are

D1s2l0,D2.2.0,D3.2.0,D4.2.0.

D0.2.0-D5.2.0,D1s2l0,D2.2.0,D3.2.0,D4.2.0

means that the backend D0.2.0 has failed and the spare backend, D5.2.0, has replaced it and the other backends are D1s2l0,D2.2.0,D3.2.0,D4.2.0.

D0.2.0-D5.2.0-,D1s2l0,D2.2.0,D3.2.0,D4.2.0

means that the backend D0.2.0 has failed and the spare backend, D5.2.0, which replaced it has also failed and the other backends are D1.2.0,D2.2.0,D3.2.0,D4.2.0.


rconf -repair .. -action start ...

rebuild the raid set

rconf -repair .. -action finish ...

Now, if the spare we allocated is faulty or the reconstruction fails, we need to turn off the "under construction" state, deallocate the spare and re-mark the original failing drive as faulty (as well as the spare drive if it was faulty). The -unrepair option does this. The options are similar to the

Specifying no options is the same as specifying the -list option.

Backend BackendSize hot|warm controller_no|Any Used|Unused Faulty|Working
 comments

13.72 REBOOT - exit K9 on target hardware + return to monitor

13.73 REBUILD - raid set reconstruction utility

The following exit values are returned:

13.74 REPAIR - script to allocate a spare to a raid set's failed backend

13.75 REPLACE - script to restore a backend in a raid set

13.76 RM - remove the file (or files)

13.77 RMON - Power-On Diagnostics and Bootstrap

The first line printed by rmon is it's version number. The second line is the size (in hexa-decimal) of DRAM. The next line indicates the type and usage of Battery-Backup SRAM found. This can be a variety of values and if there are two hexa-decimal values printed, this is then the range of inclusive addresses upon which the power-on diagnostics will test Battery Backed-up SRAM. If there is only one value this indicates the size of Battery Backed-up SRAM, but no power-on diagnostics will performed upon it. All possible values are :-

         Batt: 00000000          No Battery Backed-up SRAM present.

 

         Batt: B0000000-B007FFFC 512Kb SRAM, B0000004==0 (No data).

         Batt: B0000008-B007FFFC 512Kb SRAM, B0000004==1 (No data).

         Batt: B00XXXXX-B007FFFC 512Kb SRAM, B0000004==2 (Saved data,

                                 B0000000==B00XXXXX, where B00XXXXX is

                                 start of unused data).

         Batt: B0080000          512Kb SRAM present, but the contents of

                                 SRAM is not one of the above, or could

                                 be B0000004==2 and B0000000==B0080000.

                                 No pon testing done on it.

 

         Batt: B0000000-B03FFFFC 4Mb SRAM, B0000004==0 (No data).

         Batt: B0000008-B03FFFFC 4Mb SRAM, B0000004==1 (No data).

         Batt: B0XXXXXX-B03FFFFC 4Mb SRAM, B0000004==2 (Saved data,

                                 B0000000==B0XXXXXX, where B0XXXXXX is

                                 start of unused data).

         Batt: B0400000          4Mb SRAM present, but the contents of

                                 SRAM is not one of the above, or could

                                 be B0000004==2 and B0000000==B0400000.

                                 No pon testing done on it.

Note the case where location B0000004 has a value of 2 (Saved data present), but where B0000000 (the start of unused data) points to a word address just past the last byte of Battery Backed-up SRAM. This latter case will cause a single value (say B0080000, for 512Kb Battery Backed-up SRAM) to be printed. This case does not indicate that any contents of Battery Backed-up SRAM is deemed incorrect, but that Battery Backed-up SRAM is full of data and that no power-on diagnostics will be performed upon Battery Backed-up SRAM.

The DRAM memory sizing algorithm will cater for between 8Mb and 256Mb of DRAM, and there need be only 30 bits out of 32 bits words in each DRAM set that need work properly, for the sizing to function properly. The DRAM consists of two banks of memory, A and B, each of two slots. When using a bank of memory both slots of that bank must be occupied with a Simm of the same type and size. If only one bank is to be used, then it must be Bank A. The type of Simm memory may be single or double sided. Placing two double sided Simms (which, as mentioned above must be both the same size) into Bank A is equivalent to populating all four slots with single sided Simms of a size that is exactly one half of the double sided Simms size. The Bank B in such a case cannot contain any Simms. Each bank can potentially be using single sided Simms of different sizes, but the first bank must have the larger sized Simms.

            DRAM Size DRAM last address

            Hex Dec   Cached    Uncached

            008  8    7FFFFF    A07FFFFF

            010  16   FFFFFF    A0FFFFFF

            020  32   1FFFFFF   A1FFFFFF

            040  64   3FFFFFF   A3FFFFFF

            080  128  7FFFFFF   A7FFFFFF

            100  256  FFFFFFF   AFFFFFFF

On the fourth line is the power supply status. This will be one of two variations. The first for single power supply systems, the power supply summary status is printed at PWROK (for good) or PFAIL if this is not good. The second is for multi-power supply systems, the summary status will be followed by the individual status of each power supply. A PFAIL will be printed if any of the power supplies are faulty. At this point on non-pseudo-static DRAM systems the monitor will hang if the summary status is PFAIL. A pseudo-static DRAM system will never print this line or any of the preceding lines (i.e. version line, DRAM size or Battery Backed-up SRAM size) and always immediately hang.

On the fifth line is summary of the SCSI chips used in the system. Each item in this report consists of the index No. for that chip (from 0 to 7), followed by a ':' then a string of text indicating the model. This model consisting of it's chip type and revision level. A summary of the known part No.s vs the model is :-

                                  Bits 7 to 4 of

             Model    Part No.    MACNTL CTEST3

              720B   609-0391071   0000   0001

              720C   609-0391324   0000   0010

              720E   609-0391955   0000   0100

            720seE   609-0391949   0001   0100

              770A   609-0392179   0010   0000

              770B   609-0392393   0010   0001

There maybe a sixth indicating the serial number of the unit. If the last 12 bytes of the FlashRam boot partition is left in an unblown state (all ones), then this line will not be appear. Otherwise these last 12 bytes will be printed. These 12 bytes may include trailing blanks. The four bytes of FlashRam preceding the serial no. contain 26 bits representing the revision level of the board. If any of these bits are cleared and there is a serial no. present then the revision level from A to Z will be printed.

After printing the above configuration information, the DRAM and Battery Backed-up SRAM (if any found), will have a Knaizuk-Hartmann memory test performed upon them. This memory test does a quick non-exhaustive check for stuck-at faults in both the address lines as well as the data locations. This test is disabled on the following conditions :-

By typing a space, any memory tests that may have been started will be aborted. The monitor will print a prompt and wait for a monitor command (see below).

All other conditions will result in the main raid code to be read from FlashRam and started. Whilst the memory test is in progress a sequence of -, /, | and characters are printed on the console giving the appearance of a rotating bar. On-board LED 1 will be flashed every 10ms and LED 2 will change state between each phase of the test. As noted above these tests can be abort by typing a space on the console. When rmon is in command mode a variety of commands may be issued. These mostly relate to various diagnostics that can be performed, or to how flash RAM may be upgraded with new firmware.

which will put the byte 0 into memory location 5F eight times, then put the byte 1 into memory location 6F ten times, and repeat both these put byte commands 20 times. Thus the location 5F will have the byte value 0 written into it a total of 160 times and the location 6F will have the byte value 1 writ- ten into it a total of 200 times.

is printed. To differentiate between the three forms of memory access, the suffixes b, w and l are used for byte, short word and long word respectively.

{rmon} db 60 7F

00000060: A0 9F 9E 9D 9C 9B 9A 99 98 97 96 95 94 93 92 91

00000070: 90 8F 8E 8D 8C 8B 8A 89 88 87 86 85 84 83 82 81

{rmon} dw 60 7F

00000060: FFA0 FF9E FF9C FF9A FF98 FF96 FF94 FF92

00000070: FF90 FF8E FF8C FF8A FF88 FF86 FF84 FF82

{rmon} dl 60 7F

00000060: FFFFFFA0 FFFFFF9C FFFFFF98 FFFFFF94

00000070: FFFFFF90 FFFFFF8C FFFFFF88 FFFFFF84


{rmon} pb 60 1 2

{rmon} pw 70 04F2 002F

{rmon} pl 80 8000FF04 8000FF0F 80FFFFFF

will put the value 01 into location 00000060 and 02 into the location 00000061, put the value 04F2 into location 00000070 and 002F into location 00000072, and put the value 8000FF04 into location 00000080, 8000FF0F into location 00000084 and 80FFFFFF into location 00000088 respectively


{rmon} fb 60 70 8F

{rmon} fw 60 70 FF8F

{rmon} fl 80 90 00FFFF11

will put the value 8F into all bytes (inclusive) between locations 00000060 and 00000070, put the value FF8F into all short words (inclusive) between locations 00000060 00000070, and put the value 00FFFF11 into all long words (inclusive) between locations 00000080 and 00000090.


{rmon} tw 60 70 7B

Writing:

Checking:

{rmon}


0x0001         Fill each byte with 0x00

0x0002         Fill each byte with 0xFF

0x0004         Fill alternate bytes with 0x55 then 0xAA

0x0008         Fill each byte with the two's complement of it's address

0x0010         Fill each word with 0x0000

0x0020         Fill each word with 0xFFFF

0x0040         Fill alternate words with 0x5555 then 0xAAAA

0x0080         Fill each word with the two's complement of it's address

0x0100         Fill each long word with 0x00000000

0x0200         Fill each long word with 0xFFFFFFFF

0x0400         Fill alternate long words with 0x55555555 then 0xAAAAAAAA

0x0800         Fill each long word with the two's complement of it's address

0x0001000      Fill each byte with 0x00

0x0002000      Fill each byte with 0xFF

0x0004000      Fill alternate bytes with 0x55 then 0xAA

0x0008000      Fill each byte with the two's complement of it's address

0x0010000      Fill each word with 0x0000

0x0020000      Fill each word with 0xFFFF

0x0040000      Fill alternate words with 0x5555 then 0xAAAA

0x0080000      Fill each word with the two's complement of it's address

0x0100000      Fill each long word with 0x00000000

0x0200000      Fill each long word with 0xFFFFFFFF

0x0400000      Fill alternate long words with 0x55555555 then 0xAAAAAAAA

0x0800000      Fill each long word with the two's complement of it's address

0x1000000      Fill alternate long words with 0x00000000 then 0xFFFFFFFF

will be printed. nn is the number of milliseconds it took for the remote host to respond. If the remote host does not respond within 10 seconds, the message

Sending echo request...

timeout after 10 seconds

will be printed.

The following two examples show the transfer of the raid binary and the transfer of the RaidRunner's /bin directory from a remote host. The remote host's IP address is 192.43.198.101 (C02BC665), the RaidRunner's IP address is 192.43.198.200 (C02BC6C8) and the two files concerned will be /usr/raid/lib/raid.bin and /usr/raid/lib/raid.rc (raid binary and /bin directory respectively). The raid.bin file will be written into the bank of flash ram starting at address bfc10000 and the raid.rc file will be written into the two consecutive banks of flash ram at address bfc08000 and bfc0c000.

{rmon} {first transfer the raid binary - /usr/raid/lib/raid.bin}

{rmon} F /usr/raid/lib/raid.bin

{rmon} A C02BC665 C02BC6C8

{rmon} P

Sending echo request...

Got response after 0x000013 ms

{rmon} T 80300000 803fffff

{rmon} E bfc10000; {erase the Flash EEPROM address for raid.bin
 to be stored}

**

70000

{rmon} W bfc10000 80300000 5BDA0; {copy downloaded raid.bin
 into flash}

5BDA0

{rmon}

{rmon} {now transfer the raid /bin directory - /usr/raid/lib/raid.rc}

{rmon} F /usr/raid/lib/raid.rc

{rmon} A C02BC665 C02BC6C8

{rmon} P

Sending echo request...

Got response after 0x000013 ms

{rmon} T 80300000 803fffff

{rmon} E bfc08000; {erase the two consecutive blocks of
 flash ram where raid.rc}

4000

{rmon} E bfc0c000; {is to be stored}

4000

{rmon} W bfc08000 80300000 4000; {copy the first 0x4000
 bytes}

4000

{rmon} W bfc0c000 80304000 4000; {copy the next 0x4000 bytes}

4000

{rmon}

It is suggested that a serial number be no greater than 10 characters and consist of upper case letters, decimal digits and only a limited number of special characters such as '-' or '.' characters. The main raid code will truncate the serial number to 10 characters if a revision level is present. The main raid code will, if the revision level is present, append the revision level immediately after the serial number and present the combined string as the serial number in an inquiry command.

x000 General form of a diagnostic command it that is it preceded by an x and then immediately followed by a hexi-decimal no. Most diagnostics will loop permanently requiring a power off/on cycle to stop the diagnostic.


7  - DB11       14 - SEL         21 - DB7        28 - DB0

8  - DB10       15 - MSG         22 - DB6        29 - DP1 (*2)

9  - DB9        16 - RST (*1)    23 - DB5        30 - DB15

10 - DB8        17 - ACK         24 - DB4        31 - DB14

11 - I/O        18 - BSY         25 - DB3        32 - DB13

12 - REQ        19 - ATN         26 - DB2        33 - DB12

13 - C/D        20 - DP0 (*2)    27 - DB1

Note that the RST line (marked *1 above) is pulsed only once after the last pin is pulsed (i.e after DB12) and that the parity lines (marked as *2 above) can not be pulsed.


00: 0x3C1A9FC0   [ lui k0,0x9fc0 ]

04: 0x3C1B3C1A   [ lui k1,0x3c1a ]

08: 0xAC1B0000   [ sw  k1,(zero) ]

0C: 0x3C01A000   [ lui at,0xa000 ]

Location 0 is the start of the restart pseudo interrupt handler. The x004 command is then used after restoring power to check for this bit pattern. It is important that the arguments given to x003 are exactly the same as those given to x004. Any words detected that are different are printed, if there are no differences then nothing (but the next prompt) is printed.

13.78 RRSTRACE - disassemble scsihpmtr monitor data

13.79 RSIZE - estimate the memory usage for a given raid set

13.80 SCN2681 - access a scn2681 (serial IO device) as console

13.81 SCSICHIPS - print various details about a controller's scsi chips

which says that there is 1 hostport chip, 6 backend channels, the address of the hostport chip is 6 and the addresses of the backend channel chips are 0, 1, 2, 3, 4 and 5.

For a RaidRunner controller with two hostports and six backend channels scsichips will print

2 6 6 7 0 1 2 3 4 5

which says that there are 2 hostport chips, 6 backend channels, the addresses of the hostport chips are 6 and 7 and the addresses of the backend channel chips are 0, 1, 2, 3, 4 and 5.

13.82 SCSIHD - SCSI hard disk device (a SCSI initiator)

The SCSI disk device is bound into the namespace at bind_point. A one level directory is made at the bind_point containing the files: "raw", "data", "partition", "rconfig" and "ctl". The "partition" file is a small portion at the end of the disk used for storing partition information (in plain ASCII). The "ctl" and "rconfig" files are for internal RaidRunner use. (The "rconfig" file will be a backup copy of the RaidRunner configuration area). The "data" file is the usable part (i.e. the vast majority) of the space available on the SCSI disk connected to the SCSI bus on which this device is an initiator. The file "raw" addresses the whole disk less the partition block.

The current raid hardware has six fast wide SCSI-2 buses numbered 0 to 5 and two fast wide buses num- bered 6 and 7. Usually the first six fast wide SCSI-2 buses are set up as initiators (i.e. bind scsihd) while the two fast wide SCSI buses are set up as targets (i.e. bind scsihp).

Current implementations make the device name hd synonymous with scsihd. Therefore:

bind -k {hd chipNumber scsiTargetId

[scsiTargetLUN]} bind_point has the same effect as the similar line in the synopsis.

13.83 SCSIHP - SCSI target device

The SCSI target device is bound into the namespace at bind_point. A three level directory is made at the bind_point. The first level is the SCSI id (as specified by scsiid) for this device. The second level is the LUN within that scsi id. The third level contains the files: "data" and "cmnd" which are used for SCSI data phases and command/status phases respectively.

A scsi target command interpreter called stargd is designed to "mate" with this device.

The current raid hardware has six fast narrow SCSI buses numbered 0 to 5 and two fast wide buses num- bered 6 and 7. Usually the six fast narrow SCSI buses are set up as initiators (i.e. bind scsihd) while the two fast wide SCSI buses are set up as targets (i.e. bind scsihp).

13.84 SET - set (or clear) an environment variable

13.85 SCSIHPMTR - turn on host port debugging

13.86 SETENV - set a GLOBAL environment variable

13.87 SDLIST - Set or display an internal list of attached disk drives

which would equate to

sdlist 0.1.0 1.1.0 2.1.0 3.1.0 4.1.0 5.1.0

and those six tripletts would be stored in the list, with sdlist having already deleted any already stored list. To see what this internal list contains, the -p option can be given and the list will be printed. This command is typically executed during autoboot and would not be executed interactively unless the user is performing unusual backend manipulations and is debugging the process.

13.88 SETIV - set an internal RaidRunner variable

13.89 SHOWBAT - display information about battery backed-up ram

13.90 SHUTDOWN - script to place the RaidRunner into a shutdown or quiescent state

13.91 SLEEP - sleep for the given number of seconds

13.92 SMON - RaidRunner SCSI monitor daemon

The SCSI initiator should interpret a sense key of "Command aborted" as the target unilaterally aborting a command in progress. The SCSI-2 standard suggests the initiator should retry the aborted command.

This line will invoke this daemon and try and open the following files:

/dev/hostbus/1/6/7/cmnd

and

/dev/hostbus/1/6/7/data

The "-s 2080" switch instructs this daemon to tell SCSI initiators that it is a 1040K disk.

13.93 SOS - pulse the buzzer to emit sos's

13.94 SPEEDTST - Generate a set number of sequential writes then reads

13.95 SPIND - Spin up or down a disk device

13.96 SPINDLE - Modify Spindle Synchronization on a disk device

13.97 SRANKS - set the accessible backend ranks for a controller

ranklist a comma separated list of rank id's (scsi id's) for which the given controller is to have access. sranks will check each argument looking for the controller id corresponding to the controller it's running on and set the backend rank access as per the ranklist. Typically, this command is executed with the output of the BackendRanks GLOBAL environment variable at boot time.j

13.98 STARGD - daemon for SCSI-2 target

Debug messages are typically a single line per command and are sent to standard out. Serious errors are reported to standard error irrespective of the current debug level. If the error is related to incoming data (from the SCSI initiator) then the daemon will continue. If the error cannot be recovered from, then the daemon terminates with an error message as its exit status. The debug level can be changed during the execution of a stargd daemon by using the "-d" option on mstargd.


stargd -c -s 2M -m RS -h 1 /dev/hostbus/1 6 0 /cache/data

This line will invoke this daemon and try and open the following files:

/dev/hostbus/1/6/0/cmnd

and

/dev/hostbus/1/6/0/data

The file "/cache/data" will be used as store. The "-c" switch identifies this file id as a cache to stargd. The "-s 2M" switch instructs this daemon to tell SCSI initiators that it is a 1 GigaByte disk (i.e. 2 MegaBlocks).

00: Test Unit Ready  If backend is ready returns GOOD Status, else sets
 Sense Key to Not Ready and returns CHECK CONDITION Status

01: Rezero Unit      Does nothing, returns GOOD Status

03: Request Sense    Sense data held on a per initiator basis (plus extra
 for bad LUN's)

04: Format Unit      Does nothing, returns GOOD Status

07: Reassign Blocks  Consumes data but does nothing, returns GOOD Status

08: Read_6           DPO, FUA and RelAdr not supported

0a: Write_6          DPO, FUA and RelAdr not supported

0b: Seek_6           Does nothing, returns GOOD Status

12: Inquiry          Only standard 36 byte data format supported (not vital
 product data pages)

15: Mode Select      Support pages 1, 2, 3, 4, 8 and 10 (but none writable)

16: Reserve          Doesn't support extents + 3rd parties

17: Release          Doesn't support extents + 3rd parties

1a: Mode Sense       Support pages 1, 2, 3, 4, 8 and 10.

1b  Start Stop       If Start is requested and the Immediate bit is 0 then
 waits for backend to become ready, else does nothing and returns GOOD Status.
 If backend does not become ready within 20 seconds set Sense Key to Not Ready
 and returns CHECK  CONDITION Status

1d  Send Diagnostics Returns GOOD Status when self test else complains
 (does nothing internally)

25  Read Capacity    RelAdr, PMI and logical address > 0 are not supported

28  Read_10          Same as Read_6

2a  Write_10         Same as Write_6

2b  Seek_10          Does nothing, returns GOOD Status

2f  Verify           Does nothing, returns GOOD Status

55  Mode Select_10   Same as Mode Select     

5a  Mode Sense_10    Same as Mode Sense

13.99 STAT - get status information on the named files (or stdin)

13.100 STATS - Print cumulative performance statistics on a Raid Set or Cache Range

13.101 STRING - perform a string operation on a given value

13.102 SUFFIX - Suffixes permitted on some big decimal numbers

The resulting number must fit in a 32 bit unsigned number which means "3G" can be represented (== 3,221,225,472) but "4G" cannot. "0G" is allowable and is interpreted as zero.

13.103 SYSLOG - device to send system messages for logging

13.104 SYSLOGD - initialize or access messages in the system log area

where timestamp is the time the message was logged recorded as the number of seconds from the time the RaidRunner was booted, message class is the type of message logged indication the importance (or class) of the message. message is the message itself

  1. EMERG: messages of an extremely serious nature from which the RaidRunner cannot recover
  2. ALERT: messages of a serious nature from which the RaidRunner can only partially recover
  3. CRIT: messages of a serious nature from which the RaidRunner can almost fully recover
  4. ERR: messages indicating internal errors
  5. WARNING: messages of a serious from which the RaidRunner can fully recover, for example automatic allocation of hot spare to Raid 1, 3 or 5 file system.
  6. NOTICE :messages logged via writes to syslog device
  7. INFO: informative messages
  8. DEBUG: debugging messages options are given and cnt is set to 20.
  9. REPEATS: Indicates that the previous message has been repeated N times every S seconds since it's initial entry.

13.105 TEST - condition evaluation command

The "!" operator should appear to the left of other unary operators. Basic binary operators have higher precedence than "-a" and "-o".

13.106 TIME - Print the number of seconds since boot (or reset of clock)

13.107 TRAP - intercept a signal and perform some action

13.108 TRUE - returns the K9 true status

13.109 STTY or TTY - print the user's terminal mount point or terminfo status

13.110 UNSET - delete one or more environment variables

13.111 UNSETENV - unset (delete) a GLOBAL environment variable

13.112 VERSION - print out the version of the RaidRunner kernel

13.113 WAIT - wait for a process (or my children) to terminate

13.114 WARBLE - periodically pulse the buzzer

13.115 XD- dump given file(s) in hexa-decimal to standard out

13.116 ZAP - write zeros to a file

SYNOPSIS: zap [-b blockSize] [-f byteVal] count offset <>[3] store

DESCRIPTION: zap writes count * 8192 bytes of zeros at byte position offset * 8192 into file store (which is opened and associated with file descriptor 3). Both count and offset may have a suffix. The optional "-b" switch allows the block size to be set to blockSize bytes. The default block size is 8192 bytes. The optional "-f" switch allows the fill character to be set to byteVal which should be a number in the range 0 to 255 (inclusive). The default fill character is 0 (i.e. zero). Every 100 write operations the current count is output (usually overwriting the previous count output). Errors on the write operations are ignored.

SEE ALSO: suffix

13.117 ZCACHE - Manipulate the zone optimization IO table of a Raid Set's cache

13.118 ZERO - file when read yields zeros continuously

Example of use to make 32 Kilobyte file (called "/fill") full of zeros.

dd if=/dev/zero of=/fill bs=8k count=4

13.119 ZLABELS - Write zeros to the front and end of Raid Sets

Next Previous Contents