WikiSh

Summary: Wiki-based script language roughly emulating linux shell tools
Version: 2015-06-06
Prerequisites: PMWiki 2.2 beta or later, PHP5
Status: Beta
Maintainer: Peter Bowers
Users: +3 (View / Edit)
Download: WikiSh.zipΔ

Required Recipe: SecLayer (WikiSh relies extensively on the security provided through SecLayer - it is required)
Recommended Recipe: Toolbox (WikiSh makes use of toolbox capabilities when they are available)
Related Recipe: WikiShCrypt (WikiShCrypt depends on WikiSh)
Related Recipe: EditCrypt (EditCrypt depends on both WikiSh and WikiShCrypt)

Discussion: WikiSh-Talk

There is a live test site available to try out WikiSh at http://pmwiki.qdk.org/pmwiki.php/WikiSh/ControlPanel (or other pages on the same site). Feel free to check it out.

Table of contents

See The WikiSh Examples Page for some mini-applications and working examples of WikiSh scripts. If your particular interest is in using WikiSh as a forms processor check out The WikiSh Forms Example Page(s) for examples specific to that context.

See The WikiSh Tutorial for a tutorial (starting very basic!) on using WikiSh.

Description

A scripting language emulating linux shell tools to accomplish text & page manipulation and form validation.

WikiSh is not so much a recipe itself as it is an environment for accelerated recipe development. Many quick commands will be useful (even invaluable) in and of themselves, but generally there is going to be some design/development/testing cycle that goes into most usage of WikiSh -- as such it needs to be seen as a development "platform" in which you can very rapidly accomplish relatively complex operations while not having to worry about authorizations, reading/writing files, and things like that because WikiSh will handle that for you. If you are a relatively inexperienced PmWiki author with limited programming experienced you may well be better off looking at the example page to see if there is an existing solution for you rather than trying to develop it on your own. If, on the other hand, you have programming experience (especially if you already know bash scripting!) then WikiSh has huge potential to accelerate your efforts in developing solutions for your PmWiki site.

Questions answered by this recipe

  • Where can I find a scripting language/macro language that I can work with on a page within my wiki-site without going to php & ftp all the time?
  • How can I pull in certain lines from another page into the current page (or writing to another page), selecting lines by pattern or line number or etc?
  • How can I do search/replace either on those pages (i.e., changing those pages) or on the lines I pull into this page from other pages?
  • How can I count lines/words/characters in wiki pages or certain lines within those pages?
  • How can I delete a page (or a large number of pages) quickly?
  • How can I rename a page or move a bunch of pages into a different group?
  • How can I copy one page to another or many pages from one group into another group?
  • How can I copy source from many pages into simple textfiles for external processing?
  • How can I combine several pages into a single page so that the actual text is in that page (not included via directives)?
  • How can I programmatically manipulate wiki pages (and/or simple text files) using in-page (or form-based) mark-up?
  • How can I sort a bunch of lines within a page (or pages) by various fields?
  • How can I compare 2 different pages to see if there are differences? (Verify a copy, etc)
  • How can I validate the input fields from a form without using underlying PHP?

Common misconceptions / concerns

  • Can I use WikiSh even though my server is not linux-based or even though I don't have command-line access to my server?
    • YES! WikiSh is a recipe written entirely in PHP with no other pre-requisites other than PMWiki version 2.2 beta or above.
  • Does WikiSh open my system up to malicious users, spammers, hackers, etc.
    • If WikiSh is properly configured then there should be no risk to your system. Obviously proper configuration is a significant concern - please note the security precautions below.
    • WikiSh is designed primarily as an ADMINISTRATIVE tool and as such it is recommended that only admins have general write privileges. However, WikiSh is a very helpful tool even without write privileges (validating forms, massaging text, searching, etc.).
    • WikiSh categorically respects and enforces PMWiki auth privileges. You can override this with forceread and forceedit, but these are advanced capabilities and should not be used except by experts.
    • Any recipe must be appropriately administered to reduce the risk from malicious users. WikiSh is a general purpose tool with significant capabilities and administrators must take correspondingly greater care to be sure that write privileges are limited appropriately.
    • See security section below for more details.
  • WikiSh is a relatively large PHP program (over 100k). Won't that hurt performance on my system?
    • WikiSh (with all PHP programs) is loaded only server-side rather than being downloaded to clients. There will be some "hit" on resources but on a server with adequate capabilities the "hit" should be negligible. (Hmmm... Compile time could be affected, I suppose - haven't noticed it, though.)
      • If someone has further information on this I would be interested -- I'm no performance expert...
    • If you are really concerned about this issue then you could load the script only when you are pages within the WikiSh group so you can still have the capabilities within the ControlPanel and etc. (See installation option 3.D)
  • Doesn't WikiSh go against several of the underlying philosophies of PmWiki? Make it easy on authors, don't add unnecessary features, that kind of thing?
    • It depends how the administrators/authors use it. With the addition of functions (as of the 2008-07-31 release) potentially complex wikish scripts can be isolated from end-users simply by defining a function on another page and then calling that function within the current page. So, yes, you could go against the philosophy if you chose to use it in a complex manner within sight of non-technical authors. But you can also easily hide that complexity just as you hide the complexity of the underlying PHP of pmwiki in "invisible" PHP files...

Installation:

A simple, recommended configuration is described here. WikiSh provides a great deal of configuration control and flexibility. For more details please see WikiSh Installation & Configuration.

With this configuration all users will be able to make use of WikiSh and will have read access to all files and limited write access to only to pages in the Test group. Admins will have full WikiSh capabilities. The control panel (command line) will be available only to pages within the WikiSh group. (Again, you can choose to give less privileges or more privileges -- this recommended configuration is somewhat middle-of-the-road. See WikiShConfig for more details on other options.)

  1. Download WikiSh.zipΔ
  2. Unzip and place WikiSh.php, WikiShCL.php, WikiShCrypt.php, SecLayer.php, and toolbox.php in your cookbook folder (usually pmwiki/cookbook)
  3. Install in config.php as follows:
include_once("$FarmD/cookbook/toolbox.php");
include_once("$FarmD/cookbook/WikiSh.php");
if ($group == 'WikiSh')
   include_once("$FarmD/cookbook/WikiShCL.php");
include_once("$FarmD/cookbook/SecLayer.php"); // note this includes stdconfig.php
$EnableWikiShWritePage = true;
$EnableWikiShCreatePage = true;
$EnableWikiShOverwritePage = true;
if (CondAuth($pagename, "admin")) {
   slAddAuth($wshAuthPage, "*.*", "read,create,insert,overwrite,append,prepend,attr,delete");
   $EnableWikiShDeletePage = true;
   $EnableWikiShChmod = true;
} else {
   slAddAuth($wshAuthPage, "*.*", "read,create");
   slAddAuth($wshAuthPage, "Test.*", "insert,overwrite,append,prepend,delete");
}
# cookbook/powertools.php is *very* helpful in the wikish environment but is optional.  
# If you decide to install it you will need to go that page and download it and install
# it as follows:
#include_once("$FarmD/cookbook/powertools.php");

The configuration above, when placed in your config.php, is sufficient to start using WikiSh.

Note that if you have stdconfig.php related customizations ($EnableXyz type of thing) then the inclusion of SecLayer.php should occur after these customizations.

If you want to further utilize the capabilities of SecLayer.php with this recommended configuration, then read on below...

If you wanted to be able to manipulate your page authorizations by editing a page rather than through direct editing of the config.php you would use this code within config.php instead of that above:

include_once("$FarmD/cookbook/toolbox.php");
include_once("$FarmD/cookbook/WikiSh.php");
if ($group == 'WikiSh')
   include_once("$FarmD/cookbook/WikiShCL.php");
include_once("$FarmD/cookbook/SecLayer.php");
$EnableWikiShWritePage = true;
$EnableWikiShCreatePage = true;
$EnableWikiShOverwritePage = true;
$pagename = ResolvePageName($pagename);
slParsePage($pagename, "SiteAdmin.WikiShAuth#aliases", $wshAuthPage);
if (CondAuth($pagename, "admin")) {
   slParsePage($pagename, "SiteAdmin.WikiShAuth#admin", $wshAuthPage);
   $EnableWikiShDeletePage = true;
   $EnableWikiShChmod = true;
} else {
   slParsePage($pagename, "SiteAdmin.WikiShAuth#NONadmin", $wshAuthPage);
}
# cookbook/powertools.php is *very* helpful in the wikish environment but is optional.  
# If you decide to install it you will need to go that page and download it and install
# it as follows:
#include_once("$FarmD/cookbook/powertools.php");

This text should be placed in the page SiteAdmin.WikiShAuth:

[[#aliases]]
edit = append,prepend,insert,overwrite,create
all = edit, delete, read, attr

[[#admin]]
*.*:all

[[#NONadmin]]
*.*:read
Test.*:edit

  • SecLayer does not specify what the authorization layers are, but WikiSh honors the following 10 possible authorizations:
    • append -- allow lines to be appended to the end of a page
    • prepend -- allow lines to be prepended before existing lines on a page
    • insert -- allow lines to be inserted between other lines on a page (nothing is deleted)
    • overwrite -- allow a page or a section of a page to be overwritten
    • create -- allow a new page to be created
    • read -- allow a page to be read
    • delete -- allow a page to be deleted
    • attr -- allow the passwords on a page/group to be changed (used only in chmod)
    • forceread -- override PmWiki 'read' authorization (allow read) (use with care and only enable on pages which are edit-protected)
    • forceedit -- override PmWiki 'edit' authorization (allow edit) (use with care and only enable on pages which are edit-protected)
  • Aliases such as "edit" and "all" and "none" are entirely at your discretion. You can name your aliases Bob & Larry if you want, but normally a more meaningful name is helpful. :-)

  1. Take special note of the security section below. WikiSh is a sharp tool enabling you to get the job done, but you can also "cut your hand" with that same sharp tool if you are not very careful. The warnings in that section must be carefully noted. DO NOT GIVE WRITE PRIVILEGE TO ANYONE YOU DO NOT TRUST! A malicious user given inappropriate write permissions could cause a lot of damage with WikiSh.
  2. (optional) Create a page (recommended: WikiSh.ControlPanel) containing the markup (:wikish_controlpanel:) or (:wikish_command_line:). A page like this is not a necessary part of installation, but it is the recommended way to work with WikiSh. Note that you will need cookbook/WikiShCL.php to be included for that page - if you want this markup always available then you can include this line in your config.php:
   include_once("$FarmD/cookbook/WikiShCL.php");

Security:

SECURITY NOTE on writing: If any write permission (whether wiki pages or text files) is given through the above $Enable... variables then it is strongly recommended to make this recipe available only on pages/groups which are password protected. SecLayer mitigates the threat to some degree, but you still need to be careful.

SECURITY NOTE on text reading: If you allow text file reading ($EnableWikiShTextRead = true;), it is STRONGLY recommended to make this recipe available only on pages/groups which are password protected and/or to be exceedingly careful in your setting of the $wshAuthText variables (in the SecLayer setup). If you are not careful imagine the output of this MX: {(grep DefaultPassword TEXTFILE--local/config.php)}!!!

Textfile access, whether reading or writing, should be used only with great caution. When accessing text files your access is limited only by the SecLayer restrictions and then the OS level file permissions. Unless you really know what you are doing you should not use this feature (textfile reading & writing).

SECURITY NOTE on forceread and forceedit: If you enable forceread or forceedit it must be from a page editable only by trusted authors. Otherwise undesired reads could be obtained or undesired page changes on pages that otherwise would be protected in this way. These "force" modes are very specific tools to be used with great care and only enabled on protected pages. Unless you really know what you are doing you should not use this feature.

The default installation of WikiSh (if no changes are made to any $EnableWikiSh___ variables and no calls are made to slAddAuth() nor slParsePage) does not allow any writing to any page. Normally WikiSh will either be installed in this read-only mode or else giving administrators write authorization but not non-admin editors (or perhaps giving non-admin editors write authorization to just a few specific pages or groups). Giving broader write permissions to non-admin personnel opens you up to potential risk as significant damage can be done in a very short time via WikiSh.

Security Levels

Recognizing the potential risk a scripting language presents in a collaborative environment such as a wiki page, WikiSh implements 4 different layers of security, particularly for write authorization to pages. ALL layers must be passed for a given page write. If a single layer is not passed then the write operation will not be allowed.

  • LAYER 1: PmWiki has a fairly comprehensive set of authorizations built in and these have been expanded by various other recipes. WikiSh always enforces authorizations set by PmWiki (unless an administrator has specifically enabled a "forceread" or "forceedit" authorization for a given page). If a user wouldn't be able to hit the "edit" button and make a change on a given page then they won't be able to write to that page through WikiSh. (This layer of security is implemented through operating system file permissions for text file access.)
  • LAYER 2: ALL writing can be turned on or off with one "switch" which is the $EnableWikiShWritePage variable. If it is set to false then WikiSh will never write to any page. If it is set to true then this layer will be passed. (This layer is implemented for text files both for reading and writing through the $EnableWikiShTextRead and $EnableWikiShTextWrite. The default is for both of these to be false.)
  • LAYER 3: Writing can be broken down into 2 types: either changing existing pages or creating new pages. Authorization to do either of these activities can be turned on or off with the variables $EnableWikiShCreatePage and $EnableWikiShOverwritePage. If they are set to true then the respective authorization is given; if set to false then this layer of security will not be passed. (This layer does not exist for text file access.)
  • LAYER 4: The administrator can configure a fine degree of control using the capabilities of SecLayer (slParsePage(), slAddAuth(), etc.)

See WikiSh Installation & Configuration for more details on configuration and security.

Quick instructions for existing shell scripters

  • Plan on using the control panel for any one-off commands you want to run or test.
    • Really, I'm serious. If you don't use it you will waste a TON of time.
  • When you are going to embed commands in pages, take note of these sections in particular:
    • wikish, and particularly the {(wikish ... source PAGENAME#SECTION ...)}
    • wikish_button for making commands executable via a button on the page
  • Do not assume that the command or option that you need is available. Only a small subset of shell functionality has been implemented. If you find a very useful feature that's not implemented, drop me a line or include a note in the talk page and I'll see what I can do about implementing it.

General Usage:

{(COMMAND OPTION[s] PAGESPEC[s] OUTPUTSPEC)}
  • COMMAND
  • OPTIONS
    • Options can be set anywhere in an MX command via option=value. If you wish to use the particular WikiSh syntax (recommended) then most option settings begin with a hyphen or a double-hyphen. More details can be found in the options section.
  • PAGESPEC
    • From 0 to n pages can be specified. Wildcards (* and ?) are honored. Separate by spaces. If no group is specified the current group is assumed. Precede a pagespec with TEXTFILE-- to refer to a text-file instead of a wiki-page.
    • More details can be found in the Files section below.
  • OUTPUTSPEC
    • Normally the output produced by any MX is placed in exactly the position on the page where the MX command appeared. The output replaces the MX command.
    • However, there are times when you want the output of the command to go somewhere else, particularly into another page. In this case you append an OUTPUTSPEC to the very end of your command.
      • In wikish "lingo" (i.e., the way shell programmers refer to it) this process of sending output somewhere else is referred to as "redirecting the output" or "redirecting standard out" or abbreviated "redirecting stdout". Note that the word "redirect" in this context has nothing to do with your browser opening another page -- it simply means the the output of the command is placed (directed to) somewhere else. Because of this you will later on see options named "stdout" -- this refers to "standard output".
    • OUTPUTSPEC normally takes the form of a right-angle-bracket (">") followed by a pagename (this will replace the entire content of that page with the output of the given command), but this can be modified in various ways. More details can be found in the OUTPUTSPEC section below.

These are the COMMANDS that are currently available:

See in particular {(wikish ...)} as it can (should?) be used as the main interface to the other commands...

  • basename (from a full path, calculate the last filename portion)
  • cat (conCATenate page(s) -- simply list their contents)
  • chmod (CHange MODe -- change attributes of a page or group)
  • cp (CoPy page(s) to another page or to another group)
  • cut (cut specified fields or character ranges)
  • dirname (from a full path, calculate the directory portion)
  • diff (compare 2 pages)
  • echo (simply produce MX output via arguments)
  • fetchmail (get mail from POP3)
  • grep (search for a regular expression in a page(s))
  • head (list first n lines on page(s))
  • ls (LiSt files in various formats)
  • mv (MoVe page(s) to another page or to another group)
  • null (do nothing - /dev/null)
  • od (output special chars as labels)
  • read (read values from a page one-line-at-a-time for looping purposes)
  • rm (remove - will be done by replacing text with "delete" and calling UpdatePage()
    • Note the need to set $EnableWikiShRemove and $EnableWikiShRemoveFully to get this capability.
  • sed (extract sections of a page(s) by pattern or by line number, do search/replace on a page(s))
  • mailx (send email)
    • Note the dependency on WikiMail for this MX
  • set (set variables which will be honored by any WikiSh MX)
  • sort sort lines lexicographically (options to divide into keys, ignore whitespace, dictionary order, etc.)
  • tail (list last n lines on page)
  • test allow various boolean tests (primarily for use with if and while control structures in wikish)
  • uniq (print only unique lines, only duplicate lines, count of duplicates, etc.)
  • wc (count lines, words, or character in a page(s))
  • wikish (generic expression processor - can implement any MXes and allows limited flow control via the following:
    • if BOOL-EXPR; then; EXPR-LIST; else; EXPR-LIST; fi; (nested to your heart's content)
    • while BOOL-EXPR; do; EXPR-LIST; done;
    • for VAR in LIST; do; EXPR-LIST; done;
    • source PAGENAME [args];
    • exit
    • return
  • (xargs implemented by NOT including a hyphen prior to another MX - will be read as another list of files -- see section "Files and Nested Markup Expressions (MX)" below)
  • future: cd (ChDir -- not sure about this from a security standpoint...)
  • future: prettypage (this is not exactly SH [although it has a near relative] but it would be nice to apply fmt=x from pagelist to a result pagelist...)
  • future: allow test to be invoked via [...]

Meta-Commands (non-shell)

  • wikish_button (display a button on a page which, when pressed, will execute a WikiSh MX)
  • wikish_controlpanel (This is (:...:) markup rather than MX - provides a form-based way to run WikiSh commands, keep track of history, save to favorites, etc.)
  • wikish_active (able to activate/deactivate most wikish commands)
  • once (an MX which will return a status of 0 one time, 1 all subsequent invocations. Use in conjunction with wikish_active to make sure a command on a page only gets run once)
  • wikish_form (create a quick form, help with redirects, set fields to vars)

Alternate invocations

You can also invoke an MX using {earlymx(command ...)} if you need to interact with PVs before they are interpolated on that page. This would normally be by use of the {earlymx(set --pv -s VAR = (command ...))} to set {$VAR} to something.

Specifying options

Options are of 2 types: boolean and those accepting a value. You can also specify in different ways depending on whether you are using short- or long- options. (Short are just single-character flags while long may be an entire word.

Examples of option-setting:

  • (sed -n ...) -- sets a boolean 'n' flag to true
  • (grep --ignorecase ...) -- sets a boolean 'ignorecase' flag to true
  • (cut -d: ...) -- sets the 'd' option to a colon
  • (grep --line-prefix:"FILENAME: " ...) -- sets the 'line-prefix' option to 'FILENAME: ' (note the use of a colon rather than equal-sign -- this is necessary because of the way MXes naturally process other options - see below)

You can also specify options simply as option=value.

Generic Options (these work on nearly all commands, at least where they make sense)

As with other options you can either use the preferred WikiSh option setting syntax (as specified below) or you can use the PmWiki syntax of OPTION=VALUE (no prefixed double-hyphen). So you can either run {(echo --timeout:60 "hello, world")} or you can run {(echo timeout=60 "hello, world")} to do the same thing.

  • --timeout:n -- it resets the 30-second timelimit on the host, allowing more or less time before PHP times out; you may get more or less mileage than I have... If you are doing any significant read/write operations and your host isn't too quick then you can save yourself headache by setting this to a higher number. Note also session capabilities within the {(read ...)} MX and the variable ${SECONDSLEFT} for other ways to work around this timeout. Technical note: setting this option calls the PHP function set_time_limit($opt['timeout']);.
  • --file_prefix:"value-to-be-substituted" (see note below) (This value, after substitution, will occur on the line prior to each file/page processed.)
  • --line_prefix:"value-to-be-substituted" (see note below) (This value, after substitution, will occur on each line of output, preceding the output on that line.)
  • --line_suffix:"value-to-be-substituted" (see note below) (This value, after substitution, will occur on each line of output, following the output on that line.)
  • --stdout:pagename (Beware as you can easily overwrite a page!!!) (deprecated synonym for the preferred OUTPUTSPEC)
  • --stdout_type:wiki|text (to write out to a wiki page or a text file -- default is wiki page if not specified (note requirement to set $EnableWikiShTextRead and $EnableWikiShTextWrite if you want to use simple text files - note security risks above. There is no reason to use this option unless you are trying to send output to textfiles.)
  • --stdout_loc:X (Deprecated. use OUTPUTSPEC to set this. If used it should contain a pattern which will be found on the output page and then output will be replaced either before or after that line, depending on the setting of stdout_op below.)
  • --stdout_op:X (Deprecated. use OUTPUTSPEC to set this. If used it should be either < or > to indicate before or after the stdout_loc value.)
  • --tee (Normally if stdout is set (either explicitly or by use of OUTPUTSPEC) then all output goes to the specified page/file and no output will be returned from your MX. If --tee is specified then this will make your standard output go to BOTH the stdout file as well as being returned from your MX. Without this option the stdout will "swallow" all output and no output will be returned from your MX.
  • newline=x This allows you to change the normal newline-separated-line behavior to being separated by something else. (For instance, you could use this within (grep -l ...) to make it a CSV list of files as an input to a {(pagelist ...)})
  • markup=code (Actually currently it can be set to anything and it will result in a simple substitution of all curly-braces and parenthesis with their octal codes to prevent any further markup being processed. If there's other markup "leaking" through let me know and I'll expand the list of substitutions. FUTURE: This can be expanded to other ways of handling markup.
  • --debug:n (sets the debug level. Everything this level and above will be printed, so a lower number is more detailed. 5 turns nearly all debugging off and is the default as long as I remember to set it before I release... ;) 1 turns on all debugging and will significantly slow down processing. )
  • --stderr:LOC (Loc can be either "messages" or "echo" or "/dev/null" ["null" or "nul" synonyms for the latter] [default "echo"]. Error messages will either be outputted into the (:messages:) section ("messages") or echoed at the top of the screen ("echo") or completely suppressed ("/dev/null" and siblings). Note that if you suppress the error messages ... how do I say this ... you won't see them and they may be important!
  • --xargs (says that any piped input to this command should be treated as a list of files to be processed rather than an inline file)
  • --list:X -- This will set an option of which pages should be "stripped out", using the same semantics as the "list=X" option to (:pagelist ...:). Valid values are "normal" and "all" unless your system has explicitly set others. This option, given for a particular command, will override any ${LIST} set elsewhere.
  • --refpage:PAGE -- This will use PAGE as the reference page for PVs which are substituted. (Same as setting ${REFPAGE} but quicker and easier.)
  • --display OR --html -- Either of these options will cause special characters to be displayed in a more standard way on output. But you should not use this on output that will be put into source on a page -- it should be used only for output that will be immediately displayed on a page. (This is particularly important for markup such as [[<<]] which will only work if the characters are in display mode.) (Technical note: use of this option causes PVSE() function to be called on the return value of the MX.)
  • --encrypt -- only available if WikiShCrypt is installed - see section on encryption/decryption
  • --decrypt -- only available if WikiShCrypt is installed - see section on encryption/decryption
  • --passwd -- only available if WikiShCrypt is installed - see section on encryption/decryption

  • Note that in the above options the "value-to-be-substituted" is a simple string, but certain strings will automatically be replaced:
    • PAGENAME (use --file_prefix:"[[PAGENAME]]" (or some other prefix/suffix) if you want a link)
    • PAGETITLE
    • LINENO (only on line-prefix)
    • For instance, you could use the --line-prefix option to put the pagename and line number before each line of output from grep:
{(grep --line_prefix:"PAGENAME: LINENO: " "mysearchstring" MyGroup.*)}
  • Or you could create an entire table in a single command:
{(grep --file_prefix:"|| border=1 width=90%" --line_prefix:"|| LINENO||" --line_suffix:"||" "mysearchstring" MyGroup.MyPage)}

Encrypting and Decrypting Text on Read/Write of Pages & Files

See WikiShCrypt for installation instructions, more details, instructions on how to encrypt, etc.

Specifying Pages & Files and using nested MarkupExpressions (MX)

Pages/Files can be specified with full wildcards. If you do not specify a group (i.e., have a "." in the filename specification somewhere) then the current group will be assumed. (i.e., if you are in the page group.mypage and you ask for * you will get group.*)

Accessing a specific revision (how a page looked at a certain time in the past)

A specific revision of a page can be read by appending @### where ### is a unix timestamp. The core MX {(ftime "%s" WHEN)} is convenient to obtain unix timestamps.

This MX will display the current contents of the page:

{(cat Mygroup.Mypage)}

This MX will display the contents of the page as it existed at the given day/time:

{(cat Mygroup.Mypage@(ftime "%s" "January 1, 2009 09:21am"))}

When in wikish context this would normally be done via backquotes:

{(wikish cat Mygroup.Mypage@`ftime "%s" "January 1, 2009 09:21am"`)}

Being able to read previous revisions allows the capability of seeing the diff between 2 revisions (as opposed to a series of potentially many diffs that one sees usually) or to restore a page to a known good revision (cat Mygroup.Mypage@`ftime "%s" "4 hours ago"` >Mygroup.Mypage), etc.

Using the output of one MX as input to another

You can specify the output of one MX to be the input of another by specifying a single hyphen prior to the inner MX (that tells the outer program that the next argument is a nested MX instead of a filename. For instance, if you wanted the to count the lines between #STARTSECTION and #ENDSECTION you might do the following:

{(wc -l - (sed -n '/#STARTSECTION/,/#ENDSECTION/p' filename))}

That usage (above) means that the output of one MX will be used as if it were the CONTENTS of a file. If you are interested, instead, in having each line of output from an inner MX become a separate argument in the outer MX then simply do NOT prepend the - before the inner MX. For example:

{(wc -l (grep -l "hello" main.*))}

That will find all files which match the regular expression "hello" and then give you a count of the number of lines in each of those files. This should be clearly differentiated from this:

{(wc -l - (grep -l "hello" main.*))}

which will count the number of files which contain a match of the pattern "hello".

Note that backquotes accomplish much the same thing but guarantee the expected order of execution. Putting it in a parenthesized "sub-MX" means it will get executed before any other commands in your MX. Putting it in backquotes means it will get executed after all other preceding commands but before the command it is a part of (in other words, backquotes do what you expect while parentheses will usually mess you up).

Accessing sections of a page

A section within a page (only for reading) can be specified with the expected syntax PAGENAME#SECTION#ENDSECTION and it will only read that section. This works for reading, but not for writing. (The other section syntaxes that (:include ...:) supports are also supported here - I'm reusing the same code.)

File "types"

Specifying any page/file without a filetype prefix automatically assumes the prefix "WIKIFILE--". Thus the default is to access pmwiki pages.

However, there are other filetypes that WikiSh knows about:

PrefixDescriptionExample
WIKIFILE--This is a normal pmwiki pagecat WIKIFILE--Mygroup.Mypage
-or-
cat Mygroup.Mypage
TEXTFILE--Access text files directly (disabled by default for security reasons)cp Mygroup.Mypage TEXTFILE--mydir/mypage.txt
SESSION--Access virtual, in-memory "files" or "pages" (see section on "Virtual Pages" below)echo "Hello, World!" >>SESSION--Greetings
BADFILE--(Internal only - caused by erroneous page specification usually)Do not use.

Virtual pages for increased speed on temporary files

A much quicker and somewhat persistent virtual file system is available through in-memory files stored in your session. Any page created in the "Session" or "Temp" or "Tmp" or "Virtual" groups will automatically be used as an in-memory file (this is configurable through setting the $WikiShSessionPagePat array). You can also specify SESSION--Pagename to force a given page to be an in-memory page. Note that these virtual pages are available only within WikiSh -- you cannot try to browse them with PmWiki. They are significantly faster in reading and writing than disk-based files. Thus they are ideal for an application which will write to a page many times (appending, for instance) -- do 100 writes to an in-memory file and then at the end of your script copy that page into a normal PmWiki page once again. These virtual pages will exist as long as your session is valid. (Using a virtual page instead of writing to disk can save from 300% to 1700% on scripts which are appending single lines to a file many times.) Just don't forget to copy it from virtual space to a "real" page before you are done!

OUTPUTSPEC -- sending the output of an MX command to another page

  • Commands which produce output can have that output redirected to another page via OUTPUTSPEC. The following functionality is supported:
    • >pagename - write the output to that page, overwriting/replacing the existing contents of the page entirely.
      • EXAMPLE: You have a page named Foo.Bar and it contains the words "I think therefore I am". You run the MX command {(echo "Hello, World" >Foo.Bar)}. After it runs the text "I think therefore I am" has been completely replaced and the page Foo.Bar contains only the text "Hello, World".
    • >>pagename - write the output to the page, appending the new output to the end of the existing content of that page. (This is a shortcut for >pagename>_END as documented in the following outputspec.)
      • EXAMPLE: As above but use this MX instead: {(echo "Hello, World" >>Foo.Bar)}. (Note TWO right-angle-brackets instead of 1.) After the MX is executed the contents of Foo.Bar will be "I think therefore I am" on the first line and "Hello, World" on the 2nd and last line. You have appended the text to the end of the page.
    • >pagename>pattern - write the output to the page immediately AFTER the first line matching that (regex) pattern (_END or _BOTTOM are magic patterns matching the last line)
      • NOMENCLATURE: Think of the second right angle bracket as an arrow pointing to the right or AFTER the pattern
      • EXAMPLE: You have the page Foo.Bar which contains 3 lines. Line 1 is "abc", line 2 is "def" and line 3 is "ghi". You execute the MX {(echo "Hello, World" >Foo.Bar>def)}. After the MX executes the line "Hello, World" will be inserted immediately after the line matching the pattern "def". Thus line 1 will be "abc", line 2 will be "def", line 3 will be "Hello, World", and line 4 will be "ghi".
      • EXAMPLE: To repeat the example from the >>PAGE outputspec above: you start with "I think therefore I am" in Foo.Bar. You execute the MX {(echo "Hello, World" >Foo.Bar>_END)}. After the MX is executed the contents of Foo.Bar will be "I think therefore I am" on the first line and "Hello, World" on the 2nd and last line. You have appended the text to the end of the page.
    • >pagename<pattern - write the output to the page immediately BEFORE the first line matching that (regex) pattern (_BEGIN or _TOP are magic patterns matching the first line)
      • NOMENCLATURE: Think of the left angle bracket as an arrow pointing to the left or BEFORE the pattern
      • EXAMPLE: You have the page Foo.Bar which contains 3 lines. Line 1 is "abc", line 2 is "def" and line 3 is "ghi". You execute the MX {(echo "Hello, World" >Foo.Bar<def)}. After the MX executes the line "Hello, World" will be inserted immediately before the line matching the pattern "def". Thus line 1 will be "abc", line 2 will be "Hello, World", line 3 will be "def", and line 4 will be "ghi".
      • EXAMPLE: You have the page Foo.Bar containing the text "I think therefore I am". You execute the MX {(echo "Hello, World" >Foo.Bar<_TOP)}. Your text "Hello, World" has been prepended to that page so line 1 is now "Hello, World" and line 2 is "I think therefore I am".
    • >pagename=/pattern/ - write the output to the page REPLACING the text matched in the (regex) pattern. Note that slashes are required and PREG modifiers (such as m, s, i, etc. -- except e) are accepted.
      • NOMENCLATURE: Think of the equals sign as showing RIGHT where it will go -- overtop of the text matched
      • EXAMPLE: You have the page Foo.Bar which contains "Hello, World". You execute the MX {(echo "Universe" >Foo.Bar=/World/)}. Now the page reads "Hello, Universe".
      • NOTE: This is not line-centric (as opposed to the others). Thus you can span lines in your pattern (with the appropriate pattern or modifier). Also the replacement will occur more than once if the pattern is found more than once.
    • >pagename#section - write the output to the page replacing all text in the given section
    • >pagename$:varname - write the output to the ptv named "varname" on the page specified. Format will be standard text - varname:value. (Note that Toolbox must be installed and available for this capability to be present)
    • >pagename$varname - write the output to the ptv named "varname" on the page specified. Format will be hidden - . (Note that Toolbox must be installed and available for this capability to be present)
    • >pagename$:(varname) - write the output to the ptv named "varname" on the page specified. Format will be hidden - . This is an alternative syntax to above, perhaps more intuitive, but it requires quoting in order to protect the parentheses from the MX engine; for that reason this syntax is deprecated. (Note that Toolbox must be installed and available for this capability to be present)
    • >pagename$::varname - write the output to the ptv named "varname" on the page specified. Format will be definition list - :varname:value. (Note that Toolbox must be installed and available for this capability to be present)
  • The description of OUTPUTSPEC given in the above outline is closely related to how the linux shell redirects output. However, that's not the way pmwiki administrators and editors are used to specifying options in markup expressions or pagelists or whatever. Therefore there is an alternate (although somewhat deprecated by the shell purists <wink>) way of specifying where the output of an MX should be placed:
    • stdout=PAGE
      • EXAMPLE: From the first example above you could use this MX: {(echo stdout=Foo.Bar "Hello, World")}. Since stdout_op and stdout_loc are not set then the output of this command will replace any existing content on the page Foo.Bar.
    • stdout_op=OP where OP can be < or > (if stdout_op is set then stdout_loc is required) indicating before or after the line/location specified via stdout_loc
    • stdout_loc=LOCSPEC where LOCSPEC is a regex pattern to be matched in that file or one of the "magic" patterns _TOP, _BEGIN, _BOTTOM, _END. This identifies a line in relation to which the output will be placed either before or after depending on the value specified in stdout_op.
      • The following examples are all exactly above in the >page, >>page, etc, but now modified to use the pmwiki-friendly syntax...
        • EXAMPLE: As above but use this MX instead: {(echo stdout=Foo.Bar stdout_op=> stdout_loc=_END "Hello, World")}. After the MX is executed the contents of Foo.Bar will be "I think therefore I am" on the first line and "Hello, World" on the 2nd and last line. You have appended the text to the end of the page. (You could have used _BOTTOM instead of _END -- they are synonyms because otherwise nobody can ever remember which one was the right one.)
        • EXAMPLE: You have the page Foo.Bar which contains 3 lines. Line 1 is "abc", line 2 is "def" and line 3 is "ghi". You execute the MX {(echo "Hello, World" stdout=Foo.Bar stdout_op=> stdout_loc=def)}. After the MX is executed the line "Hello, World" will be inserted immediately after the line matching the pattern "def". Thus line 1 will be "abc", line 2 will be "def", line 3 will be "Hello, World", and line 4 will be "ghi". (Notice from the order of arguments in this example and the previous that the specification of options in this 2nd manner can be done anywhere within the MX and it is no longer required to be the last item on the command line.)
        • EXAMPLE: You have the page Foo.Bar which contains 3 lines. Line 1 is "abc", line 2 is "def" and line 3 is "ghi". You execute the MX {(echo "Hello, World" stdout=Foo.Bar stdout_op=> stdout_loc=def)}. After the MX executes the line "Hello, World" will be inserted immediately after the line matching the pattern "def". Thus line 1 will be "abc", line 2 will be "def", line 3 will be "Hello, World", and line 4 will be "ghi".
        • EXAMPLE: You have the page Foo.Bar which contains 3 lines. Line 1 is "abc", line 2 is "def" and line 3 is "ghi". You execute the MX {(echo "Hello, World" stdout=Foo.Bar stdout_op=< stdout_loc=def)}. After the MX executes the line "Hello, World" will be inserted immediately before the line matching the pattern "def". Thus line 1 will be "abc", line 2 will be "Hello, World", line 3 will be "def", and line 4 will be "ghi".
        • EXAMPLE: You have the page Foo.Bar containing the text "I think therefore I am". You execute the MX {(echo "Hello, World" stdout=Foo.Bar stdout_op=< stdout_loc=_TOP)}. Your text "Hello, World" has been prepended to that page so line 1 is now "Hello, World" and line 2 is "I think therefore I am".
  • It should be fairly obvious to even a casual observer the way these 2 methods of specifying output redirection are related. If you put a right-angle-bracket followed by the value of stdout followed by the value of stdout_op followed by the value of stdout_loc then you have the preferred wikish syntax given above. It will be replaced internally by the stdout, stdout_op and stdout_loc, but it sure saves you a lot of typing!!!
  • You should be aware that all testing of WikiSh by the author is done using the first method of specifying redirection. (I'm just too lazy to do all that typing.) Thus it's possible that an error might creep in and be undetected until you generously test the 2nd method and report the error...

Variables

A relatively flexible variable scheme is implemented through the use of the (set ...) command and surrounding variables to be interpolated with ${...}. The idea (this is just like shell although it may seem a little counter-intuitive if you are more used to PHP or something) is that when you want to SET the value of a variable you do NOT use the ${...} (just the "bare" var), but when you want to USE (i.e., interpolate) a variable then you use the surrounding ${...}. So if the variable name is var then you would set it by saying:
{(set var = 2)}
but you would USE it by saying:
{(echo "the value is" ${var})}

NOTE: In bash shell and in versions of WikiSh prior to 2008-03-09 it was valid to specify your variables as $var. THAT IS NO LONGER POSSIBLE. Since there is no capability of single-quotes in MX it was just too difficult to suppress variable interpolation on a dollar sign followed by alphabetics that were not intended to be variables. Thus the design is now changed to REQUIRE surrounding curlies to accomplish variable interpolations. If you are used to $var you must now use ${var}. (Also it interfered with substitution from FmtPagename() since that used the same $var syntax.)
NOTE: There is also a markup rule just for WikiSh variables. This means that any WikiSh variable in any of the various legitimate forms (including modifiers, etc.) can exist in the regular text of your page and they will be substituted as expected. This includes the ${prefix${counter}} form to create variable names from the contents of other variables.

Continuing on...

For example:
{(set i = 8)}
{(echo "my son is " ${i} " years old")}
Continuing on:
{(set i --)}
{(echo "my son who is 1 year younger is" ${i} "years old")}
There is no way an MX function can differentiate between strings surrounded by single-quotes as opposed to double-quotes (or a bare word, for that matter) and so variables are interpolated for everything whenever that pattern is found. The only way to suppress it is to prepend a backslash before the dollar sign:
{(set -v i = 8)} prints "8" as the return value
{(echo "4+4=" ${i})} prints "4+4= 8" as the variable is interpolated
{(echo "dollar sign followed by the letter i: \${i}")} prints "dollar sign followed by the letter i in curlies: ${i}" as the variable is NOT interpolated because of the prepended backslash before the dollar sign.
{(echo "dollar sign followed by the letter i: $i")} prints "dollar sign followed by the letter i: $i" as the variable is NOT interpolated because there are no curlies surrounding the variable name.
Strings are a bit more awkward because of the quoting and the need to limit what characters can be passed to the underlying eval statement. Thus the -s is necessary when using set for strings and no other operations are permitted in that case.

{(set -s sons = Christopher Joshua Jonathan)} {(echo "My 3 sons are named" ${sons})}

While arrays are not formally supported, you can concatenate strings within variable names resulting in a workable implementation of arrays:

for i in 1 2 3 4 5
do
   set a${i} = hello   # Set it
   echo ${a${i}}       # Use it
done

Since "$" and "{" and "}" are not valid characters in a variable name you are guaranteed to get the inner variable replaced first and then the outer variable appears as a valid variable and will be interpolated in turn.

Alternate variable types

  • Session variables can be set using the --session option of {(set ...)} and retrieved using a ~ prefix to the variable name like ${~varname}
    • Session variables remain set (sticky) from one load of a page to another rather than starting from scratch each time. But sessions are lost in certain circumstances and so you don't use this as "safe" storage -- basically it's a convenience...
  • Request variables (GET and POST and COOKIE) can be retrieved using the ! prefix to the variable name like ${!varname}

Credit to the author httpvariables for the markup idea. Imitation is the sincerest form of flattery. Those of you accustomed to httpvariables will appreciate the parallels.

Variables with existing content

These variables can be used without being set and will provide the following functionality:

${PWD}Present Working Directory (PHP: getcwd())
${WIKISH_VERSION}Version of WikiSh
${PMWIKI_VERSION}Version of PMWiki
${RANDOM}A random number between 0 and 32768. You can also set ${RANDOM_MIN} and ${RANDOM_MAX} before accessing this variable to get a random number in a different range.
${SECONDS}The number of seconds (with decimals) since WikiSh was originally initialized. (Pretty accurate idea of how long this PHP session has been executing to test for timeouts.) (You can set ${SECONDS_START} to ${NOW} if you want to restart the timer for some reason. )
${SECONDSLEFT}The number of seconds (with decimals) left before PHP execution timeout causes your script to abort. (You can set ${SECONDSLEFT_START} to adjust when this was started, but it has no effect on the underlying timeout - useful only in testing.)
${NOW}The number of seconds since some specific time/date in the long ago past.
${HOSTIP}The IP address for the host machine
${HOSTNAME}The name of the host machine
${SESSIONID}Session ID
${LPAREN}A left parenthesis (this because MXes are sensitive about putting parens in the content of the MX. This allows you to have a paren without it appearing that way to the MX engine.
${RPAREN}A right parenthesis (this because MXes are sensitive about putting parens in the content of the MX. This allows you to have a paren without it appearing that way to the MX engine.

Variables which affect other behavior (configuration variables) (ordered by likelihood of needing to set them):

These variables can be set like any other variable, but they will have impact on the way WikiSh runs:

${LIST}This will be a default list=X option, determining what files are processed by wildcards and in any commands where files are specified and processed. It uses the same underlying mechanism as the list=X option from (:pagelist list=X ...:), so the same valid values work for each ("all" and "normal" by default, others if they are configured). This is a default value which will override the default set in $SearchPattern['default'] but will in turn be overridden by any --list:X setting for an actual command. (Default: unset)
${DEBUGLEVEL}The actual level of debug statements that will be printed. Note that ${DEBUGLEVEL} is set to ${DEFAULT_DEBUG} after RC file processing (see ${RC_DEBUG} to understand why) and so if you are setting the debug level within your RC file then you should set ${DEFAULT_DEBUG} instead of ${DEBUGLEVEL}. If, however, you are setting the value within your script then you should set ${DEBUGLEVEL}. Note that the --debug:X option sets the ${DEBUGLEVEL} variable. (Default: ${DEFAULT_DEBUG})
${DEFAULT_DEBUG}The default level of debug statements that will be printed. (Under normal circumstances you will want to modify ${DEBUGLEVEL} instead of ${DEFAULT_DEBUG}. This variable (and ${DEBUGLEVEL} is set by the appropriate field in (:wikish_controlpanel:). (Default: 5)
${REFPAGE}The page used as reference when determining the values of PVs. If unset, current page will be used (page from which the script is run, not the page currently being processed). If set to a pagename then that page will be used. So if you want to use, for instance, $Title of page Foo.Bar within a script run from WikiSh.ControlPanel then you would FIRST set ${REFPAGE} to Foo.Bar and then you would use the variable $Title.
${RANDOM_MIN}This will determine the minimum value of any value returned within the ${RANDOM} variable.
${RANDOM_MAX}This will determine the maximum value of any value returned within the ${RANDOM} variable.
${SECONDS_START}Setting this value to ${NOW} will re-initialize the value returned in ${SECONDS}. (Default: the time when WikiSh was initialized)
${ACTIVE}If set to 0 then nearly all WikiSh MXes will be disabled and do nothing. (The only exception are those which are usually used to return a boolean value so you can test to see if you want to turn ${ACTIVE} back on - see wikish_active for more details.) (Default: true)
${HISTFILE}The page in which the history & favorites from the (:wikish_controlpanel:) will be stored. (Default: WikiSh.History)
${HISTSIZE}How many history commands will be available on the (:wikish_controlpanel:) form. Also determines when the "needs purging" message is displayed on that form. This variable is set by the appropriate field in the (:wikish_controlpanel:) form. (Default: 100)
${NOHIST}If set to a non-zero, non-blank value then all history/favorites processing within (:wikish_controlpanel:) will be suppressed. This can result in a savings of several seconds if the history file is large, but it also means that the command has not been saved in history and so cannot be retrieved later. This variable is set by the appropriate field in the (:wikish_controlpanel:) form. (Default: blank - history will be available)
${AUTHOR}The default author that will be used by WikiSh when writing to a page if no value is available in the $Author variable. (Default: WikiSh)
${RC_DEBUG}What level of debug will be used during RC script processing (see RC Pages) (Default: 5)
${PAGEVAR}When pagevars will be processed, in relationship to when WikiSh variables are processed. If you dynamically placed the name of a PV in a WikiSh variable then you would want the PV to be processed AFTER the WikiSh variable and so you would want "post". If you set a PV or a PTV with the value of the name of a WikiSh variable then you would want the PV/PTV to be evaluated first and so you would set this ${PAGEVAR} to "pre". Valid values: "pre", "post", "pre-post" (process PV/PTV both before AND after WikiSh variables are processed), or "" (do not process PV/PTV variables at all while WikiSh is running). Technical Note: "Processing PV" refers to calling FmtPagename() which does more than just variables -- it also includes internationalization and etc. (Default: post)
${SECONDSLEFT_START}Setting this value to ${NOW} will re-initialize the value returned in ${SECONDSLEFT}. Be aware that this will have NO effect on the underlying timeouts and so resetting this value will normally cause the ${SECONDSLEFT} value to be invalid and useful. Recommended to NEVER set this value. (Default: the time when WikiSh was initialized - note that this will "miss" a short period of time that is included in the timeouts since WikiSh is initialized early but there are other things that happen even earlier.)

Variable modifiers

There are several ways to modify the value of variables (or in some cases the variables themselves) while interpolating them:

  • ${#var} will be replaced not with the value of the variable, but with the length of the value of the variable.
  • ${var/srch/repl} will be replaced by the value of the variable after a simple search/replace has been carried out on that value
    • Note that the srch pattern is using globs (wildcard patterns like *.*) rather than fixed strings or regular expressions
  • ${var:offset:length} will be replaced by a substring of the value of the variable. You have an offset (0-based) into the string and the length. (Actually I just call substr() directly with whatever is passed in for x and y, so you have the usual capabilities there including negative numbers, etc.)
  • ${var:-dflt} will be replaced by the value of the variable unless it is unset or blank in which case it will be replaced by the text following the :-
  • ${var:=dflt} will be replaced by the value of the variable unless it is unset or blank in which case it will be replaced by the text following the := AND THAT DEFAULT VALUE WILL BE ASSIGNED TO THE VARIABLE.
  • ${var:+alt} will be replaced by the alternate text if the variable is set and non-blank. If the variable is unset or blank then the replacement will be for an empty string.
  • ${var#pat} delete text matching the pattern off the beginning of the value (smallest possible match) (glob/wildcard patterns)
    • if ${a} is abc.def.ghi ${a#*.} will be def.ghi (abc. deleted)
  • ${var##pat} delete text matching the pattern off the beginning of the value (longest possible match) (glob/wildcard patterns)
    • if ${a} is abc.def.ghi ${a##*.} will be ghi (abc.def. deleted)
  • ${var%pat} delete text matching the pattern off the end of the value (smallest possible match) (glob/wildcard patterns)
    • if ${a} is abc.def.ghi ${a%.*} will be abc.def (.ghi deleted)
  • ${var%%pat} delete text matching the pattern off the end of the value (longest possible match) (glob/wildcard patterns)
    • if ${a} is abc.def.ghi ${a%%.*} will be abc (.def.ghi deleted)
  • ${!prefix*} will return a list of all variable names which begin with that prefix
    • if you have 3 variables ${fooa}, ${fooabc} and ${fooxyz} and you use ${!foo*} then you will get "fooa fooabc fooxyz". If you put these in a variable (via a for loop or similar construct) named i, for instance, then you could get to the value of the variable like this ${${i}}
    • Example follows where the above 3 variables exist and you want to print their values
    for i in ${!foo*}
    do
       echo "The variable ${i} contains the value ${${i}}"
    done
    

Return Status

Each command returns text which goes in place of the MX itself, but in addition to that type of return value there is also an "ERROR STATUS" or "STATUS" which is set and returned by each command. Basically if the value is 0 then everything worked fine. If the value is non-zero then something unexpected happened, or we didn't find what we were looking for, or something like that.

The commands (true) and (false) do nothing except to set this return status to 0 and 1, respectively.

You can access the value of this return status using the variable ${STATUS}.

You can also use it in commands which are testing for a boolean value. For instance, in the main WikiSh command you have commands like (if ... then ... else ...) and the initial condition is determined based on the ${STATUS} value.

{(wikish if true; then; echo YES; else; echo NO; fi)}

Note that ${STATUS} is set by each successive command, so the current value is based on whatever the LAST command was that completed.

Functions

Various commands within WikiSh (or even other non-WikiSh MXes) can be combined within a single "function" which then becomes its own MX.

This capability is significant in order to remain within the "author-friendly" philosophy of PmWiki. Relatively complicated WikiSh scripts can be placed in seldom accessed pages and simple MX functions can be executed in pages where non-technical authors roam.

There are 2 ways to define a function:

  • Within a single-line MX (technically it's not an MX, but if it looks like an MX and it smells like an MX...)

{(function abc echo "Hello, World"; if true; then; echo "Logic still works"; else; echo "Uh, oh, we're really in trouble now."; fi)}

  • In a piece of source'd script

{(wikish source WikiSh.Functions#abc)}

  • If you use the second option (preferred) then you would need to have this section in WikiSh.Functions:
[[#abc]]
function abc()
{
   echo "Hello, World!"
   if true
   then
      echo "Whew! Logic still works"
   else
      echo "Uh, oh, we're really in trouble now."
   fi
}

In both cases, after the function has been defined you can run it just as you would run any other MX:

  • {(abc)}
  • {(wikish abc)}
  • [in the control panel:] abc

It is possible to pass arguments to functions using positional parameters. Your function will see them as ${1}, ${2}, etc. You can see ALL positional parameters using ${*} or ${@}.

[[#greet]]
function greet()
{
   echo "Hello, ${1}!"
   if test ${2} == 'guy'
   then
      echo "Looks like it's boys' night out.  BRAAAP!"
   else
      echo "Ladies present! Guys, cut the belching!"
   fi
}

Then when you call it you can pass arguments (in this case a name and then a gender):

{(wikish source WikiSh.Functions#greet; greet Sam guy; greet Janet gal)}

Hopefully the result would be this:

Hello, Sam!
Looks like it's boys' night out.  BRAAAP!
Hello, Janet!
Ladies present! Guys, cut the belching!

The return status of a function is the return status of the last command which is run within that function.

The reserved word "return" will operate as expected, providing a means for early termination of a function and returning to the calling context.

The reserved word "exit" terminates the entire script, regardless of whether it is within a function or not.

Note that any functions defined in the page WikiSh.Profile or WikiSh.WikiShRC will be automatically defined whenever WikiSh is active as long as at least one wikish command has been run beforehand (this is most easily accomplished simply by running the function within a {(wikish ...)} context).

Thus if I put this in WikiSh.WikiShRC:

function abc
{
   echo "I know my alphabet. Won't you sing with me?"
}

Then on any page within my site I can run this function by placing this source code on the page:

{(wikish abc)}

Or, if I have already run another wikish MX then I can access it directly:

{(echo foo)}
{(abc)}

In addition to WikiSh.WikiShRC and WikiSh.Profile you can also place group-specific source in WikiSh.GROUP-GroupRC and page-specific function definitions (or variables being set or whatever) in WikiSh.GROUP-PAGE where GROUP and PAGE are the names of the group and page of the page which needs the function definition, etc. All of these pages "understand" at least 3 pmwiki rules: if conditions, include directives, and comment directives. This allows you to include certain pages (or sections) and etc.

  • If you wish to configure these startup pages you can set 2 variables:
    • $WikiShRCPages (an array containing the list of pages, perhaps defined using PVs)
      • DEFAULT: array('WikiSh.WikiShRC', 'WikiSh.Profile', 'WikiSh.Cookbook-GroupRC', 'WikiSh.Cookbook-WikiSh')
    • $WikiShRCRules (an array containing the list of rules which will be executed over these pages before they are executed by WikiSh)
      • DEFAULT: array('include', 'comment', 'if')

These startup pages are not limited to function definitions. You could set a variable (${LIST}, for example) or create a temporary page or whatever. It's just that function definitions are probably the most common use of these startup pages. You can place any complex WikiSh code in the startup page (think of it as a per-page configuration) and then in your page all you see is {(wikish function_name argument)}.

Detailed Listing of Each Command

basename (calculate just the pagename/filename portion of a fully qualified pathname/group.pagename)

  • {(basename FILE)}
  • Options
    • --raw - pass the argument directly to PHP dirname() function (i.e., working with directories and filenames rather than groups and pages)
    • Note that if TEXTFILE-- is prefixed to the pathname then --raw is automatically assumed
  • Examples:
    • {(basename Mygroup.Mypage)}
      • returns "Mypage"
    • {(basename TEXTFILE--path/to/file)}
      • returns "file"

cat (conCATenate pages)

  • {(cat OPTIONS FILES)}
    • Options are only the generic ones
    • This simply returns the files or nested MXes which are its arguments. It's nice for pulling together results of several commands.
    • EXAMPLE: (cat - (echo || border=1) - (sed --line-prefix:"||PAGENAME||" -n '1,10p' group.file?))
      • It's not quite that simple to create a table, but almost... We're still missing the ending || -- if you want that you have to make it a little more complicated as below:
      • EXAMPLE: (cat - (echo || border=1) - (sed --line-prefix:"||PAGENAME||" -n -e '1,10p' -e 's/$/||/' group.file?))

chmod (CHange MODe - change passwords on groups/pages)

  • {(chmod OPTIONS FILES)}
    • Options
      • -p - this will apply to PAGES (this is the default if neither -p nor -g are specified)
      • -g - this will apply passwords to GROUPS
      • --read:PASS - set the "read" password to PASS
      • --edit:PASS - set the "edit" password to PASS
      • --attr:PASS - set the "attr" password to PASS
      • --upload:PASS - set the "attr" password to PASS
      • -a - the passwords will be *added* to existing passwords for this group/page rather than replacing existing passwords. (This behavior is unique to chmod and cannot be done using the usual form-based interface.)
    • This MX sets a password for the specified pages or groups. If neither -p nor -g are specified then -p is assumed. If -g is specified then the assumed -p is no longer assumed. If you want to change both page and group passwords (why?!) then you would need to specify both -p and -g.
    • Note that as in all WikiSh MXes you are responsible to obtain the necessary pmwiki authorizations prior to running the command. In this case you must have attr authorization on any pages or group.GroupAttributes pages that you are attempting to change.
    • EXAMPLE: {(chmod --edit:mysecret Main.HomePage Test.OtherPage GroupA.*)}
  • Note that if you use authorizations besides read, edit, attr, and upload you can change the configuration variable $wshPmAuthList to something besides array('read', 'edit', 'attr', 'upload'). You could also use this to limit which passwords people were allowed to change via chmod.

cp (CoPy pages)

  • {(cp OPTIONS PAGES ... DESTINATION)}
    • Options
      • --trial - don't actually copy - just tell me what you would have done
      • -q - be quiet about it. Do the copy, but don't tell me what you've done.
    • PAGES - as usual with wildcards or specifying multiple files
    • DESTINATION - if multiple source pages were specified you must specify either a group name (copy those pages to the other group) or a directory for text files (TEXTFILE--dirname)
    • EXAMPLE:
      • (cp groupa.page1 groupb.page2)
        • This will copy groupa.page1 to groupb.page2
      • (cp groupa.* groupb)
        • This will copy all pages in groupa into groupb
      • (cp groupa.* TEXTFILE--mydir)
        • This will place text files containing the source in a directory relative to the main pmwiki directory called mydir

cut (cut characters or fields from each line of input)

  • {(cut OPTIONS PAGES ...)}
    • Options
      • -dx - specify a delimiter of x for splitting into fields (default TAB)
      • -fLIST - specify a list of fields to print (fields counted from 1)
      • -cLIST - specify a list of characters to print (characters counted from 1)
      • --ofs:z - specify an output field separator (if not specified it defaults to x from -dx)
    • LIST
      • x - just that character or field
      • x-y - characters or fields from x to y
      • x- - characters or fields from x to the end
      • x,y - characters or fields x and y
      • x-y,z - characters or fields from x to y and z
      • etc.
    • PAGES - as usual with wildcards or specifying multiple files
    • EXAMPLE:
      • (wikish echo "abc\tdef\tghi\na\tb\tc" | cut -f2)
        • This will print def and b (field 2 of each line)
      • (wikish echo "abc def ghi\na b c" | cut -d' ' -f2)
        • This will print def and b (field 2 of each line, fields now separated by a space)
      • (wikish echo "abc def ghi\na b c" | cut -d' ' -f1-2)
        • This will print "abc def" and "a b" (fields 1 & 2 of each line)
      • (wikish echo "abc def ghi\na b c" | cut -c1-2,5)
        • This will print "abd" and "a c" (characters 1 to 2 and 5 of each line)

diff (compare files)

  • (diff OPTIONS PAGE1 PAGE2)
    • Options
      • -q - don't show changes, just return ${STATUS} for use in while loops or if conditionals (true if files are identical)
    • EXAMPLE:
      • (diff groupa.page1 groupb.page2)
        • This will compare groupa.page1 to groupb.page2 and print off a diff output
      • (wikish if diff -q groupa.page1 groupb.page1; then; echo "They are the same"; else; echo "They are different"; fi)
        • This will print a quick messages indicating whether the 2 pages are the same or different

dirname (calculate just the directory/group portion of a fully qualified pathname/group.pagename)

  • {(dirname OPTIONS FILE)}
  • Options
    • --raw - pass the argument directly to PHP dirname() function (i.e., working with directories and filenames rather than groups and pages)
    • Note that if TEXTFILE-- is prefixed to the pathname then --raw is automatically assumed
  • Examples:
    • {(dirname Mygroup.Mypage)}
      • returns "Mygroup"
    • {(dirname TEXTFILE--path/to/file)}
      • returns "path/to"

echo

  • (echo OPTIONS ARGS)
    • Simply echo a string as the return value of the MX
  • Options
    • -w = go ahead and expand wildcards (normally this is suppressed to avoid accidental expansion since quotes are non-available within the MX)
    • --raw = do a PHP echo instead of a usual echo. This is primarily for debugging and is NOT enabled by default - you must explicitly set $EnableWikiShRawEcho to true for it to work.

exit [#]

The reserved word "exit" is not available except within the {(wikish ...)} context. It allows for early termination of the entire instance of wikish. The optional number will set the ${STATUS} upon termination (default 0).

fetchmail

  • (fetchmail OPTIONS)
    • Get a message from a POP3 server, delete a message from the server, close the connection, etc.
    • ''Note the dependency on WikiMail version 2008-08-08 or later
  • Options
    • --profile:PROFNAME = use the wikimail profile named "PROFNAME" (must be set up elsewhere)
    • --from:varname = put the value of the address from whom the email is sent in a variable named ${varname}
    • --subject:varname = put the value of the subject in a variable named ${varname}
    • --body:varname = put the value of hte body of the message in a variable named ${varname}
    • (other options such as --to:varname, etc are possible as well)
    • --stay = don't continue to the next message
    • --delete = delete the message (default if not set unless --stay is specified in which case it must be specified to actually delete it)
    • --nodelete = don't delete the message
    • --close = close the connection to the POP3 server (if you don't do this nothing will be deleted)
  • Example
while fetchmail --profile:foo --from:from --subject:subject --body:body
do
   echo "====\n'''FROM: ${from}'''\n'''SUBJECT: ${subject}'''\n${body}\n"
done
fetchmail --profile:foo --close

grep (General Regular Expression Processor)

Search for a regular expression in a file.

  • (grep OPTIONS PATTERN FILES)
  • Options:
    • -i = ignorecase (--ic or --ignorecase are synonyms) (default is to be case sensitive)
    • -l = list ONLY filenames of files that match, not the lines themselves
    • -n = include line numbers (source-based) (--line-number is a synonym)
    • -q = don't return any lines -- just give me an appropriate return status (${STATUS}) indicating whether you found something or not (for use in if ... or while ...)
    • -v = reverse the sense of the search (return lines NOT matching regex)
    • -F = fixed string search rather than regular expressions. Allows you to not worry about escaping "magic" characters when you aren't in need of regex capabilities.
    • --targets = search in list of target links (1 target per line) rather than in text of page
  • Note: A combination of -v and -l currently acts like a -L in the "real" grep. This will be fixed to be consistent with real grep syntax in the future.

head

  • (head OPTIONS FILES)
  • Options:
    • -n # - (# is a number) specify how many lines of text to return from the start of a page
  • Note that this is functionally equivalent to (sed -n '1,np' FILES) where the second n (inside the single quotes) is a number for n

ls (LiSt pages)

  • (ls OPTIONS PAGEPATTERN ...)
    • Options
      • --os - determine all file attributes from an OS perspective instead of from a page perspective
        • size will be file size instead of number of characters on the page
        • dates will be according to file dates instead of the timestamps within the page
        • user/owner will be the UID of the owner rather than the most recent author
      • -l - give a long listing, including date, size, owner/author, etc.
      • -t - sort by date/time instead of by pagename
      • -S - sort by size instead of by pagename
      • --nosort - instead of the default sort by pagename, do no special sorting
      • -c - all dates will be create date (ctime) instead of modified date
      • -m - all dates will be modified date (time) (this is the default behavior)
      • -a - all pages will be listed - equivalent to --list:all or list=all option
      • -g - only return the group names - the period and the name will be stripped
      • -G - only return a unique list of the group names
      • -p - only return the page names - the groupname and the period will be stripped
      • -P - only return a unique list of the page names
      • --group:GROUP - identical to group=GROUP from pagelist context
      • --name:NAME - identical to name=NAME from pagelist context
    • Arguments
      • any combination of pages and files (files preceded by TEXTFILE--) can be given, using wildcards as desired
        • Any files (preceded by TEXTFILE--) will obviously always be listed with a --OS option assumed as there is no page information available
      • --list:X and list=X options (or setting of ${LIST} variable) will be honored
      • if no arguments are specified then "*.*" (all pages) is assumed
    • Simply give a list of files
    • EXAMPLE: ls -l Main.W* TEXTFILE--pmwiki.php TEXTFILE--cookbook/WikiSh.php
      • Note that the textfiles are owned by the root user and therefore have "0" as the UID
       Joe User    2643 Apr 08 08:31 Main.WikiSandbox
       Joe User     494 Jan 30 21:11 Main.WikiSh
              0  176962 Apr 14 09:13 cookbook/WikiSh.php
              0   80588 Apr 13 22:27 pmwiki.php
  • EXAMPLE: ls -l --os Main.W
    • Note the difference in sizes and UID since it is from an OS level instead of from a page level
              0   40844 Apr 08 08:31 Main.WikiSandbox
              0    1617 Jan 30 21:11 Main.WikiSh

mailx (send an email)

Shell note: options for this mailx MX are largely unrelated to options for the linux mailx program. Particularly note that while stdin is accepted, unflagged arguments will be seen as pagenames rather than as addresses.
Note that mailx is dependent on WikiMail recipe. It must be installed and configured appropriately to use this MX.
Note that mailx is disabled by default. You must explicitly set $EnableWikiShMailx = true; in your config.php if you wish to use this command.

  • Usage: (mailx OPTIONS PAGE)
    • Options:
      • -q - Do it quietly (don't report success/failure in the return value of the MX)
      • --to:address- send the email to this address
      • --cc:address- send the email to this address as cc
      • --bcc:address- send the email to this address as bcc
      • --subject:subj specify the subject "subj" for this email
      • --from:address specify the "from" address for this email
      • --html send the email in html format (runs all markup rules)
    • synonyms for above options
      • -t address = --to
      • -c address = --cc
      • -b address = --bcc
      • -s subject = --subject
      • -f address = --from
  • Examples:
    • {(wikish echo "This is a message\nwith 2 lines" | mailx -s "nice subject" -t myaddr@mydomain.com)}
    • {(wikish echo "(:include MyGroup.MyPage:)" | mailx -s "nice subject" -t myaddr@mydomain.com)}
    • {(wikish mailx -s "nice subject" -t myaddr@mydomain.com MyGroup.MyPage MyGroup.MyPage2)}

mv (MoVe pages)

  • USAGE: (mv OPTIONS PAGE1 PAGE2)
  • USAGE: (mv OPTIONS PAGE1 [...] GROUP)
    • Permissions
      • Requires $EnableWikiShRemove
      • Requires $EnableWikiShRemoveFully for -f option
    • NOTE: Page history will be lost. It is NOT transferred with the moved file.
    • NOTE: Links to this page from other pages are note modified.
    • Options
      • -f DANGER - this fully deletes the source rather than just renaming it to page-del,timestamp
      • --trial - don't actually copy nor remove the files -- just produce output indicating what would have been done
    • PAGES - as usual with wildcards or specifying multiple files
    • GROUP - if multiple source pages were specified you must specify either a group name (copy those pages to the other group) or a directory for text files (TEXTFILE--dirname)
    • EXAMPLE:
      • (mv groupa.page1 groupb.page2)
        • This will rename groupa.page1 to groupb.page2
      • (mv groupa.* groupb)
        • This will move all pages in groupa into groupb
      • (mv groupa.* TEXTFILE--mydir)
        • This will place text files containing the source in a directory relative to the main pmwiki directory called mydir
  • STATUS: alpha - currently just calls the {(cp ...)} and then checks the ${STATUS} and if successful then calls the {(rm ...)}. The same set of options (whatever you specify) are passed to both.
  • Link solution: If you are renaming a page and want to fix the links to that page in other pages this script should do the job (beware timeouts - you may want to check a set of pages at a time rather than all pages with the *.*):
    • For a single page:
      mv GroupA.Page1 GroupB.Page2
      sed -i 's|\bGroupA[/.]\)?Page1\b|GroupB.Page2|g' (grep -l --targets '^GroupA.Page1$' *.*)
      
    • For an entire group:
      for i in GroupA.*
      do
         set -s page = `echo ${i} | sed 's/^GroupA.//'`
         mv ${i} GroupB
         sed -i 's|\bGroupA[/.]\)?{page}\b|GroupB.${page}|g' (grep -l --targets '^GroupA.${page}$' *.*)
      done
      

null

  • (null OPTIONS ARGS)
    • Ummm. Do nothing. Ignore options and arguments and just swallow any output to this point.
    • Think /dev/null. Think "bit bucket."
    • May be replaced by stdout=null or >null or >/dev/null in the future.

od

  • (od OPTIONS ARGS)
    • Stands for "Octal Dump"
    • Kind of like "echo" except any non-alphanumerics will be output as a label so that they won't be interpreted by HTML.
    • Primarily useful for debugging

read

  • (read [OPTIONS] VAR1 [VAR2 [...]] <PAGENAME) | (read --stdin:PAGENAME VAR1 [VAR2 [...]])
    • Read in a line at a time (once per invocation) from PAGENAME. Assign each word (delimited by IFS) to subsequent variables VAR1, VAR2, etc. Automatically move on to the next line unless --nonext is set.
      • Do note that if more than a single "read" command will occur within a page/script then you need to understand and use either the --set or the --clear options
  • Options:
    • --stdin:PAGENAME (REQUIRED! specifies where the lines will be read from) (can also be set by <PAGE syntax)
    • --IFS:regex (specify what value to split the lines by, default \s for whitespace)
    • --set:n In certain cases you may need more than 1 read going on (nested loops, for instance). In this case you will need to specify a different "set" to hold the index pointing to the current line, etc. (default is 0 when not explicitly specified)
    • --nonext don't step to the next line after reading this line - stay on the same line ("No Next")
    • --initialize read in the input file and get things ready for subsequent use of this set (this is the default if the current set is uninitialized)
    • --saveset save the current set in your session (usually because of PHP timeouts) (--save is a synonym) (no other options are honored if this one is specified)
    • --restoreset restore the current set from your session (no error if no state exists) (--restore is a synonym) (no other options are honored if this one is specified)
    • --clearset clear the current set in the session (--clear is NOT a synonym - see below!) (no other options are honored if this one is specified)
    • --status output the index to the current line - no variables are set, no increment, etc. (useful for saving a spot and returning via goto or for a progress message indicating how many lines have been processed)
    • --linecount output the number of lines in the current set - no variables are set, no increment, etc. (useful for going to the last line via goto or for a progress message indicating how far you have to go)
    • --next go to the next line (all --next alternatives which follow are relative to the current line) (this option is a shortcut for --next:1 and is the default course of action)
    • --next:n go to the nth next line
    • --next:pat go to the next line matching pat
    • --next:/pat/ go to the next line matching regex pat
    • --prev go to the previous line (all --prev alternatives which follow are relative to the current line) (shortcut for --prev:1)
    • --prev:n go to the nth previous line
    • --prev:pat go to the previous line matching pat
    • --prev:/pat/ go to the previous line matching regex pat
    • --goto:n go to the specified line (0-based, absolute addressing)
    • --rewind go to the first line (same as --goto:0)
    • --clear clear the current set (this is done implicitly when you reach the end of the file if --noclear is not specified) (if --clear is explicitly set then no other options/actions will occur - just the clear)
    • --noclear even if you get to the end of the file (or beginning of file with --prev) don't clear the set
    • --stdout:page if --stdout is specified as an option (or set via OUTPUTSPEC as in >PAGE or >>PAGE or etc) then variables are *not* loaded and the whole line which would have been split into variables is sent to stdout
  • Examples:
    • {(wikish while read var1 var2 <MyPage; do; echo "Word1 is ${var1}, word2 is ${var2}"; done)}
      • For each iteration of the while loop it will read the next line from MyPage and assign the first word to ${var1} and the remainder of the line to ${var2}. The body of the loop simply prints off these values.
  • Saving/Restoring in your session to deal with timeouts
    • The ability to save/clear/restore the current set in your session is provided because PHP has an annoying timeout of 30 seconds of execution time. For most PHP applications this is great, but if you are doing a search/replace of every page in your wiki (or some other time-consuming operation) then 30 seconds is probably not nearly enough time. The read command provides the capability to save where you are and pick up again later.
    • Note that the state information is saved on a set-by-set basis and so if you are using multiple sets you will need to save each one individually.
    • Note that the state information is saved in your session ($_SESSION['WIKISH_READ'] to be specific). Thus anything which causes you to lose your session or start a new session will cause this state information to be lost.
    • If multiple options are specified they are executed in this order:
      • (initialize)
      • restore
      • clear
      • (movement options)
      • save
    • This allows you to restore/clear session or restore/move/save in a single call. Note particularly that this allows you to interactively process through a file using read in the Control Panel by specifying --restore and --save on each call.
    • The typical way to run this would be like this (note comment below regarding --clearset!):
read --restoreset  # this will not cause an error if there is nothing to restore
if test -z `read --status`
then
   echo "Generating list of files to work off of..."
   ls --list:normal >Tmp
fi
while read fn <Tmp
do
   echo "Processing page ${fn}"
(other processing commands here -- what you really want to do with ${fn})
   if test ${SECONDSLEFT} -lt 7   # allow 7 seconds for initial compile time plus time to process next file
   then
      read --saveset
      echo "You will need to reinvoke this script to complete.  You have finished" `read --status` "files out of" `read --linecount`
      exit
   fi
done

read --clearset

  • Note that on-screen output will *not* be saved from one invocation to the next. Thus if you are wanting to save something as a result of your processing you will need to save it in a (perhaps temporary) file via >PAGE or >>PAGE or other OUTPUTSPEC alternatives.
  • Note it would probably be wise to run {(read --clearset)} BEFORE the above script just in case you have a prior readset hanging around in your session...
  • It is recommended to run the above script from your control panel, but you may well want to "Suppress History/Favorites" (option towards the bottom of the page) while you are re-invoking the script -- that way you will maximize the time spent on processing files and minimize the time spent processing history file...

rm

  • (rm OPTIONS FILES)
  • Permissions
    • Requires $EnableWikiShRemove
    • Requires $EnableWikiShRemoveFully for -f options
  • Options:
    • -f - completely delete the files (no Group.Page,del-1234567 "backup" will remain)
    • --trial - don't actually delete any files -- just produce the usual output to tell me what you would have done
  • Note that without the -f option the file will be deleted by writing the word "delete" into the page content and then calling UpdatePage(). With the -f option that process will be done first (to hopefully clean up any page caching and etc.) and then the resulting FILE,delTIMESTAMP file will be deleted.
  • Note that the -f is still causing problems...
  • Note that if $EnableWikiShRemove is not set to true then (rm ...) will do NOTHING
  • Note that $EnableWikiShRemoveFully must be set in order for the -f flag to be noticed and acted upon.

sed (StreamEDitor)

  • (sed OPTIONS EDIT-ACTION FILES)

OR (to specify multiple edit-actions)

  • (sed OPTIONS -e EDIT-ACTION [-e EDIT-ACTION ...] FILES)

Currently only 2 edit-actions are implemented:

  • p=print. FORM: address1,address2p
    • (sed -n 'address1,address2p' file ...)
    • EXAMPLE: (sed -n '2,5p' group.file)
    • EXAMPLE: (sed -n '/pat1/,/pat2/p' group.f*)
    • each address can be either a line number (\$ = last line of file - note the \ before the $-sign) or a /regular expression/ (intermix them freely)
    • the "-n" flag is necessary to prevent automatic printing of ALL lines
  • s=substitute. FORM: s/searchfor/replace/flags
    • (sed 's/searchfor/replace/g' FILES)
    • flags:
      • g = global search/replace -- if you don't specify this then only the first occurrence on a line is replaced
      • F = fixed string search/replace -- no regular expressions so no "magic" characters - no need to escape the world if you aren't using regexes.
        • Basically uses str_replace() instead of preg_replace(). Since str_replace() has no way to limit the number of replacements a flag 'F' requires a flag 'g' and will generate an error message without it.
      • i = case Insensitive search
      • A = anchored to the beginning (no need for ^)
      • D = anchored to the end (no need for $)
      • U = Ungreedy regex
      • X = implements PCRE_EXTRA (see PCRE pattern modifiers)
      • u = pattern strings treated as UTF-8
        • Note that WikiSh makes no claims on being UTF-8-tested -- I just saw the feature was available here and allowed it to pass through to preg_replace -- use at your own risk.
    • search/replace specification
      • s is required as the command
      • the forward slashes delimiting the search and the replace can be other characters as well - the following are all equivalent:
        • s/search/replace/g
        • s|search|replace|g
        • s^search^replace^g
    • EXAMPLE: (sed 's/myname/yourname/g' Profiles.*)
  • OPTIONS
    • -i=in-place editing. This gives you capability of doing a search-and-replace on a file or set of files and write the results back to the files. (Note that no write will be made if no change is made. If you are doing search/replace and you don't find the search string then the page will not be written.)
      • -v=verbose. If you are doing in-place editing and specify -v then an output of "Updated: PAGENAME" will be generated when a change is made to a file.
    • -n=don't print ALL lines but only those specified by the "p" command. (this is necessary for the "p" command -- otherwise you say 1,2p but since all lines are printed there's no differentiation between 1-2 that you asked to be printed and all the rest)
  • FUTURE edit-commands: a=append, d=delete

You can specify multiple edit-actions with multiple uses of the -e flag:

  • {(sed -e '1,2p' -e '/#STARTSECTION/,/#ENDSECTION/p' -e 's/not-nice-word/nice-word/g' mygroup.myfile)}

set (set variables for use within WikiSh MX)

  • Usage: (set OPTIONS VAR ASSIGN-OP EXRESSION)
    • Options:
      • -v - Return the value assigned (be Verbose)
      • -s - Process expression as a String. Simply place quotes around arguments after the assignment operator. Required for string assignment.
      • --pv - Set it as a page variable instead of a WikiSh Variable. (Normally requires invoking via {earlymx(...)} instead of the standard {(...)} if you are going to use the PV.
      • --session - Set it as a SESSION variable instead of a WikiSh Variable. This makes it "stick" between loading of different pages.
      • --form - Set the value in $InitValues[] instead of a WikiSh Variable. This enables initializing forms.
    • VAR - any character sequence that will match "/\w+/" (note that you do NOT prepend the variable name with a dollar-sign. Use i instead of ${i}. See section on "Variables" above.)
    • ASSIGN-OP - normally simply =, but it can be anything recognized by php (=, +=, -=, .=, etc.)
    • EXPR - a valid PHP expression (can contain PV, PTV, WikiSh variables, etc.)
  • NOTE: There's not much error-checking going on here -- it's your responsibility to make sure you get a valid expression going... I do enforce certain valid functions and don't allow others to be used (too big a security risk otherwise).
  • Examples:
    • {(set i ++)} - increment ${i} by 1
    • {(set i += 1)} - increment ${i} by 1
    • {(set i = (wc -l main.pagename))} - set ${i} to the number of lines in the page main.pagename
  • See section above on variables for more details

sleep

  • (sleep n)

This makes your script "go to sleep" for n seconds.

  • Options:
    • no options at this point
  • Arguments:
    • n specify the number of seconds to sleep (n is a number [integer or float] and is a required argument)
  • Do note that this "sleep" delay occurs WHILE the page is rendering, before it is displayed. Thus this command is not useful to do some action after a certain period of time that the page has been displayed. For that you will need to look at AutomaticPageRefresh or Javascript with setTimeout or Interval.

sort

  • (sort OPTIONS FILES OUTPUTSPEC)
  • Options:
    • -u (unique sort - return a list of unique lines as defined by -k keys)
    • -d or --dictionary-order (ignore all characters except letters, numbers, and whitespace when sorting)
    • -b or --ignore-leading-blanks (ignore leading whitespace when sorting)
    • -f or --ignore-case (ignore case when sorting)
    • -r or --reverse (sort in descending order)
    • -t X or --field-separator:X (use X as a field separator (X is a single character)
    • -k POS[,POS][OPTS] or --key:POS,POS[OPTS] (specify which key (separated by field-separator above and 1-based) to sort on)
      • This option is repeatable as often as you want
      • Each POS takes the form FIELD[.CHAR]
      • The first POS indicates a starting point and the second POS indicates an ending point.
      • If no second POS is given then the sort will consider the remainder of the line significant in the sort
      • Any of the d/b/f can be specified on a per-key basis by appending it to the end of the key specifier
  • NON-SHELL Options:
    • --count (print a prefix of the count of duplicates of that line)
    • --duplicated (print only lines which have a duplicate)
    • --only-unique (print only lines which are unique - do not print lines that are duplicated)
    • --no-sort (don't do the sort -- do everything else but don't actually sort)
  • Examples:
    • {(sort -k 2,2 Main.*)} - sort all lines in all pages in Main. Use whitespace as the field separator and sort only on the 2nd field
    • {(sort -t: -k 2,3d Site.Passwd)} - sort lines in Site.Passwd. Use colon to separate fields and sort on fields 2 and 3 using dictionary sort order.

tail

  • (tail OPTIONS FILES)
  • Options:
    • -n # - (# is a number) specify how many lines of text to return from the end of a page
  • Note that this is functionally equivalent to (sed -n '-n,$p' FILES) where the 2nd n (inside the single quotes) is a number for n.

test

  • (test ARGS/OPTS)
  • Options & Arguments:
    • FILE OPTIONS:
      • -d FILE (is a directory/group)
      • -e FILE (exists)
      • -f FILE (exists)
      • -g GROUP (does it exist as a group) (non-shell)
      • -r FILE (is readable)
      • -s FILE (file has something - is non-zero length)
      • -w FILE (is writable)
      • FILE1 -nt FILE2 is file1 newer than file2 by modification date
      • FILE1 -ot FILE2 is file1 older than file2 by modification date
    • STRING OPTIONS
      • -z STRING (true if string is zero length)
      • -n STRING (true if string is non-zero length)
      • string1 == string2
      • string1 != string2
      • string1 ~= /regex/
      • string1 < string2
      • string1 > string2
    • INTEGER OPTIONS
      • INT1 -eq INT2
      • INT1 -ne INT2
      • INT1 -lt INT2
      • INT1 -le INT2
      • INT1 -gt INT2
      • INT1 -ge INT2
    • PMWIKI CONDITIONALS
      • --pmwiki <any arguments you would have used in (:if ...:)>

uniq

  • (uniq OPTIONS FILES OUTPUTSPEC)
    • NOTE: uniq does NOT follow shell usage for specifying fields and characters. Instead it uses the -kPOS1,POS2 syntax from sort. The -k syntax is more flexible and powerful.
    • NOTE: uniq identifies duplicates if they are next to each other. If the input is unsorted then duplicates will not be identified.
  • Options:
    • -d (only print duplicate lines)
    • -c (print the count of duplicates before each line)
    • -f NOT IMPLEMENTED (use -k syntax instead)
    • -s NOT IMPLEMENTED (use -k syntax instead)
    • -w NOT IMPLEMENTED (use -k syntax instead)
    • -u (print only lines that are unique - do not print lines that are duplicated)
    • -k POS[,POS][OPTS] or --key:POS,POS[OPTS] (specify which key (separated by field-separator above and 1-based) to sort on)
      • This option is repeatable as often as you want
      • Each POS takes the form FIELD[.CHAR]
      • The first POS indicates a starting point and the second POS indicates an ending point.
      • If no second POS is given then the sort will consider the remainder of the line significant in the sort
  • Examples:

wc (WordCount)

  • (wc OPTIONS FILES)
    • OPTIONS
      • -l = count lines
      • -w = count words
      • -c = count characters
      • -q = quiet - don't print labels "line:"/"words:"/"characters:" before the actual counts
      • Any combination of the above works as well. Note that not specifying ANY of those defaults to ALL of them.

Non-shell commands

wikish_button

  • (wikish_button [-q] ID [label])
    • wikish_button simply activates or deactivates WikiSh in general (see wikish_active below for more details)
    • ARGUMENTS
      • ID - an identifier for the button or set of actions (same ID can be used for other "buttons" to make them act in concert with one another, normally with the -q option all but one)
      • LABEL - optional label on the button. If not specified the "ID" will be used as the label.
    • OPTIONS
      • -q = don't actually display a button in this position -- just handle the activating/deactivating based on a button somewhere else on the same page with the same ID.
    • Examples
      • {(wikish_button a "List Files")(ls mygroup.*)}
        • This will list files in mygroup after the button labelled "List Files" is pressed
      • {(wikish_button -q a)(ls yourgroup.*)}
        • If on the same page as the previous example, this will list all pages in yourgroup when the above button is pressed. No button will be generated in THIS location, but the ls command will be activated/deactivated appropriately.

wikish_controlpanel

  • (:wikish_controlpanel [method]:)
    • Explanation of features
      • When this markup is placed on a page it will display a form allowing you to type in WikiSh commands, execute them, see the results, have the executed commands saved in a history file, execute commands from history, have the option to mark some of those history commands as "favorites", and then execute commands from the list of favorites as well.
    • Appearance
      • The form itself is simply a large textarea for typing in the commands, buttons underneath for executing the commands and clearing the command, and then below that a selectbox on the left containing history and a selectbox on the right containing favorites. Underneath each selectbox are buttons to execute commands, copy back and forth, etc.
    • Notes on usage
      • When typing in commands you do NOT type {(command)} -- you simply type the command. In other words if you want to do {(echo abc)} you just type "echo abc" in the command box.
      • Whatever you type will be placed in a {(wikish ...)} context, so if/then/else/fi, while/do/done, for var in list/do/done, source, etc. are all valid commands to type.
      • As in {(wikish ...)} you can separate commands with semi-colons.
      • Unlike {(wikish ...)} you can also put commands on multiple lines with indentation, comments (#) (# must be preceded by whitespace), etc. The control panel will take care of sucking out all the newlines and whitespace and putting in the semicolons to create a valid {(wikish ...)} construct. You just code as you would at a command line.
    • Administration
      • a variable ${HISTFILE} defaults to History and will be created in the same group with the current page. All history commands and favorites commands will be written to this page.
      • You can set ${HISTFILE} from config.php or other configuration file via $WikiShVars['HISTFILE'] = 'AnotherPage'; -- you can do this either before or after the inclusion of WikiSh.php. The value can include a group specification.

once

  • (once OPTIONS ONCE-ID KEY)
  • Options
    • -c page - use this page as the control point rather than the value in $WikiShControlPage
    • -r - return a useful value in addition to setting the ${STATUS} variable (for use outside of WikiSh)
  • Once uses the page found in $WikiShControlPage (settable in config.php, over-ridable via -c option) to store information about ONCE-ID and KEY. The first time it is called with a given ONCE-ID and KEY combination it will return true (${STATUS} == 0) and it will also write that ONCE-ID and KEY combination to $WikiShControlPage. All subsequent invocations of (once ...) with that particular ONCE-ID and KEY will result in a false (${STATUS} == 1) return value.
  • This command is particularly useful in combination with wikish_active to run a command (or set of commands) once but then not have to worry that they will be run again.
  • NOTE: This has nothing whatsoever to do with true shell -- it's just a meta-command that is useful in the WikiSh/pmwiki context.

wikish (Wiki SHell)

  • Usage: (wikish [OPTIONS] COMMANDS)
  • OPTIONS
    • --expandvars - expand variables earlier rather than later (primarily used for nonwikish MXes)
  • COMMANDS
    • All commands should be semi-colon separated. When in doubt put in a semi-colon - it can't hurt!
    • Newlines within an MX don't work - I believe this to be an inherent limitation of MXes? Use semi-colons instead. (See or use the source command if you want to use newlines to format your commands.)
    • You are not limited to just control structures -- any MX can be put here simply by stripping off the surrounding parentheses and ending with a semi-colon.
    • Output of one command can be piped to input of the next command via |. (If you didn't catch that, read it again - it is immensely powerful.)
    • Individual commands can be combined via && (logical and) or || (logical or) as an alternate way of implementing conditionals. A command following && will only be executed if the command prior to the && completed successfully. A command following a || will only be executed if the command prior to the || completed unsuccessfully. a && b || c constructs are valid and will result in b being executed if a was successful and c will be executed if a is UNsuccessful.
  • Special commands/control structures found only within wikish
    • if BOOL-EXPR; then; EXPR-LIST; else; EXPR-LIST; fi;
      • if statements can be nested as deep as PCRE engine will allow recursion
      • BOOL-EXPR depends on the ${STATUS} variable rather than on any typical return value. Thus only WikiSh commands will function in this capacity.
      • EXPR-LIST can be any wikish expression repeated (semi-colon delineated, as usual)
    • while BOOL-EXPR; do; EXPR-LIST; done;
      • while statements can be nested as deeply as PCRE engine will allow recursion
      • BOOL-EXPR depends on the ${STATUS} variable rather than on any typical return value. Thus only WikiSh commands will function in this capacity.
      • EXPR-LIST can be any wikish expression repeated (semi-colon delineated, as usual)
    • for VAR in LIST; do; EXPR-LIST; done;
      • for statements can be nested as deeply as PCRE engine will allow recursion
      • VAR will be assigned to each succeeding item in LIST and EXPR-LIST will be executed for that iteration
      • LIST can include wildcards or can be a simple space-separater list of strings
      • EXPR-LIST can be any wikish expression repeated (semi-colon delineated, as usual)
    • source PAGENAME [arg1 [arg2 [...]]]
      • This is WikiSh's a way of allowing wikish scripts to be formatted in a readable fashion on multiple lines. The entire page (or section if you use one of the PAGENAME#SECTION alternatives - see files above) must be wikish syntax. Multiple lines will be joined with semi-colons and #-style comments will be stripped (make sure they are either at the beginning of a line or preceded by whitespace to differentiate from pagename#section syntax), so you can implement rudimentary functions in a nicely indented, newline-separated style and have it all work as expected.
      • arguments will be passed positionally as ${0} ("function name" - pagename or page#section), ${1}, ${2}, etc.
      • ${#} will return the number of parameters passed to the most recent function call (not including #0)
      • note that these positional arguments are always global (as are all WikiSh variables) and so calling a new "function" will overwrite the old positional variables - best practice is to save them off in uniquely named variables when you enter your "function"
      • although the word "function" is being used here the reality is that the page is simply read and the contents placed within the current {(wikish ...)} command as text which will then be executed. There are no extra calls to {(wikish ...)} or anything like that -- think textual insertion only.
    • exit [#]
      • This allows for early termination of a {(wikish ...)} MX.
      • The optional digit following will become the exit status of the entire MX.
    • return [#]
      • This allows for early termination of a function within an overall script. The script will continue from the place where the function was called from.
      • When used outside the context of a function it is identical to "exit"
    • backquoted sub-programs
      • Anything found between backquotes will be executed and the results placed in that spot on the command line. Only then will the command be run. This is a way to construct a portion of your command line "on the fly". Note that currently backquotes are not honored within quotes.
  • Examples:
    • Listing several commands together separated by semi-colons
      • {(wikish cat Main.HomePage; echo "Hello, my name is So."; grep -i lines Main.*;)}
    • Simple if..then..else..fi
      • {(wikish if true; then; echo YES; echo hello; else; echo NO; echo good-bye; fi; echo this prints all the time)}
    • Using grep -q as the testing condition for an if-then-else
      • {(wikish if grep -q pattern Main.*; then; echo "Yes, that pattern exists in Main.*"; fi)}
    • Simple echo:
      • {(wikish echo "hello, World!")}
    • Attempting if..then..else:
      • {(wikish if true; then; echo "got to"; echo "true section"; else; echo "got to false"; fi;)}
    • Attempting for..in..do..done
      • {(wikish for i in main.*; do; echo "this"; echo "that ${i}"; done;)}
    • Attempting a somewhat more complicated test of while..do..done
      • {(set i = 1)(echo --stdout:WikiSh.Foo --tee ${i})(wikish while grep -q "^.$" WikiSh.Foo; do; set i = ${i} + 1; 
        echo --stdout:WikiSh.Foo --tee ${i}; done;)}
    • See read for an example using while read.
    • Using backquotes to limit the files we are searching through:
      • {(wikish grep mypattern `ls Mygroup.* | grep -v RecentChanges`)}

wikish_active (make most WikiSh commands active or non-active, depending on arguments)

  • Usage: (wikish_active EXPR)
    • The EXPR should be a valid wikish command (true and false and once being most common) which sets the ${STATUS} variable appropriately.
    • If the EXPR completes successfully (${STATUS} == 0) then the variable ${ACTIVE} will be set to true and all WikiSh commands will be active.
    • If the EXPR completes unsuccessfully (${STATUS} != 0) then the variable ${ACTIVE} will be set to false and nearly all WikiSh commands will be deactivated.
  • Exceptions (these commands are never deactivated):
    • true, false, test, once, wikish_active, wikish, wikish_button (perhaps later sort -q, diff -q, etc)
  • Examples:
    • {(wikish_active false)(echo hello, world)}
      • RESULT: nothing (echo was deactivated)
    • {(wikish_active true)(echo hello, world)}
      • RESULT: "hello, world" (echo was activated)
    • {(wikish_active once mycommand a)(echo Do This One Time)}
      • RESULT: "Do This One Time" the first time it is run.
      • RESULT: nothing all subsequent times.
  • NOTE: The variable ${ACTIVE} maintains its value throughout a single session unless deliberately changed. Be aware that turning off WikiSh in one particular MX will also keep all subsequent MXes from processing until WikiSh is turned on again.
  • NOTE: Running MXes in
    ...

    ...

    or other forms of the same directive results in a modified order of directive processing. This makes a HUGE difference when you are turning WikiSh on and off at specific points. Beware. Just because you have commands a, b, and c in that order does not mean they will process in that order if you have placed some of them in a (:markup:) context.
  • NOTE: This has nothing whatsoever to do with true shell -- it's just a meta-command that is useful in the WikiSh/pmwiki context.

wikish_form (interact with forms)

  • Usage: (wikish_form [method=GET|POST] [QUICKFORM] [PROCESS] [REDIRECT target=Group.Page])
    • QUICKFORM
      • Simply does the (:input form method=X:) and (:input hidden n={$FullName}:) to save you the work.
    • PROCESS
      • Processes the field variables, maintaining them between submissions and making them available in WikiSh variables (${var} syntax)
      • If the optional source=PAGE (or --source:PAGE) option is set then initial defaults will be taken from PTVs on the specified page. HOWEVER, you will not use PTV syntax in your (:input text name=$:var:) -- you will simply use (:input text name=var:) and wikish will handle the translation. (If you are messing with PTVs on other pages note the writeptv MX found in MiscMX which will be of great help to you.)
    • REDIRECT
      • Redirect to page specified using the target=PAGE option (or --target:PAGE)
      • Note that the redirect is pretty basic - limited error checking
        • You will be prevented from redirecting to a pagename that is empty (i.e., target="")
        • You will be prevented from redirecting to a nonexistent pagename unless you also specify newpage=1 (or --newpage)
  • So, for instance, you could create an entire form and have it maintain values and have variables available for validation just by doing this:
    • {(wikish_form QUICKFORM PROCESS)} (:input text foo:) (:input submit:)
    • Then you have the entire programmatic capabilities of WikiSh to validate the data, print messages, modify the current or other pages, etc.
  • NOTE: QUICKFORM and PROCESS play nicely together - use either one or both in the same MX. REDIRECT is quite antisocial - don't put him in the same MX with either of the other 2.

Examples (feel free to add useful or interesting examples) (also see WikiShExamples)

(Note that all MXes given within a normal page must be on a single line. If an example here appears to be on multiple lines that is only for clarity -- you need to put it on a single line for it to work)

Return the last 6 lines of the page Main.FiftyLines and of the combination of all the last 3 lines of Main.Fivelines, Main.HomePage, and Main.Twolines
{(tail --file_prefix:"Lines from OUTER tail PAGENAME" -n 6 Main.FiftyLines - (tail file_prefix="Lines from INNER tail PAGENAME" --n:3 
Main.Fivelines Main.HomePage Main.Twolines))}
Return lines 2-5, 12-15, anything from any string matching /LINE20/ until line 25, lines 18-27, and lines 48 to the end from the file Main.FiftyLines.
Replace every string "/LINE/" with the string "Pretty, Flowery Line ". Take all that output and write it to a TEXT file (not a wiki page!) foo2.txt.
There are some prefix lines at the start of the file and some line prefixes at the start of each line as well.
{(sed stdout='TEXTFILE--foo2.txt' file_prefix="SED EXPERIMENT 2-5, 12-15, LINE20-25, 18-27, 48-$" line_prefix="PAGENAME Line#LINENO: " 
-n -e '2,5p' -e '12,15p' -e '/LINE20/,25p' -e '18,27p' -e 's/LINE/Pretty, Flowery Line /' -e 48,$p' Main.FiftyLines)}
Take the output of the following 2 points:
-> print the text "||border=1"
-> output each line of Main.FiftyLines with the following modifications:
---> put a line at the beginning of the file with the text "|| ||LIST ALL OF PAGENAME||" (where PAGENAME will be replaced by Main.FiftyLines)
---> put a prefix on the beginning of each line which looks like this: "||PAGENAME Line#LINENO||" (where PAGENAME and LINENO will be replaced)
---> replace the end of each line with "||" (this could have been done just as easily with line_suffix)
Take all that output and put it together (conCATenate it) and output it to a page mygroup.abc.
{(cat stdout=mygroup.abc - (echo "||border=1") - (sed file_prefix="|| ||LIST ALL OF PAGENAME||" line_prefix="||PAGENAME Line#LINENO||" 
's/$/||/g' Main.FiftyLines))}

Search for the pattern "LINE" and replace it with the text "Line" in the file Main.Twolines. Write the result back to Main.Twolines.
{(sed --inline 's/LINE/Line/g' Main.Twolines)}

Show the first line of a text file (note this is not a wiki page):
{(head -n 1 TEXTFILE--foo.txt)}

print off the number of lines in main.homepage
{(wc -l main.homepage)}

print off the number of lines in every file in the group "main":
{(wc -l main.*)}

print off the number of lines, words, and characters in main.homepage
{(wc main.homepage)}

print off the number of lines in which "mystring" is matched in main.homepage
{(wc -l - (grep mystring main.homepage))}

set the variable "myarg" to the number obtained above
{(set myarg = (wc -ql - (grep mystring main.homepage)))}

print the result of the variable myarg
{(echo "I found" ${myarg} "lines in main.homepage that matched pattern")}

(:nolinebreaks:)

Notes

Error messages are simply dumped out with PHP echo statements -- is there a better way to handle this?! (There is no some limited ability to control this. You can set the generic option --stderr:OPT where OPT is "echo" (default), "messages", or "null".

Since we are pretty line-based in shell scripts it may make your life easier if you surround your WikiSh MXes with (:linebreaks:) ... (:nolinebreaks:). You don't have to, but it gets really confusing to figure out WHY you got this back if the lines are all re-wrapped...

This recipe makes extensive use of a debugging function called wdbg(). If you want to see debug output you need to do 2 things:

  • In your config.php put a line like this:
   $WikiShDefaultDebug = 4;

(the lower the number the more detailed the debug output will be -- 1 is everything, 5 is nothing). Or you can use the option --debug:N option for a given command which will change the debug level for that particular MX...

  • On the page you are testing make sure you have a (:messages:) somewhere. I usually put it at the bottom after a horizontal line just for looks...

Release Notes

ReleaseDescription
2015-06-06Implemented Markup_e() for PHP 5.5 compatibility. Since the changes were very limited there has been very limited testing - previous version is WikiSh-pre55.zipΔ if needed to revert. Note that while all occurrences of Markup(.../e) have been fixed there are still several occurrences of @@preg_replace(/.../e, ...) which need to be corrected before WikiSh will be PHP 5.5 compatible.
2010-06-09Added TEXTFILE--file.ext capabilities to the {(test -s|-f|-g|-r|-w ...)} command. No need to update unless you need this particular capability.
2009-12-18LIST options are no longer applied except to wildcarded pages -- this is a more intuitive behavior. Ls is more careful about listing textfiles for which the person does not have read permission. Grep includes pagename prefix to each line as a default if there are more than 1 pages being searched. RECOMMENDED UPDATE.
2009-11-14Changed split() to explode() in several places to reach PHP 5.3 compatibility. Includes a fix to SecLayer.php for farm usage.
2009-08-22Added -n (or --line-number) to grep for display of line numbers.
2009-06-24Added "return" to allow early termination of a function. Made "exit" stop all execution, even from within a function (previously acted as "return" within a function). Allowed hyphens in pagenames in several instances.
2009-05-15Fix a problem with PTVs on a page with a dash in the pagename.
2009-04-20Correspond to a release of toolbox.php in relation to ResolvePageName() and installation of WikiSh.
2009-04-19Worked some more on redirection, fixed a small problem with read being cleared.
2009-01-25Standardize diff output (fixes small bugs)
Added "tmp" in addition to temp, virtual, etc as special group names for virtual pages
Fixed redirect to work with external links
Added revision access on page read via Page@1234567 where 1234567 is a unix time indicating which revision to read
2008-09-28Added {latemx(...)} markup
Allowed variables in {(wikish source ...)} filename/section
Added {(od ...)} MX, primarily for debugging
2008-09-20Added --expandvars option to wikish
2008-09-17Fixed bug related to piping output to a "while read ..." loop
2008-09-14Added ${MAILXRULES} for running rules on pages sent via mailx. (Don't depend too heavily on this feature - it will probably be completely redesigned and reimplemented shortly.)
Fix mv/cp with same source/dest or with nonexist source
fix various probs with variables containing slashes, backrefs, etc.
added --form option to (set...) to work with $InputValues[]
added ${LPAREN} and ${RPAREN} constants
added >group.page=/pattern/ syntax
2008-08-21AMinor bug fix for a problem during deletion
2008-08-21Several small improvements:
* RC Pages now has a 'WikiSh.-GroupRC' definition for group level
* wikish_form improvements:
** protect against empty target on redirect
** protect against nonexistent target on redirect (unless newpage=1 is specified)
** fix escaped quotes in PROCESS
** allowed source=PAGE option on PROCESS to read from PTVs and handle defaults wikish-style
2008-08-17Added forceread and forceedit capabilities if specifically enabled via SecLayer.
2008-08-08Several small changes/bug-fixes:
* Added fetchmail MX (depends on wikimail version 2008-08-08 or later)
* Added ${*} and ${@} for positional parameters
* Fixes on quotes within backquotes
* Fix wdbg() to allow multiple arguments
* Fixed/enhanced wikish_form for method= and formname= options
* Fixed a problem with positional parameters for functions
2008-08-03Added mailx MX
2008-08-02Minor release.
* Wikish_form redirect now does a MakePageName() to ensure properly formed pagename.
* Wikish_form process is much more relaxed about backslashes
* Bug fixed on append redirect of stdout in simple MX (i.e., when wikish MX is not used)
* New global variable for configuring different types of authorization levels in chmod: $wshPmAuthList
* If the per-page RC page in WikiSh group is not readable no login page is given now
2008-08-01Bug fix release. Added --html and --display as synonym options to any MX that outputs -- basically it will fix some vspace and htmlspecialchar problems if the text is going to be displayed on the page (as opposed to be written to source somewhere). Also fixed a quoting problem within wikish when the quotes came from the underlying data.
2008-07-31Major release.
* Significant changes to configuration necessary for existing installations due to migration to SecLayer.php security.
* Recipes related to WikiSh need to be updated to match name-changes for standardization. MiscMX and others.
* Functions implemented, allowing a non-complicated presentation of wikish to non-technical authors.
* Loops enhanced with "break n" and "continue n" functionality.
* Basename and dirname both made pmwiki-centric so they by default return pagename and groupname, respectively (previous behavior still available via --raw option or TEXTFILE-- prefix on a pathname).
* Output can now be redirected to a section (echo "Hello, World" >Group.Pagename#section and other similar syntaxes)
* Output can now be redirected to PTVs (echo "Hello, World" >Group.Pagename$:varname and other similar syntaxes).
* (:wikish_Control_Panel:) now has a synonym (:wikish_command_line:) (or simply (:wikish_cl:)) and is now coded in a separate PHP file WikiShCL.php which allows more finegrained configuration (Note the implications for configuration). CommandLine also is significantly speeded up and has additional capabilities now.
* The MX chmod is a new command allowing changes to page/group authorizations ($EnableWikiShChmod must be true to allow it to be used as a security precaution even though all the usual authorizations are respected.)
* In general when specifying options you can now enter a -- (double-dash) to indicate no more options (this allows further arguments which begin with a dash but are not flags).
* The MX ls has --group, --name, -g, -G, -p, and -P as additional options.
2008-07-10Added raw echo (--raw) (with corresponding $EnableWikiShRawEcho) for debugging. Implemented deleting of session-type (virtual, temp) pages. Defaulted non-numeric numeric comparisons in test to just use 0 value. Significant performance increase by not evaluating debug_space() if not debugging. Fixed a bug with redirecting stdout to a session-type file in certain edge conditions. Fixed problem with tail where offsets were incorrectly calculated. Allowed WikiSh to be active when action=approvesites.
2008-06-30Allowed action=search to use WikiSh in search templates. I think variable expansion works on non-backquoted MXes now. Allow newlines in sed search/replace.
2008-06-22Allowed variable expansion on non-WikiSh MXes in backquotes (specifically for pagelist). Gave better error handling when a non-MX command is specified within wikish.
2008-06-08Fixed a problem with ${var:0:3} (whenever the offset was 0).
2008-05-12Added WikiShCompatible function for MiscMX and others. Implemented backquotes within quoted strings.
2008-05-05Fixed supporting functions for WikiShCrypt for using text files for IV storage. Gave nicer error reporting in the Control Panel if the admin forgets to give us write privilege to the history file.
2008-05-04Added the concept of positional parameters (${#}, ${0}, ${1}, etc.) to the {(wikish source pagename)} MX. Fixed a bug with cp which didn't allow copying multiple pages to a single group. Fixed another bug with cp which was causing problems when copying to a non-existent file. Allowed for sessions on systems without session_autostart. Fixed a bug with non-file options to for/do/done loops. Made some changes related to processing during $action=='edit' to allow for encryption during editing.
2008-04-29Fixed a problem with my release-generation which has been causing the primary WikiSh.php file to be unusable for the last several releases - someone casually mentioned that they had to use the zip version and when I researched I found the problem. Sorry about that. Added crypt capability to nearly all commands (--encrypt, --decrypt, and --passwd options) if you install the (separate) WikiShCrypt.php file and have the MCRYPT installed in your version of PHP. Added a markup rule for WikiSh variables in the body text of a page. Fixed WikiShSystemCall to be able to secure variable substitutions. Fixed rm with textfiles to respect $WikiShTextWriteList.
2008-04-25Fixed a problem with slashes in the new $WikiShTextReadList and WriteList.
2008-04-23Added WikiShSystemCall. Fixed variable substitution on for-loops. Added capability for "virtual" in-memory files in "Session", "Temp", and "Virtual" groups. Added $WikiShTextReadList[] and $WikiShTextWriteList[] arrays for added security on textfiles. Fixed appending to textfiles. Fixed handling of nonexistent files/pages in several commands. Added --session option to set. Made nicer error message if someone forgets the "=" in set. Fixed bug with sort when handed an empty file. Made --refpage option as an alternate to setting ${REFPAGE}. Added ${!prefix*} syntax to list vars. Added content variable ${SESSIONID}. Added ${!var} to get values from $_REQUEST and ${~var} to get values from $_SESSION. Implemented honoring PTVs in variable substitution.
2008-04-15Added --xargs flag which will be honored in any WikiSh MX accepting piped input. Added --list:X (honored in all WikiSh MXes which allow wildcard expansion) mirroring pagelist functionality (Can set a default via ${LIST} also -- allows to be used in for loop). {(ls ...)} greatly expanded, including -l, -t, -S, --os, etc. Bug with incorrectly reporting a needed history purge fixed. Made some changes to variable interpolation so WikiSh variables could hold PVs and still handle things appropriately. Note new ${PAGEVAR} variable to control where variable processing occurs (pre, post, or prepost). Allowed testing on blank values coming from backquotes in the {(test ...)} MX. The old --debug option in {(cp ...)} and {(rm ...)} (and {(mv ..)} by extension) has been renamed to --trial because it was conflicting with the generic option to set the debug level.
2008-04-07Added -v flag to sed and added optimized to prevent write with inplaceedit (-i) if no change had been made. Fixed a problem with comments not being stripped from code segments in certain cases via source or in the control panel. Turned off some debug statements that should have been turned off before.
2008-04-06Rewrote {(read ...)} to give session-saving capability and vastly increased navigational capability around the page. If you have anyscripts with more than the most basic use of {(read ...)} then you may need to alter them to fit the new --set:X option. Fixed a bug in control panel where sometimes certain characters caused the displayed command to be incorrect. Also in control panel made the message for purging history file occur at ${HISTSIZE} instead of an arbitrary 500. (more than 100 was causing performance drop). Also added capability to suppress history/favorites display and processing for same performance issue. Worked more on performance issues with {(ls ...)} and made the default with no arguments {(ls)} be equivalent to {(ls *.*)}. Fixed --file_prefix bug which prevented its use in {(ls ...)}. Added a new {(sleep n)} MX. Added ${SECONDSLEFT} content variable to help with timeout processing (related to ini_get('max_execution_time').
2008-04-03Made some more changes to the control panel form (with more help from Hans). Fixed {(wikish_forms target=page REDIRECT)} (also changed url=URL to target=PAGE as the name and functionality of the option.) Fixed a bug with content variable ${WIKISH_VERSION}. Implemented modifiers on variables for length, search/replace, substring, pattern-deletion, defaults, alternate, and default-assign. Improved error message when couldn't write a page because no pmwiki edit authorization. Turned off WikiSh processing in general unless ($action == 'browse'). I think I'm going to have to change that back at some point, but it's in as an experiment for now.
2008-04-02Fixed a bug with read which could cause an infinite loop of wikish_active was turned off in a read-controlled loop. Fixed another bug in read related to newlines when read from files. Add capability of using "<page" within read to specify the stdin source (it can be used in other commands as well to set a --stdin:page option, but that option will be ignored as only {(read...)} honors the stdin setting at this point). Added "exit" to the flow control structures within {(wikish ...)} for convenient getting out of loops/if-then/etc. (Still no continue or break) 'NOTE: Changed $WikiShWriteList semantics! Now multiple inclusive patterns can be specified in different elements of the array and they will be combined with a logical OR. Excluding patterns are still combined with all patterns with a "AND NOT". The control panel is *much* improved in the look-and-feel thanks to the resident forms expert, Hans, and his invaluable input.
2008-03-29Installation file now zipped and comments stripped. {(sed 's/search/replace/' ...)} now defaults to replacing only the first occurrence. If you want to replace all occurrences you will have to append the "g" flag: {(sed 's/search/replace/g' ...)}. Several other flags are now also supported (see documentation of sed). Note particularly the "F" flag which uses "fixed strings" rather than a regex for search/replace. (Basically it uses str_replace() instead of preg_replace().) The important thing to note is that the "F" flag implies the "g" flag (there's no way I can find to make str_replace() replace just the 1st occurrence). The 's/foo/bar/flags' syntax can also now use different characters instead of the forward slash so 's|foo|bar|flags' and 's^foo^bar^flags' are equivalent - this helps with the "leaning toothpick syndrome" where forward slashes have to be escaped with backslashes and regexes get very hard to read. Sed has the 1,$p syntax fixed. Again. Grep also supports a -F flag for fixed string search now. Fixed a problem introduced by FmtPagename() implementation where the '1,$p' syntax in sed wasn't working. Implemented RC initialization capabilities (Put your commands on WikiSh.Profile or WikiSh.WikiShRC pages - they will be run at the beginning of any WikiSh session.) {(ls ...)} greatly optimized. New contentful variables added (see variables for details. Startup scripts (RC) now supported. Put your commands in either WikiSh.WikiShRC or WikiSh.Profile. These are adjustable by setting $WikiShVars['WIKISHRC'] or $WikiShVars['PROFILE'] during initialization. Gave better error reporting when a write to a file fails. Most comments have been stripped out to optimize loading (and to get it under the 150k limit imposed by pmwiki.org on php files). A commented version is available by request.
2008-03-23NOTE: Corrected documentation regarding $WikiShWriteList[] array. If you had trouble with it, read that corrected section. Added capability to suppress debug statements arising from the control panel itself (new default). Fixed display of history/favorite items in the control panel which include :) text. Fixed Sed() to appropriately handle escaped slashes in both the find and the replace side.
2008-03-22CFixed problem with single-quotes being assigned via {(set -s ...)}. Fixed a problem with reading from --stdin from a text file with {(read ...)}. Prettied up a display glitch in the control panel.
2008-03-22AImproved the interface on the control panel with debug level and history count. Fixed a compatibility bug between windows/linux on the {(read ...)} MX related to newlines. Implemented FmtPagename() when expanding vars. Started using MakePagename() for consistency. Implemented a requirement for comments to have whitespace preceding so as to allow Pagename#Section syntax to sneak through. Oops on my design there...
2008-03-22Implemented back-quote in wikish. Some other stuff I'll document later.
2008-03-21Fixed -q option to grep -- somehow got lost. Added ({wikish_form ...}) in a rudimentary form - more potential for forms help later. Added ~= operator to the test command. Fixed some relatively rarely occurring bugs including sourcing from an anchored section.
2008-03-20Added $WikiShWriteList. NOTE that this now means the DEFAULT installation will allow writing ONLY to the WikiSh.* group. This is a significant change and you should carefully take note of it if you are upgrading. Also added processing of tabs via \t escape sequence in {(echo ...)} and elsewhere.
2008-03-16Fixed a bug in cat and sort which caused an extra line to be inserted at the top of the line. Implemented {(cut ...}) and {(uniq ...)}. Note that uniq uses a non-shell usage to specify fields and characters. Implemented new uniq-related options on sort (non-shell extension). Fixed a bug with cp when copying an empty file to a textfile.
2008-03-15Did a whole re-write of the history/favorites read/write operations for the control panel. Fixed piping into a while/do/done loop. Fixed read so it accepts input from a pipe and doesn't demand stdin to be set. Fixed a bug where a pipe into 1 command was used for subsequent commands as well. Added (:easter <year>:) markup just for fun - not really part of WikiSh and won't be documented, but is available.
2008-03-11Fixed -i option with sed. Fixed problem with --stdin when group was specified. Fixed a problem with wikish...source. Basically a bug-fix release.
2008-03-10Fixed a problem with (:wikish_controlpanel:) related to stripmagic() on the post. Added -u option to sort. Fixed problem with setting author on writing logs and etc.
2008-03-09Added {(wikish_button ...)} and (:wikish_controlpanel:) as a way to provide a form-based way of interacting with wikish. Added piping stdout to stdin within wikish. Added && and \|\| as alternate ways to combine commands within wikish. Added -w option to echo to expand wildcards. Implemented expansion of variables in the test MX. NOTE: Variable expansion REQUIRES curlies now - ${var} WILL expand while $var will NOT expand. Design change based on the fact that I have no single-quote alternative to suppress expansion. Improved debugging infrastructure with the debug_space() function.
2008-03-03Added {(diff ...)} and {(mv ...)}. Added {earlymx(COMMAND ...)} invocation. Changed the default for {(set ...)} to be quiet. You can return a value by specifying -v now (-q is ignored). Added section on pagename specification for reading (does NOT work on writing. Beware!)
2008-03-02Finished {(test...) with string and integer operators. Added --debug:n generic option. Added {(read ...)} MX.
2008-03-01Added {(test...)} with the file operators. Added infrastructure for stderr, but not ready for use yet. Added infrastructure for --debug:n option, but not ready for use yet.
2008-02-28Added {(sort...)}, a relatively full implementation of the shell tool. Speed may be an issue there - not sure. Changed the operation of {(set ...)} so that it would not do a "free" eval but rather only allow selected PHP functions to be used. Stole code from {(calc)} for this. (Is it stealing if you take from open source and give full credit? Not sure...) Thanks to whomever put that together. That should have taken care of that security hole. Fixed some other bugs.
2008-02-24A little more headache and for...do...done and while...do...done now can be nested as well. Also OUTPUTSPEC is fairly completely implemented. (Theoretically could include a line number or something, but why not use sed for that?!) Fixed some minor bugs.
2008-02-22After much pain and headache if...then...else...fi can be nested. The problem with {(once ...)} being processed twice during the redirect after an edit has been solved (we don't process anything in wikish if ($_POST['action'] == 'edit')). "source" capability added to wikish to allow simple functions and multi-line formatting capabilities.
2008-02-17Added wikish_active and once for meta-handling to help with the awkwardness of having the command run every time you browse the page.
2008-02-15Added while/do/done and for var in/do/done looping/control structures to wikish. Fixed a few problems there.
2008-02-13Added -q flag for grep, added concept of return status and implemented for all commands, added cp command, took a shot at the wikish command as far as if/then/else/fi.
2008-02-10Added -s flag for set, added temporary wikishpagelist, standardized some error messages, added some infrastructure in the form of a (wikish ...) MX which processes other MXes. (This will later be used to implement control structures such as if/then/else and looping.) Added general option newline=x to change the way line-oriented output is processed (you can separate lines with a comma instead of a newline, for instance). Added general option markup=code to suppress further markup processing.
2008-02-08Fixed grep -vl combination so it works together. Optimized normal grep -l. Implemented (ls ...) in a very rudimentary way. Fixed a small problem in the return value of (set ...)
2008-02-07Fixed (echo ...) so it correctly handles newlines. Implemented (dirname ...) and (basename ...)
2008-02-04AFixed (rm -f ...) so it actually works now. (Nonstandard use of $WorkDir and glob() may be suspect?). Implemented ${varname} syntax and made non-existent variables "disappear" to be consistent with shell (note that for now this requires a '1,\$p' syntax for sed -- I simply cannot figure out any way to tell whether single or double quotes have been used when within MXes...). Changed names of some $Enable... variables from $EnableWSh... to $EnableWikiSh... just for consistency.
2008-02-04Added (rm ...) and (set ...) with introductory implementation of variables. Added -l flag to grep and capability of NOT prepending a - before another MX to have it process the results of that MX as a list of files instead of the contents of a file. This gives XARGS capability (use the output of one command as the arguments of another command).
2008-01-31AFix somee glaring security holes by requiring $EnableWShTextRead and $EnableWShTextWrite to be set explicitly in config before allowing reads and writes to simple text files
2008-01-31Initial release. ALPHA. No, pre-Alpha. Better than "it compiles" but don't rest your weight on it just yet... Testing with bug reporting greatly appreciated.

Development Roadmap

Note that the order of the roadmap is not definite. If I get interested in another aspect I may move ahead out of order...

  • Allow some way of enabling/disabling each command on an individual basis for security sensitive situations
  • enhance stderr implementation. Currently supports stderr=messages, stderr=echo, stderr=/dev/null. Perhaps add stderr=inline, stderr=PAGE, etc.
  • mv -f to include history (differentiate options for cp and rm)
  • cp -f to include history
  • work on better methods of formatting markup (use simplified markdown?)
  • test suite
  • undel or restore or something like that. Invocation with multiple undelete possibilities would result in ls listing. Invocation with one possibility would result in restored file.
  • implement "find" with some shell options and some non-shell options (--linkedto=, group=, category=, var=, etc) - probably not using the page cache because that would just be reproducing what pagelist already does.
  • implement override file/page lists which will not respect other permissions ($WikiShOverrideTextRead, $WikiShOverrideTextWrite, $WikiShOverridePageRead, $WikiShOverridePageWrite)
NOTE: It would be nice if pmwiki allowed some kind of alteration of the standard Keep() mechanism. I'd like to use one namespace for single-quotes and another namespace for double-quotes. But more to the point it would be nice to have a little more control over these internals without having to reproduce the code redundantly as MarkupExprPlus has done... ParseArgs should be over-ridable, Keep() mechanism as mentioned, others?

See Also

Several useful (and somewhat explained) examples are given in WikiShExamples.

Recipes making use of Markup Expressions can be found in Markup Expressions. Most of these recipes will be interoperable with WikiSh.

Contributors

Somebody by the last name of Bourne put together sh for the first time for the *nix platform. Then a bunch of other people contributed tools and etc. Eventually the Bourne shell (sh) evolved into the Bourne-Again shell (bash). This set of tools (while not sharing ANY of the code-base, obviously) is based on the fruits of their excellent design labors.

Initial code and several implementation ideas came from Hans and his very excellent TextExtract recipe.

Learned a lot from reading code in TextExtract, PowerTools, MarkupExpressionsExtensions, MarkupExpressionSamples (note credit for code taken directly from {(calc)}), and MarkupExprPlus. Appreciation to the respective authors. Note that much of these recipes (except MarkupExprPlus - that recipe uses its own engine and so probably won't work in conjunction with WikiSh) is interoperable and can be used in combination with WikiSh.

Comments

See discussion at WikiSh-Talk

DO NOT MAKE CHANGES ON THIS PAGE - THIS PAGE IS REGULARLY OVER-WRITTEN BY THE COPY I MAINTAIN ON MY HOME SYSTEM AND YOUR CHANGES WILL BE LOST. Instead please make any changes to the WikiSh-Talk page.

User notes +3: If you use, used or reviewed this recipe, you can add your name. These statistics appear in the Cookbook listings and will help newcomers browsing through the wiki.