# RootPrompt.org   Nothing but Unix.[Home] [Features] [Programming] [Mac OS X] [Search]

 Simple Middleware

Matt Young has written an article describing his middleware tool Uscript.

"Since macros engines issues receive, parse and generate URL encoded structures, they naturally communicate. An SQL select, formatted for the local engine, could just as easily be encapsulated within an HttpGet header and issued to another engine. I added a simple system executive that performs an Http Get call. The macro engines exchange Http messages with each other, triggering macroexpansion in the remote SQL databes. "

 (Submitted by Matt Young Tue Nov 11, 2003 )


Simple Middleware

Middleware is the glue for enterprise applications. Middleware range from the strict object exchange protocols (CORBA) to XML based information descriptors. The newer technologies, like PHP, trend toward more simplicity, but make no effort toward meta descriptors that would let anonymous objects communicate. SOAP attempts to maintain simplicity while using XML for its meta information system. The simple scripting languages work well when a uniform browser standard is the only target. In these case half of the connection is known to be a HTML display, and, especially if SQL is the only data producer, most of the meta information problem goes away.

I started down the middleware path looking for the a simple method to generate pages from my open source data base to a browser. I adopted the least complicated web server, Thttp from Acme and downloaded MySql. I skipped perl, as the many libraries confused me. I considered starting with SQL Ledger and looked at the pages generated by that application. This application used the HTML form variables efficiently but SQL Ledger still requires a mass of Perl script. My reasoning, after months of experimentation, led me to the idea of using the HTML query variables themselves as the middleware language. After all, the query information coming from the browser looked like a scripting language, or more precisely, like a macro language. All I really want was a macro processing engine server, one that uses the same grammar as URL to deliver variables and maintain state information. In other words, why carry scripts in the comments of HTML pages when the variable fields in forms and anchors are themselves scripts.

The idea appealed to me in particular because my goal was to gain expertise in SQL, and a scan through the SQL text led me to believe that SQL had enough smarts to handle the odd formatting tasks. My model was (and is) SQL ledger from DWS Systems Inc. Under the hood of their web pages are the transaction state variables, in hidden variables that identify the return path, database, account numbers; as well as the data entered by the user. Nothing surprising, but interesting is the SQL system at the other end which contains more than enough scripting power for most medium size businesses. So, as my next step toward simplicity, I defined a run time macro processor for web applications.

A macro processor, by way of summary, simply performs text substitution as text passes through in a single pass. The Unix based M4 scans text and triggers on defined words that cause it to look up a replacement text, possibly with arguments. The definitions are supplied within the text by the author. The one important characteristic of macro processors is that they embed into the host text, compared to traditional languages in which the "user" text must be embedded into the elements of the language. I consider macro language as a script in which 90% of the text are comments ignored by the macro processor. This viewpoint leads to the goal or embedding the macro scripts anywhere within HTML and SQL, and combining the two into a single application code..

Remember m4, the macro processor for Unix? That system was (and is) widely used to embed macro expansions into scripts and documents? As long a some simple care was taken, it really didn't matter where M4 scripts were placed within the document. If HTML is capable of transferring so much variable space with its URL encoding, then why not expand the URL coding scheme into an M4 style macro language? URL encodes variables in an algebraic notation vs the post fix like notation of M4, (M4 expresses macros with its arguments in a call structure), while URL the '=' assignment for definitions. Otherwise, the variable assignments moving back from the browser are nothing but macro assignments, good enough to be extended by a more powerful server side macro processor. I knew that if my run time macro architecture had any sticking points, then M4 would be the model for a solution.

Developing the URL variable grammar as a macro language.

URL coding has few special characters to worry about. The system uses three characters, =,&,and %. The '=' defines a macro, '&' terminates a macro definition, and '%' is there for special formatting. Just add an expansion operator, '$', support recursion and all the tools are available. Yet much can be done with these without interfering with the outer SQL and HTML scripts. The macro engine sits between a database and the web, data flowing in either direction being subject to macro definition and expansion. It offers natural variable management at the server side, essentially transforming between HTML forms and SQL statements. Using the URL grammar leads to efficient processing since the queries from the web already from the macro language. The results of my experiment is here , but I shall summarize the experiments.

URL coding grammar

The grammar is a strict enough copy of URL grammar that URL querieses are passed directly into the parser. Here is a short summary of the grammar:

In URL an assignment is:

apples= some kind of fruit&
And the standard expansion operator is $, so that $apples Produces:
some kind of fruit

A argument expansion example:

oranges= oranges, another kind of fruit&
apples= Apples are  fruit while $oranges &
$apples yields:

Apples are a fruit while oranges, another kind of fruit.

And recursive definitions:

fruit= Either apples= a red fruit& or oranges- another colored fruit&&

I apply this to a database application. The web data base transaction begins this something like:


This is likely to appear in a HTML form as:

< FORM NAME="GetRecord" ACTION="U?%$SqlGet" >
< /FORM>

You can guess that the $tb and $db text is replaced as the form leaves the macro engine on its way to the browser. The % is the URL escape and tells the server engine not to expand SqlGet, but to pass it on. The U is the URL based macro server engine. When a user clicks the form submit button, the variables are passed back to the engine and transformed into an SQL statement. An SQL macro definition for the process would be:

select from $dg.$tb where id = $id;

So we haven't introduced a new scripting language. we just enhanced the system that has been there since the beginning. But how is the result from SQL to be presented back to the browser? The problem is the the SQL output does not naturally present variable definitions that we can directly bind with our macro language. The solution needs to bind the SQL output into forms that can be parsed by the macro engine. That solution is in the form of format characters.

Format characters blindly place variables where they belong. The M4 macro processor (and c lib) have a rich formatting system, and in a way, the URL escape character, % is the natural formatter (probably not by accident). The macro engine needs a formatting scanner, not unlike printf. My formatting system relies on the regular structure of SQL data blocks and applies an indexing system over the records and fields. The MySql API generates pointer lists to the field elements making index management a snap. Examples of using the format function:

Let %n gets the nth field in the current record, while %. gets the current element. %+ bumps to the next field, and %( ) runs the enclosed text as until the field count is exhausted. so, we have:



) or alternatively %(<.td> %. <./td>)

which may be part of the SqlGet macro (see above). When the the macro engine scans the macro, it triggers on the % formatters and performs the access to an existing block of records and fields. Its clear that the formatting system relies advantage of the "square" output from the SQL engine and maintains counters to march through the data.

The square structure of SQL output is is the limit of "meta knowledge" within the system. But this is not as limiting as first glance would indicate. MySql, at least, has a fairly rich structure of meta statements that produce regular square forms of data base descriptors. Hence, the macro language combines these descriptors with rote row/column operations to produce HTML forms with field name bound with field data. The system can blindly discover the available tables in a database, and bind these with table operations that generate usable database management pages on the browser. Hence, the requirement that all data come from the SQL engine has not been a limit but rather an asset.

My experimental system includes row and column counters, which are incremented automatically as the SQL output is scanned. The problem is greatly simplified by the ability of browsers to manage the presentation as long as the text is formatted into the proper tags. Or in other words, just give me some looping macros, and I don't need the rest of the scripting languages.

I use some builtin utility macros, like $HttpText which expands into the content type field of HTML, and $End which terminates the HTML page, as well as numerous standard forms for SQL statements. Where are the predefined macros stored? In SQL, naturally. You can now see where this all leads. There is some built in system executive macros we need for system management and IO control, there are a few "gotchas" with the text quoting and character escaping, compounded since SQL is the container for both scripts and data.

To summarize, the URL encoded query passes into the engine, triggering macro expansions, one of which would be an SQL statement with its fields filled in. A built in system macro executes the SQL statements and obtains a block of data, which is HTML formatted by semi-intelligent indexing across the rows and columns. A return macro is triggered that adds the remaining HTML boiler plate and emits the result back to the browser. In multi-stage transactions the state variables are passed back in hidden variables.

Networking macro engines

Since macros engines issues receive, parse and generate URL encoded structures, they naturally communicate. An SQL select, formatted for the local engine, could just as easily be encapsulated within an HttpGet header and issued to another engine. I added a simple system executive that performs an Http Get call. The macro engines exchange Http messages with each other, triggering macroexpansion in the remote SQL databes. The return text from the remote engine appears in the local macing transparently as if it were a local SQL access. This completes the simple, self contained enterprise information architecture. It would appeal to small and medium sized companies that would buy low cost, pre-configured Linux machines with a closed and complete networked application architecture. (Closed; All input and all output obey the scripting language. Complete; The formatting and macro capabilities allow generation of any form usable within the system.)

An entire system has been built using this concept, with a production macro engine by EverydaySoftware dot biz (my company). I chose a good name for the concept, Uscript (from URL) , and the site is completely written in Uscript. It includes an integrated development environment, also written in Uscript, and an almost complete, public domain definition of the grammar. I have written a basic ledger accounting package in the language, also at the web site. Of course, rather that claim I write in the macro language, to be fair, I write in SQL and HTML; and let the macro language bind variables between the two.

I freelance from Saratoga,CA and can be reached here

Our content can be syndicated: Main page Mac Page

Copyright 1999-2005 Noel Davis. Noel also runs web sites about sailing and kayaking.
All trademarks are the property of their owners.
All articles are owned by their author