Aditionally, 2 years ago I hinted at the need and possibility that Javascript should be used at in all of the tiers of aplication development. And 1 year ago, at Codebits, I restated the need for a totally Javascript-based, REST-accessible, JSON-formatted web development framework. I tried to challenge people at Codebits to work in it, but no such luck. I also tried to start developing it but it was a no go: I dont have as much patience for coding as I used to (age is a bitch) and the context for serverside Javascript development just wasnt right (lack of standards, lack of OS API access, etc). And even having some Javascript interpreters (like jslibs which is what I was using at the time) with access to stuff like files and sockets wasnt much of a help: its not very much fun to try and reinvent the wheel writing a webserver from the (socket) ground up.

Well, things have changed a lot in this past year. The ServerJS/CommonJS group has been doing a lot of work on standardising a unified JS API. And there’s been a lot of work done on top of several Javascript interpreters/compilers/VMs (like Spidermonkey, Rhino, V8, etc) to provide for basic system level functionality. Plus a lot of Javascript-based basic blocks have been developed (like Narwhal, Nitro, Node, Awesome, Lawnchair, DjangoDE, PURE, etc) that allows us to get more concrete about what is needed and what can be done.

So its time to once again tell the Lazyweb what I think is needed or (better yet) what I want in a new web development framework that lasts for 10 years. Here goes:

• I want it to be Javascript based. All of it. No Erlang, Python, Ruby, etc in sight. I want the frontend/View/presentation in Javascript. I want the appserver/Controller/logic in Javascript. I want to program the appserver in Javascript. I want the DBserver/Model/data in Javascript.
• I want all the Javascript code to be compliant with the CommonJS standards. The code should be portable to any of the existing engines (Spidermonkey, Rhino, V8, flusspferd, Jaxer, Helma, etc) although I would touch anything Java-based with a long pole, so please make it work on Spidermonkey or V8.
• I want all the data to be JSON formatted and REST-accessible. The data on the database should be stored as JSON data and accessible via REST. The appserver API should be REST-based, accept JSON-formatted parameters and return JSON-formatted results. The templating system should accept JSON-formatted data for rendering the view and make REST JSON-formatted calls to the appserver.

If this isnt clear enough, let me refer some analog components that are written in other languages and what is already available in JS:

• for the database I want CouchDB but written in Javascript. Awesome or Lawnchair might be starting points. The endpoint would be something like Persevere but without the Java/Rhino dependency. Maybe Pintura is *it*…
• for the appserver I dont have the right analog. Of course that it must support JSGI. But *the* most important thing: it can not and must not allow the generation of HTML and the return of HTML to the browser/client! All methods/functions should only accept parameters and return results in JSONified format. Maybe NitroJS could be *it*. If it threw away the sucky template format. Completely. No templates on the serverside. No serverside HTML generation. Pure HTML is the new template standard (see below).
• URL routing should use the object hierarchy on the appserver as default. Explicit URL routing (static or using regexp) would have to be done explicitly.
• for the presentation layer I want pure HTML+JS. The browser should be the only component responsible for rendering templates. The clientside part of the framework should make all calls to the server using REST/AJAX and providing JSON-formatted parameters. No GET or POST parameters. It should only accept JSON-formatted results. No HTML. Standard HTML is the only templating system needed. Page elements (DIV, SPAN, etc) should have an “id” as an identifier for serverside databinding. If a page has a tag/field with “id=xpto” then that tag/field should be filled with the contents of the “id” key returned from the server in a JSON string. A good starting point would be PURE, but without using “class” as the databinding identifier (“id” should be the standard).
• I want the appserver to have features that Zope has had for years: I want acquisition (aka prototype-based programming, aka JS object-orientedness). I want a through-the-web management interface. I want ZClasses. I want automatic form generation for classes/objects and an automatic management (CRUD) interface for objects.

So, how’s that? It’s Xmas anyway, so I can ask for whatever I want. Gimme, gimme… Or I can start working on it… Who’s with me? I can be the Main Bitcher, Overall Architect and Project Manager.

As usual on the Internet, the same idea seems to bubble up
from several people all at once.

For the last 2 years or so, I’ve been thinking a lot about
web app architecture and app server frameworks. My thinking
has led to an approach that I’ve been trying to explain in
some previous posts.

But now Peter Svensson has given it a name:

Thin Server Architecture
http://unclescript.blogspot.com/2008/02/thin-server-architecture.html

Sounds like a nice buzzword and a nice TLA: TSA.

Peter has gotten together a group of people (me included) and
kickstarted a working group, already working on putting up some
clustered information in a place of its own:

Thin Server Architecture Working Group
http://www.thinserverarchitecture.com/

Meanwhile, while researching on the subject, I got a couple of
aditional links that are important for our thinking and for my
continuing posts about a future web framework:

How Nexaweb Works
http://www.nexaweb.com/home/us/index.html@cid=2295.html

Developing Rich Internet Applications with Declarative Markup
http://rockstarapps.com/wordpress/wp-content/uploads/2007/08/java-basedria.pdf

Slides 6-9 are especially interesting. In slide 9 you can easily see
what TSA is all about: its represented by the 4th stack, the one where
the app container has only UI stuff (view+logic) and server side you
only have biz logic and data.

— MV

I should start this post (hopefully the last detail before
going into a synthesis post) by remembering some words from
my initial post in this series:

“Meanwhile, lots of stuff has come up: Django, Pylons,
TurboGears, Ruby on Rails, you name it… I dont want to
go over extensively why I dont like any of them, but let
me just state this: if the “framework” allows for the
mixing of layout/design/HTML with code/programming, for
me is enough of a disqualifier.”

The problem, as I stated before, is that current frameworks
stuff a lot of power into the templating languague. And that
is not only unnecessary but its a bad, bad thing.

If you dont understand why 1) embedding Python/PHP/Ruby/etc
in HTML is bad or 2) outputting HTML from Python/PHP/Ruby/etc
is worse, I’m not even going to try and explain. Not only
because that means you are clueless but also because someone
else has done it better that me:

http://www.stringtemplate.org/about.html
http://www.cs.usfca.edu/~parrt/papers/mvc.templates.pdf

Now the problem is that although I agree with this paper and
the rules set forth in it, I disagree with the implementation.
Allowing for “$var” statements in the templates is still, in
my view, some form of entanglement and breakup of the MVC pattern.

What would be the best way to do it? Well look no further than
some already existent template engines:

PyMeld
http://entrian.com/PyMeld/

HTMLTemplate
http://freespace.virgin.net/hamish.sanderson/htmltemplate.html

Meld3
http://www.plope.com/software/meld3

They allow for the insertion of values (data binding) through
the use of HTML element attributes. Both HTMLTemplate and Meld3
create new attributes (“node” and “meld:id” respectively) but
PyMeld does it better: it simply uses the “id” attribute to do
value replacement. This is, in my opinion, the best way to do
a templating system. And there’s no need to make it “powerful”
through the introduction of loop control and other stuff like
Genshi or Zope Page Templates.

Does this mean that I have found my perfect templating engine?
No. You see, these are serverside templating engines and the
template/view should be implemented at the client level, if you
want to adhere to the MVC pattern.

What I would like is something like these templating engines:

Ajax Pages
http://ajax-pages.sourceforge.net/

Javascript Templates
http://www.fishwasher.com/jst/

Trimpath Junction Javascript Templates
http://code.google.com/p/trimpath/

Unfortunately all of these engines make the usual mistake of
introducing new syntax to HTML (the “$var” problem or variants
thereof instead of relying on the existing HTML markup, namely
the “id” element markup.

To sum it up, what I would rely like is a Javascript implementation
of Meld.

Aditional templating functionalities that might not be provided
through the use of the “id” attribute (although I think that it
would be all that is needed) could be added through the use of
aditional semantic potential to existing HTML markup with Javascript.
See for example Inc, for JS code include (although I would prefer
for this to be done through the implementation of the “src=” attribute
for DIVs.

http://johannburkard.de/blog/programming/javascript/inc-a-super-tiny-client-side-include-javascript-jquery-plugin.html

— MV

Like I said before, most of the week’s linkage
has had to do with CouchDB or, generally speaking,
to do with JSON-formatted REST-accessed database and
application servers.

http://mvalente.eu/cat.cgi?JavaScript%20in%20All%20of%20the%20Tiers
http://del.icio.us/mvalente

CouchDB is sure fun to use, but its Erlang dependency just
pisses me off.

http://couchdb.org/
http://code.google.com/p/couchdb/
http://jan.prima.de/~jan/plok/archives/108-Programming-CouchDB-with-Javascript.html

The way that I’ve been thinking about a new web development
framework indeed involves a REST interface and JSON or XML
formatted data. But I just think that a) you might as well
standardise the programming languague used at all tiers (and
that means Javascript) and b) you should have REST/JSON as
interface not only to the DB tier but also to the logic tier.

Sure enough there are some alternatives if you want to go
that way.

Basura DB (Python based JSON/REST DB implementation)
http://www.intertwingly.net/code/basura/
http://intertwingly.net/blog/2007/09/24/Tests-Id-Like-CouchDB-to-Pass

Persevere (Java based)
http://code.google.com/p/persevere-framework/
http://www.ibm.com/developerworks/web/library/wa-aj-objmap/

Robaccia (Python based web framework includes a RESTful JSON server)
http://bitworking.org/news/restful_json_server

JSONStore
http://taoetc.org/97

DBSlayer (HTTP/JSON interface for MySQL databases)
http://code.nytimes.com/projects/dbslayer

But aditionally to databases, I’m thinking more in line with
the proposed RADAR architecture of having the REST interface
also available for the app server. There arent as many options
here…

The RADAR Architecture
http://pragdave.pragprog.com/pragdave/2007/03/the_radar_archi.html

Naked Objects
http://www.nakedobjects.org/tutorial/index.shtml

NetKernel
http://www.theserverside.com/news/thread.tss?thread_id=47695
http://www.1060research.com/netkernel/

And sure enough all of the above is typically Java-based (yuck!),
with some other languages thrown in (Python, Erlang), but none
goes all the way and standardises on Javascript as the implementation
language and the glue language. Apparently I am not the only one
considering this idea…

Alt Framework
http://marcello.cellosoft.com/projects/alt/

Simile’s Exhibit Framework
http://simile.mit.edu/exhibit/

— MV

I’m trying to wrap my mind around Model-View-Presenter,
Presentation-Abstraction-Control and other UI architectural
patterns, so I might as well warn you that my brain might
explode…

http://ctrl-shift-b.blogspot.com/2007/08/interactive-application-architecture.html
http://www.martinfowler.com/eaaDev/uiArchs.html

Definitely stuff to think about regarding a future web
app framework.

— MV

“SOFEA is an architectural style which in its definition
criticizes (and manages to make a very good definition of)
the current web frameworks, and (as they say) “none of them
satisfies”.

http://unclescript.blogspot.com/2007/11/coming-frameworks-sofea-jda-squirrel.html

This is what I have been talking about in my posts about a
future web framework. The guys that are proposing SOFEA make
a very good technical argument in their paper and presentation.

http://wisdomofganesh.blogspot.com/2007/10/life-above-service-tier.html

If you pay particular attention to page 26 in the paper (about
how the Service Oriented Front-End Architecture model should work)
you will see that it is precisely what I’ve been talking about.

David Coffey also seems to have come to the same conclusions. And,
like me, that it all converges into Javascript. He’s building Triad,
a Javascript MVC framework.

http://blog.davidadamcoffey.com/4/triad-a-javascript-mvc-framework

Compare his diagram:

http://blog.davidadamcoffey.com/wp-content/uploads/2006/11/mvc2.png

With my previous post:

http://mvalente.eu/cat.cgi?A%20Future%20Web%20Development%20Framework%20-%20Architecture

— MV

There is even less to say about the controller part of
the framework. We dont even have to deal with religious
wars regarding what language to use. Your option is Javascript
and nothing else. Anyone thinking “what about Java” or “what
about Flash” (and current iterarions, Air, JavaFX, etc) will
have to explain what is the justification to have a VM/interpreter
available (the browser) and to start yet another VM/interpreter
(JRE or Flash plugin).

Nonetheless there are a couple of points that should be made.

A future web framework would have to guarantee that the controller
part, in Javascript, doesnt allow for business-logic embedded in
the controller code and doesnt allow for obtrusive Javascript (in
particular inline code)

The right way to do it would be to make declarative the use of
the UI behaviour the same way that, using CSS, we did away with
inline styles.

An obvious candidate would be, of course, Behaviour

http://bennolan.com/behaviour/

But there are other options. One is event:Selectors

http://alternateidea.com/event-selectors/

(although its based on Prototype.js and much prefer JQuery.js)

Another would be XForms and any Javascript implementation of
it (ie. not having to depend on the browser implementing the
standard)

http://ajaxforms.sourceforge.net/
http://www.formfaces.com/

Here though we start getting into a lot more than the UI
behaviour, namely we start getting into template/view stuff
and I want to deal with this in a separate post.

Its not always easy to define where the UI/controller part
is different from the UI/view/template part. I have a button
that takes 2 fields and computes some function. Should that
be done in the controller or in the template? I have a button
on a menu that takes me to another application screen. Where
should that be done, in the controller or in the template?

(just for the record, the first one is template level since
it has to do with data processing, data binding and business
logic, and the second one is controller level since it has to
do with application flow)

But it is difficult to define and decide and it is one of the
major problems with current web frameworks since it leads to
spaghetti code mixing controller stuff and model stuff. A new
web framework should be able to stop or minimize this problem.

— MV

There really isnt much to say about the middle component for
a 3-tier architecture (the business logic). You can of course
get into religious wars about which language it should use. This
is the component that usually generates all the talk precisely
because of the language issues. When discussing Django, Rails,
.NET, etc, people arent really arguing about what the application
(logic) server should do or what its architecture should be: they’re
just arguing on whether Python is better than Ruby is better than
Rails is better than PHP…

Me, although I currently favour Python, I am more of a syncretic
integrator: whatever solves the problem faster without spending
too much of my brain processing time (this immediately eliminates
Java.. LOL :-). Thats why I like some of the more recent options
available like Virtuoso or ActiveGrid that allow you to program
the business logic in a variety of languages (either choose one or
use all at the same time).

http://www.activegrid.com/
http://www.openlinksw.com/virtuoso/

Virtuoso, for example, by including the VM runtime for several
languages, is able to run code in Java, .NET, Python, Perl or PHP.

http://www.openlinksw.com/virtuoso/architect/vtechnical.htm

(nice graph there, maps some of my ideas nicely)

Nonetheless there are two points worth exploring about the logic
layer, whatever the language of your choice: how the logic layer
interfaces to the storage layer and how the logic layer interfaces
to the presentation layer.

The first problem I have already referred to in my previous post
about storage. Even though the usual interface is the SQL language
(due to the fact that 99% of the times the storage layer is implemented
using a relational database), nowadays most programming is being done
using an object-oriented language. Hence the object-relational impedance
problem and the proliferation of object-relational mappers (ORM), which
have become especially popular due to its use in Ruby-on-Rails ( more
precisely ActiveRecord).

As I stated in my previous post I dont think that this is the way of
the future. The indiscriminate use of the relational/SQL database is
way past due in an object/document/unstructured/hierarchical oriented
world and object-oriented databases should become the norm. On the other
hand I do have to admit that having my data closed down in some format
which is proprietary really bothers me a lot. Since I wrote my post about
storage, CouchDB has entered the radar of many people and it seems to
contain the basis for what is needed: a OO database that stores its
objects in a simple text format (JSON) with a REST interface.

http://couchdb.org/

The second problem (logic-to-presentation interface) actually mimics
the logic-to-data interface and it also has something to do with a later
issue (templating or view/controller). As in the logic-to-data interface
problem, where the logic level (ex: Perl or Python) currently embeds a
lot of data stuff (ie. SQL), the logic-to-presentation interface also
suffers from a lot of confusion, especially when you have templating systems
that allow you to mix code and presentation. PHP is a particularly bad
example of this problem.

This could be solved by explicitly denying the logic level to output
any kind of presentation code. Presentation level stuff (ie. templates
or view/controller) would be in charge of doing any presentation stuff,
getting all the data/logic from the applicatin server (through REST calls,
with data in JSON, XML, SOAP, whatever), with the application server being
explicitly prohibited from outputing any kind of HTML markup code (it just
wouldnt be taken into account or rendered).

What comes to mind as an example is an application server, using whatever
language, that would behave exclusively like TurboGears/CherryPy with the
jsonify decorator: you could call any URL in AJAX form and you would receive
results in JSON (or any other format), no HTML allowed.

http://docs.turbogears.org/1.0/JsonifyDecorator

Although I see no problem with the same server delivering both page
templates (ie. view/controllers) and logic (they are both HTTP after all),
that implies some discipline. An actual separation of concerns would
imply separate servers: one would only serve static pages (ie. templates)
and could be a standard HTTP server (Apache, etc) and the other would
only serve the processing results for received requests.

Event though I’m a bit agnostic regarding the choice of language for
your application server, I do have to state my opinion regarding current
trends and what I see as being the future endgame and the best choice.

Everyhing points towards a Javascript future.

http://steve-yegge.blogspot.com/2007/02/next-big-language.html

This is probably unpopular to say and there will be several voices
talking about Javascript’s speed. But the fact is that it doesnt make
much sense to use a language at logic/application level and then to
use another at presentation/template/view/controller level. Especially
when you take into account the current AJAX and JSON trends. You just
create aditional impedance when your doing stuff in Javascript at
presentation level (lets say field validation for example) and then
you’re encoding form fields into XML or POST fields, requesting some
processing from the application server which has to unmarshall the data
into whatever its format is, process it and remarshall the data back
into XML or JSON for delivery to the presentation layer.

In the last couple of years Javascript, which was available server
side back in the Netscape days (it was called LiveWire), is making
its comeback in the application server layer with a bunch of new
developments like OpenMocha, Phobos or Helma.

http://helma.org/
http://openmocha.org/

https://phobos.dev.java.net/

So, to sum it up: a future web development framework would include
an application server that exclusively served webservice requests
preferably in a REST way and in JSON format, explicitly barring any
output of presentation/HTML code. Although it could work in any
language, Javascript would probably be the best option given current
trends.

— MV

The way I see it, there are two problems regarding
storage in web frameworks that are orthogonal to
each other:

– one problem is the fact that we need storage at
each level of the 3-tier architecture and/or at
each component of the MVC pattern. We need to store
the presentation templates/views, to store the
logic/model and to store the data itself. Right now,
and this is one of the problems of current frameworks,
different types of storage are used at different levels.
Usually templates and code are stored using files at
filesystem level and data is stored using a relational
database.
This generates two subproblems: one is that you have
a disconnect between types of storage used (and dont
tell me that you could use the relational database to
store scripts and templates; you just look silly) which
gets even more ugly when the data you want to store is
not structured/relational (files? images? documents?).
If you suggest using blobs in the relational database
you wont just look silly but also stupid.
The second subproblem is the disconnect you get from
using a relational database to store the object oriented
models from the MVC pattern. The infamous object relational
impedance, usually solved by the use of OR-mappers (that
only go so far because they store objects properties but
not objects methods.

– the second major problem is the type of storage to
use, preferably at all levels of the 3-tier architecture.
As I said before, usually filesystem files are used for
templates and code and relational databases for data.
But this is just “legacy” thinking and it generates
impedance problems, not only between MVC/OO thinking
and relational storage but also between managing files
and managing tables.

What should or could be the way to tackle these two
problems? Well, I think that using a single uniform
storage would be the way to do it. You either decide
that you’re going to use files all the way, relational
database all the way or whatever other option all the
way.

I think that relational is out of the question. Not
only is it stupid to store templates and/or code in
this way, but you also get the OR impedance problem
and the “binary files in blobs” problem.

So its either filestorage for all or a second option.

Filestorage is quite atractive. Templates are already
there, code is there, easy to store documents/blobs…
The problem is structured data, usually stored in
relational databases.
One option to solve this would be to use a file level
abstraction of relational storage. Plan 9 stuff comes
to mind; an interesting option would be the use of
libferris, since that would also bring aditional file
abstraction views of several resources.

http://witme.sourceforge.net/libferris.web/

Doing “ferrisls -0 pg://localhost/xmldotcom2007/customers”
to get a listing of the Customers table in PostgreSQL is
quite cool…

Another option would be to use plain old files to
store data. This data could be stored formatted in XML
(as is now in fashion), although with current interest
in REST and JSON, it comes to mind that plain old RFC822
formatted files would fall in quite nicely (even the
separator ‘:’ is the same…).

For this option I can think of at least two tools
that might come in handy. APE and plain old IMAP/NNTP
storage…

http://hathawaymix.org/Software/Ape
http://deflexion.com/messaging/
http://pessoal.org/papercut/

Filestorage would have aditional benefits of not being
opaque, easily accessed by other tools. There would also
be the benefit of being able to use “normal” tools (like
vi) or more “advanced” ones (like versioning and backups)

The second option for storage: object oriented databases.

Their time is now more that due. Early ones had huge
performance problems and didnt have the SQL version du
jour. But there are now some OO databases that have quite
good performance, possibility of XML and document storage
and even SQL queries.

Some standard relational databases, like PostgreSQL,
already have the OO usage option.
Another possibility is the use of native XML databases
like eXist.
Finally, the way that my longtime favourite Zope has
chosen, the use of a native OO database (ZODB in Zope’s
case)

I can tell you that I’ve been a bigger fan of Zope/ZODB
in the past. At the end, the fact that ZODB is opaque (ie.
you can only get at your data using Python as tool)
actually becomes a problem. You get nice “things” like
acquisition, persistence, versioning, etc but you miss
out on accessibility (using vi, doing a grep on all your
code, etc). Unfortunately the development of APE and
related tools seems to have stopped, since that would seem
to be the perfect solution.

http://www.zope.org/Members/jdoyon/svnapezope

In conclusion: a future web framework would have to
have a single storage system for all its components. This
could either be a filesystem based system providing an
interface acceptable to the code level (SQL? XML?)
or an OO database with storage viewable as filesystem
objects. An important thing: the interface provided to
the code level could have SQL, SPARQL, XPath, whatever,
as options; but as time goes by its becoming clearer that
a REST interface is obligatory.

http://www.25hoursaday.com/weblog/2007/07/13/GoogleBaseDataAPIVsAstoriaTwoApproachesToSQLlikeQueriesInARESTfulProtocol.aspx
http://exist.sourceforge.net/devguide.html#N1021D
http://phprestsql.sourceforge.net/

— MV

Nearly 8 years later, even though I’m dissatisfied with the path
Zope took (in particular version 3), I’m still searching for something
that is better, since all the other web frameworks suck.

http://groups.google.com/group/pt.comp.programacao/msg/01af20cd35371370

http://zopereplacement.wikidot.com/

Turbo Gears comes close to what I would like to use; but its
still missing so many important things that I prefer to keep
looking and to brainstorm on how to do right a future web
development framework.

http://www.turbogears.org/

http://www.reahl.org/wfmwiki

— MV