Mei-Ling L. Liu
Computer Science Department
Cal Poly, San Luis Obispo
September 2000
(An abbreviated version of this paper appeared in the proceedings of PDPTA 2001, Las Vegas)
The fast-evolving area of distributed or network-centric computing is a hot bed of ever-changing paradigms, terminologies, and new technologies. Seemingly a new buzz-word, another acronym, or yet one more groundbreaking technology surfaces everyday. To a casual observer or a beginning student, sorting out the terminologies and technologies proves a daunting task.
Whole books have been written on the large array of technologies available for developing distributed applications. This article does not claim to achieve the same task in the limited space of one paper. It does present an enumeration of the various paradigms for distributed applications, as well as a classification of the existing well-known tools, facilities, and technologies based on these paradigms. However, the focus of this article is the use of abstraction in computing, as illustrated by these paradigms.
Distributed applications, abstraction, paradigms, interprocess communication, event synchronization.
The emergence of distributed computing as a dominant computing paradigm is well acknowledged. The prevalence of individual workstations, personal computers, and the ubiquitous Internet are all contributing factors to the popularity of net-centric software. Distributed applications, especially “the Web”, have assumed a major role in our society, to an extent unthinkable to early pioneers in the field.
Along with the increased demand for such applications came the need for tools to facilitate the formidable task of their development. Whereas there was a paucity of such facilities initially, the late 1990’s saw an explosion of ideas and technologies in the offerings. Today, there is a bewildering array of approaches and tools available to developers of distributed applications.
Arguably the most fundamental concept in computer science, abstraction is the idea of detail hiding. To quote David J. Barnes1:
We often use abstraction when it is not necessary
to know the exact details of how something works or is represented, because we
can still make use of it in its simplified form. Getting involved with the detail often tends to obscure what we
are trying to understand, rather than illuminate it … Abstraction plays a very
important role in programming because we often want to model, in software,
simplified versions of things that exist in the real world … without having to
build the real things.
In software engineering, abstraction is realized with the provision of tools or facilities which allow software to be built without the developer having to be cognizant of some of the underlying complexities. It is not an overstatement to say that these tools for abstraction are the force behind modern-day software, and exist in every aspect of application development. We all make use of compilers to abstract the detail of the machine languages. Java programmers use the abstract window kit (AWT) to rapidly develop graphic displays.
In the area of distributed applications, there has been an explosion of tools and facilities based on a divergence of paradigms, offering varying degrees of abstraction.
Webster’s dictionary defines the word paradigm as “a pattern, example, or model.” In the study of any subject of great complexity, it is useful to identify the basic patterns or models, and classify the detail according to these models. This paper aims to present a classification of the paradigms for distributed applications. The paradigms will be presented in the order of their level of abstractions, as shown in Figure 1. At the lowest level of abstraction is message-passing, which encapsulates the least amount of details. Object space occupies the other extreme of the spectrum, as it is the most abstract of all the paradigms.

Figure 1.
Distributed Computing Paradigms and their Level of Abstraction
Throughout the discussion that follows, a common application will be used to illustrate how each paradigm may be applied.
The example application is an online auctioning system. So that we may focus on the distributed aspect of the application, we will simplify the system to one which handles only one auctioned item per session. During each auctioning session, an item is open for biddings placed by the participants. At the end of a session, the auctioneer announces the outcome.
The implementations described are not necessary practical: they are meant to serve as a common thread by which the reader may compare and contrast the differences and the effects of the abstractions provided by the different paradigms.
1. Message Passing
The basic approach to interprocess communications is message passing. In this paradigm, data representing messages are exchanged between two processes, a sender and a receiver.
Message passing is the most fundamental paradigm for distributed applications. A process sends a message representing a request. The message is delivered to a receiver, which processes the request, and sends a message in response. In turn, the reply may trigger a further request, which leads to a subsequent reply, and so forth. The message-passing paradigm has been widely used in the development of applications, including well known Internet services such as the HyperText Transfer Protocol (HTTP, commonly known as “the web”) and file transfer protocol (FTP). Figure 2 illustrates the message passing paradigm.

Figure 2. The Message Passing Paradigm
The basic operations required to support the basic message passing paradigm are send, and receive. For connection-oriented communication, the operations connect and disconnect are also required. With the abstraction provided by this model, the interconnected processes perform input and output to each other, in a manner similar to file I/O. The I/O operations encapsulate the detail of network communication at the operating-system level.
The socket application programming interface1,2 is based on this paradigm. Using sockets, a logical construct, two processes exchange data. A sender writes or inserts a message into the socket. At the other end, a receiver reads or extracts a message from the socket.
2. The Client-Server Paradigm
Perhaps the best known paradigm for network applications, the client-server2 model assigns asymmetric roles to two collaborating processes. One process, the server, plays the role of a service provider which waits passively for the arrival of requests. The other, the client, issues specific requests to the server and awaits its response. Figure 3 illustrates the paradigm.
Simple in concept, the client-server model provides an efficient abstraction for the delivery of network services. Operations required include those for a server process to listen and to accept requests, and for a client process to issue requests and accept responses. By assigning asymmetric roles to the two sides, event synchronization is simplified: the server process waits for requests, and the client in turn waits for responses.
Many Internet services are client-server applications. These services are often known by the protocol that the application implements. Well known Internet services include HTTP, FTP, DNS, finger, gopher, etc.

Figure 3. The Client Server Paradigm
The two basic models, client-server and message-passing, are sufficient for the
implementation of our auctioning system, as follows:
Each participant, as well as the auctioneer program, assumes both the roles of client and server, as follows
For session control:
· As a server, a participant waits to hear an announcement from the auctioneer when (i) the session starts, (ii) there is an update on the current highest bid, and (iii) when the session ends.
· As a client, the auctioneer sends a request which announces the three types of event mentioned above.
For accepting biddings:
· As a client, a participant sends a new bid to a server.
·
As a server, an auctioneer accepts new bids and update
the current highest bid.
The client-server paradigm is inherent in most of the
facilities and technologies for distributed applications. The
connection-oriented socket API provides operations specifically for servers and
clients respectively, and the Java Remote Method Invocation API (to be
described) also refers to the participating processes as clients and servers.
3.3. The peer-to-peer paradigm
In the client-server paradigm, the participating processes play different roles: client processes issue requests while the server processes listen passively for service requests and provide the requested service in response. In particular, the paradigm makes no provision to allow a server process to initiate communication.
In the peer-to-peer paradigm, the participating processes play equal roles, with equivalent capabilities and responsibilities (hence the term “peer”). Each participant may issue a request to another participant and receive a response.

Figure 4
Peer-to-peer Paradigm
The peer-to-peer paradigm can be implemented with facilities which support message the point-of-point model of the Message System paradigm (see 3.4.2). For web applications, the web agents is a protocol specified by the XNSORG (the XNS Public Trust Organization) for peer-to-peer interprocess communication21.
3. The Message System Paradigm
The Message System or Message-Oriented Middleware (MOM) paradigm (see Figure 5) is an elaboration of the basic message-passing paradigm.
In this
paradigm, a message system serves as an intermediary among separate,
independent processes. The message
system acts as a switch for messages, through which processes exchange messages
asynchronously, in a decoupled manner. A sender deposits a message with the
message system, which forwards it to a message queue associated with each
receiver. Once a message is sent, the
sender is free to move on to other tasks.

Figure 5.
The Message System Paradigm
There are two subtypes of message system models.
The MOM
paradigm has had a long history in distributed applications. Message Queue Services (MQS) have been in
use since the 1980’s. The IBM MQ*Series6
is an example of such a facility. Other
existing support for this paradigm are Microsoft’s Message Queue (MSQ)5
and Java’s Message Service7.
The
message-passing model works well for basic network protocols and for basic
network applications. However, as
applications grew increasingly complex, it became necessary for further
abstractions to be provided for network programming. In particular, it was desirable to have a paradigm which allows
distributed software to be programmed in a manner similar to conventional
applications which run on a single processor.
The
Remote Procedure Call (RPC) model provides such an abstraction. Using this model, interprocess
communications proceed as procedure, or function, calls, which are familiar to
application programmers. Remote Procedure Call involves two independent
processes, which may reside on separate machines. A process, A, wishing to make a request to
another process, B, issues a procedure call to B, passing with the call a list of argument values. As in the case of local procedure calls, a remote procedure call triggers a
predefined action in a procedure provided by process B. At the completion of the procedure, process B returns a value to process A.
Figure 7 illustrates the RPC paradigm. A procedure call is made by one process to another, with data passed as arguments. Upon receiving a call, the actions encoded in the procedure are executed as a result.
As a comparison, the message-passing model is data-oriented,
with the actions triggered by the message exchanged, while the RPC model is action-oriented,
with the data passed as arguments.

Figure 7. The Remote Procedure Call Paradigm
RPC allows programmers to build network applications using a programming construct similar to the local procedure call, providing a convenient abstraction for both interprocess communication and event synchronization.
Since its introduction in the early 1980s, the Remote Procedure Call model has been widely in use in network applications. There are two prevalent APIs for Remote Procedure Calls. One, the Open Network Computing Remote Procedure Call, evolved from the RPC API originated from Sun Microsystems in the early 1980s. Details of this API can be found in [8]. The other well-known API is the Open Group Distributed Computing Environment (DCE) RPC 9. Both APIs provide a tool, rpcgen, for transforming remote procedure calls to local procedure calls to the stub.
Using
RPC to implement our auctioning system will proceed as follows:
The idea of applying object orientation to distributed applications is a natural extension of object-oriented software development. Applications access objects distributed over a network. Objects provide methods, through the invocation of which an application obtains access to services.
5.1 Remote Method
Invocation (RMI)
Remote method invocation (Figure 8) is the object-oriented equivalent of remote method calls. In this model, a process invokes the methods in an object, which may reside in a remote host.
As with RPC, arguments may be passed with the invocation.
The implementation of our auctioning system is essentially the same as with RPC, except that object methods replace procedures:

Figure 8.
The Remote Method Call Paradigm
5.2. The Network Services Paradigm
In this
paradigm (Figure 9), service providers register themselves with directory
servers on a network. A process
desiring a particular service contacts the directory server at run time, and,
if the service is available, will be provided a reference to the service. Using the reference, the process interacts
with the service.
This
paradigm is essentially an extension of the remote method call paradigm. The difference is that service objects are
registered with a global directory service, allowing them to be look up and
accessed by service requestors on a federated network.

Figure 9. The
Network Services Paradigm
The
implementation of our auctioning system is the same as under the RMI paradigm,
except that the auctioneer registers itself with the directory service,
allowing the participants to locate it, and, once the session has commenced, to
make bids. The participants provides
callback methods to allow the auctioneer to announce the start and the end of
the session, and to update the status of the session.
Java’s
Jini12 technology is based on this paradigm.
5.3 The Object Request broker Paradigm
In the
object broker paradigm (Figure 10), an application issues requests to an object
request broker (ORB), which directs the request to an appropriate object
that provides the desired service. The
paradigm closely resembles the remote method invocation model in its support
for remote object access. The
difference is that the object request broker in this paradigm functions as a
middleware which allows an application, as an object requestor, to potentially
access multiple remote (or local) objects.
The request broker may also function as an mediator for heterogeneous
objects, allowing interactions among objects
implemented
using different APIs and /or running on different platforms.

Figure 10. The Object Request Broker Paradigm
Implementation of the auctioning system is similar to that with RMI, with the exception that each object (auctioneer, participant) must be registered with the ORB and requested from the ORB. Each participant requests for the auctioneer object to register for the session and to make bids. The auctioneer requests for the objects of the participants, and invoke of the methods of each to announce the start of the session, the update the bidding status, and to announce the end of the session.
This paradigm is the basis of the Object Management Group’s
CORBA (Common Object Request Broker Architecture) architecture10,11. Tool kits based on the architecture include
Inprise’s Visibroker, Java’s Interface Development Language (Java IDL), Orbix’s
IONA, and TAO from the Object Computing, Inc.
Component-based
technologies such as Microsoft’s COM, Microsoft DCOM, Java Bean, and Enterprise
Java Bean are also based on distributed-object paradigms, as components are
essentially specialized, packaged objects designedto interact with each other
through standardized interfaces. In addition,
application servers, popular for enterprise applications, are
middleware facilities which provide access to objects or components.
Perhaps
the most abstract of the object-oriented paradigms, the object space paradigm assumes the existence of logical
entities known as object spaces.
The participants of an application converge in a common object
space. A provider places objects as
entries into an object space, and requesters who subscribe to the space access
the entries. Figure 11 illustrates the
paradigm.

Figure 11.
The Object Space Paradigm
In
addition to the abstractions provided by other paradigms, the object space
paradigm provides a virtual space or meeting room among provides and requesters
of network resources or objects. This
abstraction hides the detail involved in resource or object lookup needed in
paradigms such as remote method invocation, object request broker, or network
services.
For the
auctioning system, all participants as well as the service provider subscribe
to a common object space. Each
participant deposits an object into the object space to register for the
session, and to be notified when the auctioning session starts. At the onset of the session, the auctioneer
deposits an object into an object space to accept registrations for the
session, and to subsequently accept bids.
Each participant reads the auctioneering object, then invokes one of its
methods to make new bids during the session. The auctioneer in turn invokes a
method in each participant object to announce each new highest bid, as well as
the final outcome at the end of the session.
Current
facilities based on this paradigm include JavaSpaces
6. The Mobile Agent Paradigm
A mobile agent is a transportable program or object. In this model, an agent is launched from an originating host. The agent travels from host to host according to an itinerary that it carries. At each stop, the agent accesses the necessary resources or services, and performs the necessary tasks to accomplish its mission. The paradigm is illustrated in Figure 12.

Figure 12.
The Mobile Agent Paradigm
The
paradigm offers the abstraction for a transportable program or object. In lieu of message exchanges, data is
carried by the program/object as the program is transported among the
participants.
The
mobile agent paradigm provides a novel way of implementing our auctioning
system.
At the onset, each participant launches a mobile agent to the auctioneer. The mobile agent carries with it the identity, including the network address, of the participant that it represents. Once the session starts, the auctioneer launches a mobile agent which carries with it an itinerary of the participants, as well as the current highest bid. The mobile agent circulates among the participants and the auctioneer until the session ends, at which time the auctioneer launches the agent to make one more round among the participants to announce the outcome.
Commercial
packages which support the mobile agent paradigm include Mitsubishi Electric ITA’s Concordia system15, and IBM’s Aglet system16.
7. The Collaborative Application (Groupware) Paradigm
In this
model (Figure 13), processes participate in a collaborative session as a
group. Each participating process
may contribute input to part or all of the group. Processes may do so using multicasting to send data to all or
part of the group, or they may use a
virtual sketchpads or whiteboards which allows each
participant to read and write data to a shared display. Figure 13 illustrates the two categories of
the groupware paradigm.
To implement the auctioning system using the
message based groupware paradigm, the auctioneer initiates a group, to be
joined by all interested participants.
At the onset of the session, the auctioneer multicast a message
announcing the start. Each participant
may subsequently multicast a message to place a bid. The message is received by all participants to inform each of the
current highest bid. Finally, the
auctioneer terminates the session by multicasting a message announcing the
outcome.
It is not hard to see how the whiteboard paradigm can be applied to our auctioning system. The auctioneer and the participant share a virtual whiteboard. The auctioneer starts the bidding process by writing an announcement to the whiteboard. Subsequently, each participant may place a bid by writing to the whiteboard. Eventually, the auctioneer terminates the session by writing a final announcement .

Figure 13. The Collaborative Action Paradigm
The collaborative paradigm is the basis of a large number of existing groupware such as Lotus Notes20. Application programming interfaces supporting the message-based shareware paradigm include the Java mutlicast API and the Java Shared Data Toolkit (JSDT)17. The whiteboard paradigm is the basis for a number of applications such as the expert system CLIPS18, Microsoft NetMeeting19, and Groove22. The Notification Service Transfer Protocol (NSTP) has been proposed as “an infrastructure for building synchronous groupware. It is based on the idea of a coordinating notification server that is independent of any given synchronous groupware application. NSTP is intended in some ways to be the synchronous analog of Hypertext Transfer Protocol (HTTP).”23
This article looked at a wide range of paradigms for distributed applications as an illustration of the power of abstraction. To varying degrees, these paradigm provide abstractions that insulate the developers from the detail of interprocess communication and event synchronization, allowing the programmer to concentrate on the bigger picture of the application itself.
1. Object-Oriented Programming, David J. Barnes, Prentice Hall, 1999.
2. Internetworking with TCP/IP, Vol. 3: Client-Server Programming and Applications, Douglas E. Comer and David L. Stevens, Prentice Hall, 2001.
3. Java Network Programming, Elliotte Rusty Harold, O’Reilly, 1997.
4. Java Distributed Computing, http://java.sun.com/products/javaspaces/background, 2000.
5. Designing Applications With Msmq : Message Queuing for Developers, Alan Dickman, Addison Wesley, 1998.
6. IBM MQ Series Family home page, http://www-4.ibm.com/software/ts/mqseries/
7. Messaging Systems and the JavaTM Message Service, John Wetherill, http://developer.java.sun.com/developer/technicalArticles/Networking/messaging/.
8. RFC1831: Remote Procedure Call Protocol Specification Version 2, August 1995, http://www.ietf.org/rfc/rfc1831.txt.
9. DCE1.1; Remote Procedure Call, Open Group Standard, Document Number C706 August 1997, http://www.opennc.org/public/pubs/catalog/c706.htm.
10. The Object Management Group homepage, http://www.corba.org/.
11. The Essential CORBA, Mowbray and Zahavi, Wiley, 1995.
12. The Community Resource for Jini Technology, http://jini.org/.
13. Sun Microsystems Jini Network Technology, http://www.sun.com/jini/.
14. Sun Microsystems JavaSpaces Technology, http://java.sun.com/products/javaspaces/.
15. Concordia’s welcome page, http://www.meitca.com/HSL/Projects/Concordia/Welcome.html.
16. IBM Aglets Software Development Kit, http://www.trl.ibm.co.jp/aglets/.
17. Java Shared Data Toolkit User Guide, http://java.sun.com/products/java-media/jsdt/2.0/jsdt-guide/introduction.doc.html#15891.
18. CLIPS, A Tool for Building Expert Systems, http://www.ghgcorp.com/clips/CLIPS.html.
19. Microsoft Techologies – Windlows NetMeeting, http://www.microsoft.com/windows/netmeeting/
20. Lotus Notes, http://www.lotus.com/home.nsf/welcome/notes.
21. The XNSORG homepage, http://www.xns.org/.
22. Groove homepage, http://www.groove.net/
23. The Notification Service Transfer Protocol (NSTP): Infrastructure for Synchronous Groupware, Mark Day, John F. Patterson, David Mitchell, Lotus Technical Report 96-13.