Revised 2015 Nov 161
Handouts:
XML-RPC HOWTO by Eric Kidd et al. Read these sections:
(2) What [Is] XML-RPC?
(3) XML-RPC vs. Other Protocols
(4) Common XML-RPC Interfaces (optional, describes extensions to the basic XML-RPC API)
(5) A Sample API: sumAndDifference
(7) Using XML-RPC with Python. Note: since xmlrpclib is part of the standard Python distribution, you can probably skip the installation instructions. The rest of this section describes a simple XML-RPC client.
XML-RPC: It Works Both Ways by Dave Warner. This short article describes an XML-RPC server in Python.
Reference. Read these sections as needed. You should at least browse through them to see what is there. You don’t need to know everything in these reference sections, but you need to know what there is and how to find it when needed. That’s what a reference manual is for, after all!
From the Python Library Reference:
The following notes cover remote procedure call (RPC) in general, and XML-RPC in particular, by means of an extended example.
Procedure is another name for function, in the sense of a piece of code that is named and can be called. We all know what happens in a procedure call—for example,
def foo (a, b, c): # line 1
return a + 2 * b + 3 * c # line 2
x = 7 # line 4
y = 2
z = 31
w = foo(x, y, z) # line 7
print w # line 8Lines 1–2 define the function foo. Line 7 calls the function foo, passing into it the three parameter values x, y, and z. At this point, control transfers to the body of foo (line 2), which evaluates a + 2 * b + 3 * c and returns the value, putting us back at line 7 with the right-hand side evaluated. This value is then assigned to w, completing line 7.
In the normal course of events, the function foo (lines 1–2) and the rest of the code (lines 4–8) reside on a single computer and are executed in a single process. Executing in a single process means that they are in the same memory space. Therefore, main can call foo by simply (a) copying the arguments x, y, z into the place where foo will expect them, and (b) transferring control (“jumping”) to the memory address of the starting instruction in foo. If main and foo were in different processes, in different memory spaces, main could not call foo in this way, could it?
Now imagine that instead of both being in the same process, main exists in a client process on one computer, and foo exists in a server process on another host.
| Client | Server |
|---|---|
|
|
Imagine that by some magic, main is able to call foo, even though foo resides in a different process on another host. This magic is what we name remote procedure call (RPC). A closely related kind of magic is remote method invocation (RMI), where the remote procedure is actually a method of a remote object. In other words, RMI is an object-oriented variation of RPC. The term “RMI” is especially used in Java network programming.
It’s not really magic, of course. Sorry, Harry Potter fans, but it’s a muggle world we live in, and the computer and even network programming are no exception. So where was the rabbit hiding, before the magician pulled it out of his hat?
The “rabbit” is a hidden layer of socket programming. It’s hidden because the application programmer doesn’t write it; the RPC system “plugs it in” for us behind the scenes. When the client executes line C4, making the remote procedure call to foo, what really happens is that the client sends a message through a socket to the server, requesting that the server call procedure foo and supplying the argument values for the procedure call. The server obliges, and when foo returns, responds with a message back to the client giving the return value. I.e., something like this is going on:
foo(x, y, z), the client sends a message to the server, saying something like “please run the procedure foo with arguments 7, 2, and 31, and tell me the result.”foo(7, 2, 31).But, the beauty of the system is that the application programmer doesn’t need to be concerned with these messages. The application programmer just defines a procedure on the server, and writes a procedure call on the client. All the rest is automatic—or do I mean automagic?
Earlier I wrote about distributed computing paradigms (2007) and three paradigms of distributed computing (2011): the client-server paradigm, the peer to peer paradigm, and the message passing paradigm. Well, guess what—remote procedure call is another paradigm! RPC is another “big idea” that we can use for network programming. It hides away most of the complexity of the message passing (sockets) paradigm. And that is a good thing.
The programmer in the message passing paradigm must be concerned with:
In short, it’s a lot of work getting a request from one host to another and getting a response back. Life would be so much simpler if we could just write a procedure call, such as query("Indiana"). And that simplification is what RPC brings about. RPC is an important paradigm because it frees the application programmer from some messy details, allowing the programmer to work at a higher level of abstraction. That’s an important benefit!
On the other hand, there are some costs to balance against that benefit. They may be small costs in many cases, but they’re worth knowing about.
One cost is that there may be some inefficiency in the RPC system. In large part this is because each remote procedure call creates a new network connection. There is no persistent network connection used by a whole series of messages. For XML-RPC in particular, which is a verbose format, there is also some extra overhead in generating and parsing the XML messages. (There are leaner formats, including JSON-RPC.)
Another problem arising from the lack of a persistent network connection is that, if there is a series of related messages, there is no connection to establish their relatedness. So the relatedness must be made clear in some other way. For example, suppose we’re sending and receiving benefit information about a particular employee. If this is done in a single persistent connection, as in the message passing paradigm, then the employee’s identity can be established once, and it remains established throughout the connection. But if it’s a series of brief connections, then we must re-establish the employee’s identity in each connection.
Some versions of RPC work only with particular languages or systems. For example, Java’s RMI works only between two Java programs: a Java program using RMI cannot “talk to” a Python or a C program. However, other versions of RPC support multiple languages and systems; these include both XML-RPC and JSON-RPC.
(Compared to message passing or sockets)
Do you know what XML is? If not everyone does, give a brief explanation….
(At this time, I would like you to understand what XML is, but not to become proficient with generating and parsing it through the DOM or other means. The ability to work with XML in these ways is certainly useful, and similar topics are covered in INFO I308 Information Representation. But it is a not prerequisite for using XML-RPC, because the Python XML-RPC library handles the XML generation and parsing for us; and it is not essential for this course (I320). When we come to the topic of web services, we may look at XML in a little more depth, but the emphasis there will be on HTML.)
XML-RPC is an API for making remote procedure calls by bundling each message (the client calling the remote procedure, and the server returning a value to the client) in an XML “document” which is sent over HTTP (the protocol for web page transmission).
In the remainder of these notes, we shall work our way through three examples. Example 1 presents a non-object-oriented RPC service, and introduces the ideas and techniques for developing protocols or APIs, servers, and clients. Example 2 and Example 3 extend the RPC technology by using the Python pickle module to serialize objects, transforming them into a sequence of bytes, so that they can be transported through the networks or stored in files. In this way we are able to use XML-RPC for remote objects, i.e., XML-RPC + pickle supports remote method invocation. Actually, XML-RPC all by itself lets us call methods of remote objects, but with pickle we can have even more by calling functions or methods with objects as parameters and receiving objects as returned values—objects as first-class network citizens, so to speak! Example 4 extends the XML-RPC paradigm to allow the server to “call back” to the clients.
The “protocol” for an RPC-based system is simply a list of the procedures or methods that can be called, showing their names, arguments and argument types, and return values and their types. In short, it describes the interface to the remote procedures. It is common to call it an “API” rather than a “protocol”, since it’s really like the API for any software library—the only difference being that there is a network between the caller and the called function or method.
Here’s an example. Remember our AJ protocol for arithmetic and jokes, which we implemented using sockets? Here’s the protocol, or API, for an XML-RPC version of this service:
Ah, simple and sweet!
Server source code: aj_server.py (XML-RPC version)
To run the server, for example on port 8010 on localhost:
$ python3 aj_server.py localhost 8010Commentary:
The server creates an instance of the class SimpleXMLRPCServer class, from the module of the same name.
server = SimpleXMLRPCServer((host, port))It then registers each procedure which is to be served. Registration enables the server to provide the function as a remotely callable procedure.
server.register_function(add) # etc.Note that the functions that are registered are defined in various ways. The add, sub, mul, and div functions are nested functions, defined within the function start_server. Like local variables, they are only accessible within the start_server function. The get_joke function is defined at the top level of the file, like start_server. The pow function is built into Python.
register_introspection_functions enables the server to be queried about its capabilities, e.g., a list of the methods it serves.The method serve_forever runs the server—forever, unless interrupted. So, to stop the server, we need to type Control-C (or Control-Break for Windows), or send a kill signal to its process.
If our server were an object, we could also define methods such as:
class Server:
def add (self, x, y): return x + yand register the functions like this:
self.register_function(self.add)The expression self.add binds the first argument of the add method, self, to the variable self, resulting in a function with two arguments: x, y (instead of three: self, x, y).
Client source code: aj_client.py (XML-RPC version)
To run the client, give the server’s Uniform Resource Identifier (URI), for example with the server started above,
$ python3 aj_client.py http://localhost:8010The URI is a generalization of the Uniform Resource Locator (URL) that we are all familiar with from using the World Wide Web, and we are using it here instead of separate host, port arguments because XML-RPC messages are transported over HTTP.
If you start the client with the word “trace” as a second argument,
$ python3 aj_client.py http://localhost:8010 tracethen it will display the XML documents that it sends to and receives from the server. This is informative, and may even be entertaining! Try it!
Commentary:
The client starts by getting a ServerProxy object given the server’s URI. This happens in start_proxy, called by main. The proxy will be used as an intermediary: when the client calls methods of the proxy, the proxy object calls methods on the server, gets the result from the server, and returns it to the client. Thus the ServerProxy object represents the server from the client’s point of view.
In short, the ServerProxy handles all the low-level (socket-level) networking details for the client side. Neat!
Using the proxy and the server’s introspective capability, the client gets a list of the server methods and prints it.
Then, in the text_uri function, the client goes into an interactive loop, accepting requests from the users. For each request, the client calls the corresponding proxy method (add, sub, mul, div, or tell_joke) and prints the response. (Behind the scenes, the proxy object calls the remote server over the network and receives a return value from the server.)
Note that there is no lasting connection, and hence no need to tell the server “goodbye, I’m finished.” Rather, each remote procedure call creates a new, short-term connection. The client sends its request over the connection, reads the response, and then closes the connection. All of this, of course, happens behind the scenes.
Note: This section describes how to send full-fledged objects back and forth in XML-RPC, using the pickle module and, optionally, the Binary wrapper. Objects are bundles of data (instance variables) and functions (methods).
But if you just want to send the data part, pickling and Binary wrapping are not necessary. XML-RPC will treat the object as a structure, and Python sends structures as dictionaries. So if you have an instance of class Crazy with instance variables x = 1 and y = 2, the way Python will send that through XML-RPC is as the dictionary object
{'x': 1, 'y': 2}.
Dictionaries and lists an also be sent and received without explicit data marshalling (pickling and Binary wrapping). In fact, you might be better off to use those than Python objects.
In the next two examples, the remote procedure call passes an object representing a geometrical solid (Sphere or Cube) as the first argument, and a density as the second. The server calls the solid object’s volume method, and multiplies the result by the density, obtaining the object’s mass, which it returns.
For this to work, the class definitions of Sphere and Cube must be available to both the client and server. These classes are defined in solid.py; both server and client import them.
Both examples use the protocol/API described in obj_any_protocol.txt.
Sending an object in a remote procedure call requires the object to be marshaled by the client into a form suitable for network transmission, and then unmarshaled by the server to reconstruct the object (actually, a copy of the object). This marshaling and unmarshaling is provided by the Python pickle module. The function dumps (“dump to string”) produces a bytes representation of the object, and the function loads (“load from string”) reconstructs a copy of the object from the bytes:
obj_string = pickle.dumps(obj)
obj_copy = pickle.loads(obj_string)In Python 2, the pickle dumps and loads functions converted between Python objects and strings; hence the “s” in their function names. But Python 3 makes a strict distinction between bytes and string objects.
This leads to a further complication, because string is a data type recognized in XML-RPC, i.e., an allowed type for parameter and return values, but bytes is not. Hence, for XML-RPC, we must wrap the bytes in an
xmlrpc.client.Binaryobject.
The dumps function can have a second argument, protocol, with the value 0, 1, 2, or 3. Currently, the constant HIGHEST_PROTOCOL means protocol 3. Using protocol 0 results in an ASCII string representation which is semi-readable by human beings (actually, a bytes object encoding an ASCII string, in Python 3). Using protocol 1-3 results in a binary string representation which is quite incomprehensible to humans, but a little more efficient for the computer and network. However, with the overhead of an XML wrapper and HTTP, it is still not going to be efficient compared to sockets. If you want to get really efficient, you could use the cpickle module with sockets.
The loads function does not need to be told which protocol to use for unpickling.
obj_string_ascii = pickle.dumps(obj, 0)
obj_copy = pickle.loads(obj_string_ascii)
obj_string_bin = pickle.dumps(obj, 2)
obj_copy = pickle.loads(obj_string_bin)
The client (source code: obj_ascii_client.py, the client also needs to have solid.py) is much like the client in Example 1, except that instead of getting jokes and arithmetic computations, it wants to make the remote call proxy.mass(obj, density), where obj is a geometrical solid. We can’t do this directly, but have to pickle. Unfortunately, pickling results in a bytes, which is not a valid XML-RPC data type. Since pickle protocol 0 results in ASCII encoded bytes, we can decode the bytes to a string (or use the more general “Binary-wrapper” method of Example 3).
bytes_obj = pickle.dumps(obj, 0)
string_obj = bytes_obj.decode('ascii')
mass = proxy.mass(string_obj, density)The server (source code: obj_ascii_server.py, the server also needs to have solid.py) is much like the server in Example 1, except that it wants to provide the remote procedure mass:
def mass (obj, density):
return density * obj.volume()As stated above, the object has been pickled and Binary-wrapped by the client, so the server must unwrap and unpickle it:
def mass (binary_obj, density):
bytes_obj = binary_obj.data
obj = pickle.loads(bytes_obj)
return density * obj.volume()Note that the value of obj.volume depends on the type of object (Sphere or Cube) as well as on its instance variables (side or radius). The server is calling a method of an object created by the client and transmitted to the server. More precisely, it’s a copy of the object created by the client.
The client source code is in obj_binary_client.py; solid.py is also needed. Again, pickling results in a bytes object, which is not a valid XML-RPC data type. Since binary data may contain bytes that are invalid in XML, we must wrap it in an instance of the Binary class to encode it in a form that is allowed in XML:
pickled_bin = pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)
wrapped_bin = xmlrpclib.Binary(pickled_bin)
...
mass = proxy.mass(wrapped_bin, density)The server source code is in obj_binary_server.py; don’t forget, the server also requires solid.py. Since this version of the server receives a binary-wrapped, pickled object from its client, it must unwrap the binary and unpickle the result:
def mass (wrapped_bin, density):
# wrapped_bin is a Binary wrapping of a pickled object
pickled_bin = wrapped_bin.data # extract binary data from Binary
obj = pickle.loads(pickled_bin) # unpickle binary data
return density * obj.volume()
It may seem that XML-RPC (or RPC in general) is a one-way affair: the client makes calls to the server, and has values returned to it. But can the server also make calls to the client? Arranging for such “callbacks” is surprisingly easy. All we need to do is have the client also act as an XML-RPC server.
We now implement a simple MUD (Multi-User Domain/Dungeon/…) using XML-RPC. The MUD has two rooms, East and West. The server uses two lockable sets to keep track of who is in which room. It uses a lockable dictionary to associate each client with the client’s callback proxy.
The use of locks here is probably unnecessary, because the standard library implementation seems to be a single-threaded XML-RPC server. (There are no promises of this in the standard library documentation, though.) We can test this by putting a call to
sleepin the definition of an RPC method and making overlapping calls from two clients.
This example also illustrates the need for every remote procedure to return some kind of value which is a valid XML-RPC data type. What happens if a Python function has no interesting information to return, and therefore does not use a return statement? In the ordinary world of the Python interpreter, the function actually returns a special value, None; the read-eval-print loop of the interpreter ignores these None values, i.e., it does not print them. But in the XML-RPC world, every remote function must return a value, and Python’s None is not a valid XML-RPC type. Therefore, the functions must always return a valid value, even if it’s an arbitrary value like False, an empty string, or 0. Another way of getting around this is to use the optional allow_none argument when constructing the XML-RPC server and client; however, this common extension is not supported by all implementations of XML-RPC.
Source code:
(Optional)
In Java RMI (Remote Method Invocation), it is easy to send and receive objects. Java does the equivalent of pickling, Binary-wrapping, unwrapping, and unpickling behind the scenes—the programmer does not do these explicitly, only declares implements java.io.Serializable in each class needing such treatment. In that respect, Java RMI is a little simpler.
On the other hand, Java RMI is much more complicated in these other respects:
rmic, the RMI compiler, which generates “stub” and “skeleton” files.These are the main changes in XML-RPC and pickle between Python 2 and 3:
Besides XML-RPC, there are some newer options for Python programmers, as noted in the second edition, pages 313–319.
pickle module, probably like I have done, but hiding the details.