Twisted Documentation: The Evolution of Finger: adding features to the finger service

# The Evolution of Finger: adding features to the finger service

## Introduction

This is the second part of the Twisted tutorial Twisted from Scratch, or The Evolution of Finger.

In this section of the tutorial, our finger server will continue to sprout features: the ability for users to set finger announces, and using our finger service to send those announcements on the web, on IRC and over XML-RPC. Resources and XML-RPC are introduced in the Web Applications portion of the Twisted Web howto. More examples using `twisted.words.protocols.irc` can be found in Writing a TCP Client and the Twisted Words examples.

## Setting Message By Local Users

Now that port 1079 is free, maybe we can use it with a different server, one which will let people set their messages. It does no access control, so anyone who can login to the machine can set any message. We assume this is the desired behavior in our case. Testing it can be done by simply:

    % nc localhost 1079   # or telnet localhost 1079
    moshez
    Giving a tutorial now, sorry!
    ^D

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    # But let's try and fix setting away messages, shall we?
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic

    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            d = self.factory.getUser(user)

            def onError(err):
                return 'Internal error in server'
            d.addErrback(onError)

            def writeResponse(message):
                self.transport.write(message + '\r\n')
                self.transport.loseConnection()
            d.addCallback(writeResponse)

    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol

        def __init__(self, **kwargs):
            self.users = kwargs

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

    class FingerSetterProtocol(basic.LineReceiver):
        def connectionMade(self):
            self.lines = []

        def lineReceived(self, line):
            self.lines.append(line)

        def connectionLost(self, reason):
            user = self.lines[0]
            status = self.lines[1]
            self.factory.setUser(user, status)

    class FingerSetterFactory(protocol.ServerFactory):
        protocol = FingerSetterProtocol

        def __init__(self, fingerFactory):
            self.fingerFactory = fingerFactory

        def setUser(self, user, status):
            self.fingerFactory.users[user] = status

    ff = FingerFactory(moshez='Happy and well')
    fsf = FingerSetterFactory(ff)

    application = service.Application('finger', uid=1, gid=1)
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79,ff).setServiceParent(serviceCollection)
    internet.TCPServer(1079,fsf).setServiceParent(serviceCollection)

Source listing - listings/finger/finger12.tac

This program has two protocol-factory-TCPServer pairs, which are both child services of the application. Specifically, the `setServiceParent` method is used to define the two TCPServer services as children of `application`, which implements `IServiceCollection`. Both services are thus started with the application.

## Use Services to Make Dependencies Sane

The previous version had the setter poke at the innards of the finger factory. This strategy is usually not a good idea: this version makes both factories symmetric by making them both look at a single object. Services are useful for when an object is needed which is not related to a specific network server. Here, we define a common service class with methods that will create factories on the fly. The service also contains methods the factories will depend on.

The factory-creation methods, `getFingerFactory` and `getFingerSetterFactory`, follow this pattern:

1. Instantiate a generic server factory, `twisted.internet.protocol.ServerFactory`.
2. Set the protocol class, just like our factory class would have.
3. Copy a service method to the factory as a function attribute. The function won't have access to the factory's `self`, but that's OK because as a bound method it has access to the service's `self`, which is what it needs. For `getUser`, a custom method defined in the service gets copied. For `setUser`, a standard method of the `users` dictionary is copied.

Thus, we stopped subclassing: the service simply puts useful methods and attributes inside the factories. We are getting better at protocol design: none of our protocol classes had to be changed, and neither will have to change until the end of the tutorial.

As an application service, this new finger service implements the `IService` interface and can be started and stopped in a standardized manner. We'll make use of this in the next example.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    # Fix asymmetry
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic

    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            d = self.factory.getUser(user)

            def onError(err):
                return 'Internal error in server'
            d.addErrback(onError)

            def writeResponse(message):
                self.transport.write(message + '\r\n')
                self.transport.loseConnection()
            d.addCallback(writeResponse)

    class FingerSetterProtocol(basic.LineReceiver):
        def connectionMade(self):
            self.lines = []

        def lineReceived(self, line):
            self.lines.append(line)

        def connectionLost(self,reason):
            user = self.lines[0]
            status = self.lines[1]
            self.factory.setUser(user, status)

    class FingerService(service.Service):
        def __init__(self, **kwargs):
            self.users = kwargs

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def setUser(self, user, status):
            self.users[user] = status

        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol = FingerProtocol
            f.getUser = self.getUser
            return f

        def getFingerSetterFactory(self):
            f = protocol.ServerFactory()
            f.protocol = FingerSetterProtocol
            f.setUser = self.setUser
            return f

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService(moshez='Happy and well')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79,f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(1079,f.getFingerSetterFactory()
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger13.tac

Most application services will want to use the `Service` base class, which implements all the generic `IService` behavior.

## Read Status File

This version shows how, instead of just letting users set their messages, we can read those from a centrally managed file. We cache results, and every 30 seconds we refresh it. Services are useful for such scheduled tasks.

    moshez: happy and well
    shawn: alive

sample /etc/users file - listings/finger/etc.users

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    # Read from file
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic

    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            d = self.factory.getUser(user)

            def onError(err):
                return 'Internal error in server'
            d.addErrback(onError)

            def writeResponse(message):
                self.transport.write(message + '\r\n')
                self.transport.loseConnection()
            d.addCallback(writeResponse)

    class FingerService(service.Service):
        def __init__(self, filename):
            self.users = {}
            self.filename = filename

        def _read(self):
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def startService(self):
            self._read()
            service.Service.startService(self)

        def stopService(self):
            service.Service.stopService(self)
            self.call.cancel()

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol = FingerProtocol
            f.getUser = self.getUser
            return f

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    finger = internet.TCPServer(79, f.getFingerFactory())

    finger.setServiceParent(service.IServiceCollection(application))
    f.setServiceParent(service.IServiceCollection(application))

Source listing - listings/finger/finger14.tac

Since this version is reading data from a file (and refreshing the data every 30 seconds), there is no `FingerSetterFactory` and thus nothing listening on port 1079.

Here we override the standard `startService` and `stopService` hooks in the Finger service, which is set up as a child service of the application in the last line of the code. `startService` calls `_read`, the function responsible for reading the data; `reactor.callLater` is then used to schedule it to run again after thirty seconds every time it is called. `reactor.callLater` returns an object that lets us cancel the scheduled run in `stopService` using its `cancel` method.

## Announce on Web, Too

The same kind of service can also produce things useful for other protocols. For example, in twisted.web, the factory itself (`Site`) is almost never subclassed — instead, it is given a resource, which represents the tree of resources available via URLs. That hierarchy is navigated by `Site` and overriding it dynamically is possible with `getChild`.

To integrate this into the Finger application (just because we can), we set up a new TCPServer that calls the `Site` factory and retrieves resources via a new function of `FingerService` named `getResource`. This function specifically returns a `Resource` object with an overridden `getChild` method.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    # Read from file, announce on the web!
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic
    from twisted.web import resource, server, static
    import cgi

    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            d = self.factory.getUser(user)

            def onError(err):
                return 'Internal error in server'
            d.addErrback(onError)

            def writeResponse(message):
                self.transport.write(message + '\r\n')
                self.transport.loseConnection()
            d.addCallback(writeResponse)

    class FingerResource(resource.Resource):

        def __init__(self, users):
            self.users = users
            resource.Resource.__init__(self)

        # we treat the path as the username
        def getChild(self, username, request):
            """
            'username' is a string.
            'request' is a 'twisted.web.server.Request'.
            """
            messagevalue = self.users.get(username)
            username = cgi.escape(username)
            if messagevalue is not None:
                messagevalue = cgi.escape(messagevalue)
                text = '<h1>%s</h1><p>%s</p>' % (username,messagevalue)
            else:
                text = '<h1>%s</h1><p>No such user</p>' % username
            return static.Data(text, 'text/html')

    class FingerService(service.Service):
        def __init__(self, filename):
            self.filename = filename
            self.users = {}

        def _read(self):
            self.users.clear()
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol = FingerProtocol
            f.getUser = self.getUser
            return f

        def getResource(self):
            r = FingerResource(self.users)
            return r

        def startService(self):
            self._read()
            service.Service.startService(self)

        def stopService(self):
            service.Service.stopService(self)
            self.call.cancel()

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(f.getResource())
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger15.tac

## Announce on IRC, Too

This is the first time there is client code. IRC clients often act a lot like servers: responding to events from the network. The reconnecting client factory will make sure that severed links will get re-established, with intelligent tweaked exponential back-off algorithms. The IRC client itself is simple: the only real hack is getting the nickname from the factory in `connectionMade`.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    # Read from file, announce on the web, irc
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.words.protocols import irc
    from twisted.protocols import basic
    from twisted.web import resource, server, static

    import cgi

    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            d = self.factory.getUser(user)

            def onError(err):
                return 'Internal error in server'
            d.addErrback(onError)

            def writeResponse(message):
                self.transport.write(message + '\r\n')
                self.transport.loseConnection()
            d.addCallback(writeResponse)

    class IRCReplyBot(irc.IRCClient):
        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)

                def onError(err):
                    return 'Internal error in server'
                d.addErrback(onError)

                def writeResponse(message):
                    irc.IRCClient.msg(self, user, msg+': '+message)
                d.addCallback(writeResponse)

    class FingerService(service.Service):
        def __init__(self, filename):
            self.filename = filename
            self.users = {}

        def _read(self):
            self.users.clear()
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol = FingerProtocol
            f.getUser = self.getUser
            return f

        def getResource(self):
            r = resource.Resource()
            r.getChild = (lambda path, request:
                          static.Data('<h1>%s</h1><p>%s</p>' %
                          tuple(map(cgi.escape,
                          [path,self.users.get(path,
                          "No such user <p/> usage: site/user")])),
                          'text/html'))
            return r

        def getIRCBot(self, nickname):
            f = protocol.ReconnectingClientFactory()
            f.protocol = IRCReplyBot
            f.nickname = nickname
            f.getUser = self.getUser
            return f

        def startService(self):
            self._read()
            service.Service.startService(self)

        def stopService(self):
            service.Service.stopService(self)
            self.call.cancel()

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(f.getResource())
                       ).setServiceParent(serviceCollection)
    internet.TCPClient('irc.freenode.org', 6667, f.getIRCBot('fingerbot')
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger16.tac

`FingerService` now has another new function, `getIRCbot`, which returns the `ReconnectingClientFactory`. This factory in turn will instantiate the `IRCReplyBot` protocol. The IRCBot is configured in the last line to connect to `irc.freenode.org` with a nickname of `fingerbot`.

By overriding `irc.IRCClient.connectionMade`, `IRCReplyBot` can access the `nickname` attribute of the factory that instantiated it.

## Add XML-RPC Support

In Twisted, XML-RPC support is handled just as though it was another resource. That resource will still support GET calls normally through render(), but that is usually left unimplemented. Note that it is possible to return deferreds from XML-RPC methods. The client, of course, will not get the answer until the deferred is triggered.

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    # Read from file, announce on the web, irc, xml-rpc
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.words.protocols import irc
    from twisted.protocols import basic
    from twisted.web import resource, server, static, xmlrpc
    import cgi

    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            d = self.factory.getUser(user)

            def onError(err):
                return 'Internal error in server'
            d.addErrback(onError)

            def writeResponse(message):
                self.transport.write(message + '\r\n')
                self.transport.loseConnection()
            d.addCallback(writeResponse)

    class IRCReplyBot(irc.IRCClient):
        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)

                def onError(err):
                    return 'Internal error in server'
                d.addErrback(onError)

                def writeResponse(message):
                    irc.IRCClient.msg(self, user, msg+': '+message)
                d.addCallback(writeResponse)

    class FingerService(service.Service):
        def __init__(self, filename):
            self.filename = filename
            self.users = {}

        def _read(self):
            self.users.clear()
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol = FingerProtocol
            f.getUser = self.getUser
            return f

        def getResource(self):
            r = resource.Resource()
            r.getChild = (lambda path, request:
                          static.Data('<h1>%s</h1><p>%s</p>' %
                          tuple(map(cgi.escape,
                          [path,self.users.get(path, "No such user")])),
                          'text/html'))
            x = xmlrpc.XMLRPC()
            x.xmlrpc_getUser = self.getUser
            r.putChild('RPC2', x)
            return r

        def getIRCBot(self, nickname):
            f = protocol.ReconnectingClientFactory()
            f.protocol = IRCReplyBot
            f.nickname = nickname
            f.getUser = self.getUser
            return f

        def startService(self):
            self._read()
            service.Service.startService(self)

        def stopService(self):
            service.Service.stopService(self)
            self.call.cancel()

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(f.getResource())
                       ).setServiceParent(serviceCollection)
    internet.TCPClient('irc.freenode.org', 6667, f.getIRCBot('fingerbot')
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger17.tac

Instead of a web browser, we can test the XMLRPC finger using a simple client based on Python's built-in `xmlrpclib`, which will access the resource we've made available at `localhost/RPC2`.

    1
    2
    3
    4
    5
    # testing xmlrpc finger

    import xmlrpclib
    server = xmlrpclib.Server('http://127.0.0.1:8000/RPC2')
    print server.getUser('moshez')

Source listing - listings/finger/fingerXRclient.py

Index

Version: 11.0.0