SCTP Linux API: One-to-many style interface

Introduction

In the previous post we saw one very simple example of SCTP client-server application with the one-to-one style API. Here I will reimplement it with the one-to-many interface and we will see different approach to the same application. Again all the code used in this post is available for you on this GitHub project. Switch to 'one-to-many_basic' branch before you continue. This and the following posts will heavily use information from RFC 6458 Sockets API Extensions for the Stream Control Transmission Protocol (SCTP). As usual I will provide links to the relative sections for your convenience. If you are serious about using Linux's SCTP socket API I highly recommend you to read the entire specification. You will find tons of interesting information there.

One-to-many in a nutshell

One-to-many interface is somewhat similar to datagram (UDP) sockets, meaning that from user point of view single messages are exchanged between the endpoints. As you know this is not entirely true because SCTP is connection oriented. So to achieve this the network stack handles association creation and tear down transparently for the user.

A typical server using one-to-many API will execute these system calls:

and for the client:

Note that the server here doesn't call accept(). New connection requests are automatically accepted by the stack and the user is notified either by notification (if they are enabled) or when DATA chunk is received. I plan a dedicated post for the notifications so I will not discuss them here. bind() system call is optional. If it is not called, the network stack will pick arbitrary IP address and port. This is good enough for a client, but I am sure you don't want to do this for a server. On client side, there is no connect() call. Recipient IP address and port are passed to sendmsg() and the network stack establishes new association if required. close() tears down all SCTP association created for the socket.

The code

Before we go on, make sure you are working on one-to-many_basic branch:

git checkout one-to-many_basic

Try to build the code and run it. The output should be similar to the example in the previous post. This time I will review the code in more detail.

common.h

Only one change here - socket type is changed from SOCK_STREAM to SOCK_SEQPACKET. As we discussed this is the way to indicate that we are using the one-to-many interface.

server.c

The command argument handling in main() is kept the same. Then new socket is created (with the new socket type - SOCK_SEQPACKET). The socket is bound to the desired port number with bind(). Note that you can call bind() only once. In case you want to bind to multiple local addresses, sctp_bindx() should be used. I will introduce this function in one future post, dedicated to multi-homing.

Then listen() is called, to indicate that we want to accept incoming connections. Here is the first difference from SOCK_STREAM. As we discussed already, calling listen() is sufficient to start accepting new connections. accept() is not called in one-to-many style server.

Finally get_message() and send_reply() are called in an infinite loop. They call in themselves recvmsg() and sendmsg() in order to receive messages from the client(s) and send back responses. Let's see what they do.

Receiving messages with recvmsg()

get_message() uses recvmsg() in a loop to get a message from the network. According to man recvmsg(2) recvmsg() has got the following definition:

ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags);

The function accepts three parameters and returns the number of bytes read. The first parameter is the file descriptor (returned from socket()). The third one is flags, which I will not discuss here. If you need information about them check man recvmsg(2). In the sample code this parameter is set to 0, which means that no flags are set.

The interesting here is how to correctly fill in struct msghdr. It is described in man recvmsg(2) like this:

struct msghdr {
    void *msg_name;           /* ptr to socket address structure */
    socklen_t msg_namelen;    /* size of socket address structure */
    struct iovec *msg_iov;    /* scatter/gather array */
    int msg_iovlen;           /* # elements in msg_iov */
    void *msg_control;        /* ancillary data */
    socklen_t msg_controllen; /* ancillary data buffer length */
    int msg_flags;            /* flags on received message */
};

Generally speaking it contains three arrays (msg_name, msg_iov and msg_control) and a flag variable. msg_name is a buffer with the IP address and port of the sender. This field should point to struct sockaddr_in (for IPv4) or struct sockaddr_in6 (for IPv6). The sample code handles only the IPv4 case. msg_namelen is the size of the address buffer.

msg_iov contains the buffer(s) where the payload of the message will be saved. It is an array of struct iovec which man recvmsg(2) defines this way:

struct iovec {                    /* Scatter/gather array items */
    void  *iov_base;              /* Starting address */
    size_t iov_len;               /* Number of bytes to transfer */
};

iov_base points to a buffer with preallocated memory and iov_len is its size.

msg_control is a buffer with ancillary data and msg_controllen is its length. It allows the programmer to receive various SCTP protocol parameters. Working with this data requires handling of different set of data structures and I will discuss it in another post.

Finally msg_flags contains message related flags. Don't confuse this with recvmsg() flags! The message flags are strictly specific for the currently received message. The most important of them is MSG_EOR, which indicates end of message. It is guaranteed that each call of recvmsg() will return data from single message. However if the buffers supplied aren't big enough, the message will be retrieved with multiple recvmsg() calls. MSG_EOR flag being set is an indication that the whole message is retrieved. You can find the rest of the flags in man recvmsg(2).

Finally, let's see a code sample from get_message(), which fills in struct msghdr:

char payload[1024];
int buffer_len = sizeof(payload) - 1;
memset(&payload, 0, sizeof(payload));

struct iovec io_buf;
memset(&payload, 0, sizeof(payload));
io_buf.iov_base = payload;
io_buf.iov_len = buffer_len;

struct msghdr msg;
memset(&msg, 0, sizeof(struct msghdr));
msg.msg_iov = &io_buf;
msg.msg_iovlen = 1;
msg.msg_name = sender_addr;
msg.msg_namelen = sizeof(struct sockaddr_in);

while(1) {
    int recv_size = 0;
    if((recv_size = recvmsg(server_fd, &msg, 0)) == -1) {
        printf("recvmsg() error\n");
        return 1;
    }

    if(msg.msg_flags & MSG_EOR) {
        printf("%s\n", payload);
        break;
    }
    else {
        printf("%s", payload); //if EOR flag is not set, the buffer is not big enough for the whole message
    }
}

On lines 1-3 a statically allocated buffer is defined. We expect to receive string, so as a precaution its size is decremented with one for the null terminator.

Lines 5-8 declare struct iovec. iov_base is set to the preallocated buffer and iov_len - to its size.

Lines 10-15 declare struct msghdr. msg_iov is set to the address of the structure we just created. iovlen is set to 1, because there is only one struct iovec. If we have created an array of struct iovec, here we would have passed the number of its elements. msg_name is set to the address of struct sockaddr_in (passed as parameter) and msg_namelen is just sizeof(struct sockaddr_in).

Lines 17-31 handle message receiving. recvmsg() is called in an infinite loop. On each call we print data to the screen. Please note that in the real world, the programmer usually needs to move the payload to a bigger buffer, until the whole message is received. In this post however we want to focus on the socket API, so it is just printed out. Also note how we check if MSG_EOR is set after each call (line 24). When this happens we break out from the loop.

Sending messages with sendmsg()

send_reply() function sends response to the client, which is again simple string. This job is done by sendmsg(). It is very similar to recvmsg(). It accepts the same parameters, but the flags are a bit different. You can read about them in man sendmsg(2). The meaning of struct msghdr fields are also a bit different:

  • msg_name and msg_namelen represent the IP address and port of the receiver of the message.
  • msg_iov and msg_iovlen are the payload of the message.
  • msg_control and msg_controllen again contains ancillary data.
  • msg_flags are the flags, defined for recvmsg().

By default sendmsg() is blocking, which means we should not bother to check if the whole message is sent. This of course is not true for non-blocking sockets or when fancy sendmsg() flags are set. Here is a sample code from send_reply(), which handles the message sending:

char buf[8];
memset(buf, 0, sizeof(buf));
strncpy(buf, "OK", sizeof(buf)-1);

struct iovec io_buf;
io_buf.iov_base = buf;
io_buf.iov_len = sizeof(buf);

struct msghdr msg;
memset(&msg, 0, sizeof(struct msghdr));
msg.msg_iov = &io_buf;
msg.msg_iovlen = 1;
msg.msg_name = dest_addr;
msg.msg_namelen = sizeof(struct sockaddr_in);

if(sendmsg(server_fd, &msg, 0) == -1) {
    printf("sendmsg() error\n");
    return 1;
}

It is pretty similar to the one in get_message() so I believe it doesn't need any explanations.

Conclusion

In this post we have reviewed a sample client-server application which uses one-to-many API. As you saw it is message oriented and hides most of the association establishment details. On client side, new associations are transparently created for the user as soon as she sends message to new destination. On server side removing the blocking call to accept() allowed us to process all the messages in one thread, which leads to completely different server design.

The sample code demonstrated one client talking to one server, which is not the exact purpose of one-to-many interface. If you wish you can start multiple clients talking to the same server and see how associations are created on the fly and how DATA chunks are sent via each one. If you want to get your hands dirty with some coding try to run multiple server instances and rework the client to speak with all of them.

Comments

Comments powered by Disqus