Working with SCTP ancillary data in FreeBSD

Introduction

The socket interface provides good generalized interface for the underlying networking protocol, which is exactly what one programmer needs in most of the time. However for some more specific use cases someone might want to exploit specific protocol features, which can't present in the generalized interface. Ancillary data allows the socket interface to handle such specific protocol parameters/features, otherwise unaccessible from the socket API. Today we will add some ancillary data support to the one-to-many style application from the previous post.

Before digging right into the ancillary data I believe I owe you an explanation. My initial intention was to demonstrate the SCTP interface in Linux and as you have probably noticed, up to now I wrote only for Linux. However it turned out that not all ancillary data control messages are supported out of the box in the Linux distribution I currently use (Ubuntu 14.04.2 LTS with kernel version 3.16.0-36). Because the SCTP stack in Linux was ported from FreeBSD I decided to see what is supported there. Luckily all control messages described in Section 5.3 from RFC 6458 were supported in FreeBSD. That's why for this post I will use this operating system.

If you need to use some SCTP specific features in your Linux powered project I suggest you to use sctp_sendmsg, sctp_connectx, etc.

How ancillary data works

In struct msghdr there are two fields which I ignored up to now - void* msg_control and size_t msg_controllen. The first one points to preallocated buffer, which holds ancillary data structures and the second one is its length. This buffer is filled in with the ancillary data structures when recvmsg() is called. You receive only explicitly enabled (with socket option) parameters. The same buffer is also used to pass additional SCTP parameters to sendmsg() without the requirement to enable additional socket options.

Each ancillary data structure is struct cmsghdr, which is defined in RFC 3542 Section 20.2 like this:

struct cmsghdr {
    socklen_t  cmsg_len;   /* #bytes, including this header */
    int        cmsg_level; /* originating protocol */
    int        cmsg_type;  /* protocol-specific type */
               /* followed by unsigned char cmsg_data[]; */
};

cmsg_len is the size of the structure, including the header. cmsg_level defines the protocol, which in case of SCTP is IPPROTO_SCTP. cmsg_type is the type of the data structure, which contains the parameters. cmsg_data is buffer, which contains the data structure itself. Because of the different size of each ancillary data structure, additional padding might be added before cmsg_data. I strongly advice you to read RFC 3542 Section 20.2, which explains very well how struct cmsghdr is populated.

In both cases (sending and receiving) msg_control can contain more than one control message, which makes memory handling a bit tedious. For convenience there are CMSG_XXX macroses which can help you to manipulate msg_control buffer correctly. They defined in RFC 3542 Section 20.3:

struct cmsghdr *CMSG_FIRSTHDR(const struct msghdr *mhdr);

The input parameter is pointer to struct msghdr and returns pointer to the first struct cmsghdr in msghdr's msg_control buffer.

struct cmsghdr *CMSG_NXTHDR(const struct msghdr *mhdr, const struct cmsghdr *cmsg);

The input parameters are pointer to struct msghdr and pointer to struct cmsghdr. It returns pointer to the next struct cmsghdr in msg_control buffer.

  • CMSG_DATA - its declaration is equivalent to:
unsigned char *CMSG_DATA(const struct cmsghdr *cmsg);

The input parameter is pointer to struct cmsghdr. It returns pointer to the actual data structure in cmsg_data. The return type is unsigned char and it has to be casted to the correct data type.

socklen_t CMSG_SPACE(socklen_t length);

The input parameter is the length of the structure that should be kept in cmsg_data field. The return value is the required size of msg_control buffer for this ancillary data type. The length returned includes the space required for the control message, its corresponding struct cmsghdr and the padding. This function is usually used when the size of msg_control buffer is calculated. This function should not be used to fill in cmsg_len in struct cmsghdr.

  • CMSG_LEN - its declaration is equivalent to:
socklen_t CMSG_LEN(socklen_t length);

The input parameter is the length of ancillary data structure and it returns the length, that should be saved in cmsg_len (in struct cmsghdr). Please note the difference between this function and CMSG_SPACE. CMSG_LEN returns the length of struct cmsghdr, while CMSG_SPACE returns the required length for struct cmsghdr AND the padding after it. If this is still not completely clear for you, have a look at the diagram in RFC 3542 Section 20.2. It shows what each length represents.

Code snippets showing how to accomplish a few common tasks

Now let's see some examples. All of them are extracted from the sample code that we will review later.

  1. Iterating over the buffer msg_control buffer and processing SCTP_RCVINFO (more on this data type later):
struct msghdr* msg;         //Let's pretend that this is the pointer to the struct msghdr, returned from recvmsg()
struct cmsghdr *cmsgptr;
for(cmsgptr = CMSG_FIRSTHDR(msg); cmsgptr != NULL; cmsgptr = CMSG_NXTHDR(msg, cmsgptr)) {
    if (cmsgptr->cmsg_len == 0) {
        //this is error!
        break;
    }
    if(cmsgptr->cmsg_level == IPPROTO_SCTP && cmsgptr->cmsg_type == SCTP_RCVINFO) {
        u_char *ptr = CMSG_DATA(cmsgptr);       //Remember that CMSG_DATA returns unsigned char* ...
        struct sctp_rcvinfo* ri = (struct sctp_rcvinfo*)ptr;    // ... that you cast to the desired data type
        //Process the data here
    }
}
  1. Dynamically allocating memory for msg_control buffer for two ancillary data structures (SCTP_RCVINFO and SCTP_NXTINFO)
char* ancillary_data = NULL;
int ancillary_data_len = CMSG_SPACE(sizeof(struct sctp_rcvinfo))        //for SCTP_RCVINFO
                            + CMSG_SPACE(sizeof(struct sctp_nxtinfo));  //for SCTP_NXTINFO

if((ancillary_data = malloc(ancillary_data_len)) == NULL) {
    //this is error
}
memset(ancillary_data, 0, ancillary_data_len);

struct msghdr msg;
memset(&msg, 0, sizeof(struct msghdr));
msg.msg_control = ancillary_data;
msg.msg_controllen = ancillary_data_len;
//continue with the other parameters
  1. Prepare msg_control buffer and pass it to sendmsg()
//Let's pretend that this is the pointer to the struct msghdr that we are about to pass to sendmsg()
struct msghdr* msg;

//this is the first data structure. All consequent calls will be CMSG_NXTHDR(msg, cmsg)
struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = IPPROTO_SCTP;
cmsg->cmsg_type = SCTP_INIT;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct sctp_initmsg));
struct sctp_initmsg* init_dt = (struct sctp_initmsg*) CMSG_DATA(cmsg);
//fill in the parameters via init_dt pointer

//for all consequent data structures use CMSG_NXTHDR
cmsg = CMSG_NXTHDR(&msg, cmsg);
cmsg->cmsg_level = PROTO;
cmsg->cmsg_type = SCTP_SNDINFO;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct sctp_sndinfo));
struct sctp_sndinfo* sndinfo_dt = (struct sctp_sndinfo*) CMSG_DATA(cmsg);
//fill in the parameters via sndinfo_dt pointer

Ancillary data in SCTP protocol

SCTP ancillary data structures are defined in Section 5.3. There you can find the cmsg_level (IPPROTO_SCTP each structure), the type name that should be set in cmsg_type and the structure name of the data buffer. In the description you will also find the name of the socket option that should be enabled in order to receive the data with recvmsg(). Please note that there is no guarantee for the order of the control messages, so don't rely on it.

Basically we can divide the ancillary data types in two groups - parameters you can get with recvmsg() and parameters you can set with sendmsg(). The specification lists some deprecated data types that I will not review here.

Ancillary data that can be received with recvmsg()

Again don't forget to enable each ancillary data type from this section with its corresponding socket option.

SCTP Receive Information Structure (SCTP_RCVINFO)

This control message is defined in Section 5.3.5. It contains various SCTP related parameters for the received message like stream number, TSN, etc.

You need to enable SCTP_RECVRCVINFO socket option. cmsg_type is SCTP_RCVINFO. cmsg_data is struct sctp_rcvinfo.

The parameters of the structure are:

  • uint16_t rcv_sid - stream number
  • uint16_t rcv_ssn - stream sequence number
  • uint16_t rcv_flags - there is only one flag specified - SCTP_UNORDERED
  • uint32_t rcv_ppid - payload protocol ID
  • uint32_t rcv_tsn - TSN of the DATA chunk
  • uint32_t rcv_cumtsn - cumulative TSN
  • uint32_t rcv_context - context which was set with SCTP_CONTEXT socket option.
  • sctp_assoc_t rcv_assoc_id - the ID of the association, also known as association handle.

If you don't know what each parameter means you can check my post about SCTP data transfer.

SCTP Next Receive Information Structure (SCTP_NXTINFO)

This control message contains information of the next message that will be delivered with the next call of recvmsg(). It will be delivered only in case there is an incoming buffered message. Message structure is defined in Section 5.3.6. To receive this control message you need to enable SCTP_RECVNXTINFO socket option. cmsg_data will contain struct sctp_nxtinfo. The parameters are similar to SCTP_RCVINFO:

  • uint16_t nxt_sid - stream number
  • uint16_t nxt_flags - SCTP_UNORDERED when the next message is unordered, SCTP_COMPLETE when entire message is received, SCTP_NOTIFICATION when the next message is notification
  • uint32_t nxt_ppid - payload protocol ID
  • uint32_t nxt_length - the length of the message currently in the message buffer. Please note that if partial delivery is in progress this is not the length of the entire message. Only if SCTP_COMPLETE flag is set, nxt_length represents the length of the entire message.
  • sctp_assoc_t nxt_assoc_id - association handle.

Ancillary data that can be passed to sendmsg()

These control messages doesn't require any specific socket options. You directly pass them to sendmsg().

SCTP Initiation Structure (SCTP_INIT)

This control message sets some parameters used in association initialisation with sendmsg(). It is defined in Section 5.3.1. cmsg_type is SCTP_INIT and struct sctp_initmsg should be added in cmsg_data. Its parameters are:

  • uint16_t sinit_num_ostreams - max number of outbound streams
  • uint16_t sinit_max_instreams - max number of inbound streams
  • uint16_t sinit_max_attempts - how many attempts to send INIT should be made
  • uint16_t sinit_max_init_timeo - the largest timeout for INIT retransmission in milliseconds.

For more information about SCTP association initialisation check this post.

SCTP Send Information Structure (SCTP_SNDINFO)

This is the alternative of SCTP_RCVINFO. It is defined in Section 5.3.4. Its cmsg_type is SCTP_SNDINFO and struct sctp_sndinfo should be added in cmsg_data. Its parameters are identical to the ones in SCTP_RCVINFO:

  • uint16_t snd_sid - stream id

  • uint16_t snd_flags
    • SCTP_UNORDERED - requests unordered delivery
    • SCTP_ADDR_OVER - overrides the default destination IP address with the one specified in sendmsg() call. Only for one-to-many style
    • SCTP_ABORT - causes ABORT to be sent to the peer with error cause 'User initiated abort'
    • SCTP_EOF - causes graceful shutdown of the association
    • SCTP_SENDALL - for one-to-many style causes the message to be sent to all established associations
  • uint32_t snd_ppid - payload protocol ID

  • uint32_t snd_context - in case of sending error this context is returned to the application with recvmsg() call

  • sctp_assoc_t snd_assoc_id - association handle. You can obtain this value either with SCTP_RCVINFO control message or with SCTP_COMM_UP notification.

SCTP PR-SCTP Information Structure (SCTP_PRINFO)

This control message specifies options for partial reliable SCTP. It is defined in Section 5.3.7, its cmsg_type is SCTP_PRINFO and struct sctp_prinfo should be added in cmsg_data. You can read more about partial reliable SCTP in RFC 3758.

SCTP AUTH Information Structure (SCTP_AUTHINFO)

This control message is defined in Section 5.3.8 and allows setting different preshared key identifier with sendmsg(). I won't discuss this in detail so if you are interested please check RFC 4895. The cmsg_type is SCTP_AUTHINFO and the corresponding structure is struct sctp_authinfo. It has got only one parameter uint16_t auth_keynumber which is the ID of the key.

SCTP Destination IPv4 Address Structure (SCTP_DSTADDRV4) and SCTP Destination IPv6 Address Structure (SCTP_DSTADDRV6)

They are defined in Section 5.3.9 and Section 5.3.10. With them you can provide more than one destination IPv4/IPv6 addresses and utilise the multi-homing feature of SCTP. cmsg_type is SCTP_DSTADDRV4 (for IPv4 address) and SCTP_DSTADDRV6. The corresponding data types are struct in_addr and struct in6_addr.

The code

First of all you need to gain access to a FreeBSD machine. One easy way is to use vagrant. There are ready to use FreeBSD boxes so you will have working virtual machine in minutes. The code for this post is in branch one-to-many_ancillary. You can switch to it with this command:

git checkout one-to-many_ancillary

The server

The server application enables two control messages - sctp_rcvinfo and sctp_nxtinfo. The code that enables them is in the function enable_ancillary_data():

int enable_ancillary_data(int fd)
{
    int boolean_as_int = 1;
    if(setsockopt(fd, PROTO, SCTP_RECVRCVINFO, &boolean_as_int, sizeof(boolean_as_int))) {
        printf("Error setting SCTP_RECVRCVINFO\n");
        return 1;
    }

    if(setsockopt(fd, PROTO, SCTP_RECVNXTINFO, &boolean_as_int, sizeof(boolean_as_int))) {
        printf("Error setting SCTP_RECVNXTINFO\n");
        return 2;
    }

    return 0;
}

Both socket options use integer, which is threated as bool variable, so we set it to 1.

After that we need to provide buffer for the control messages. This is performed just before the recvmsg() call in server's get_message() function. Remember that you have got CMSG_SPACE macro to help you with the buffer size calculation. Here is the code that calculates the buffer size for our case:

char* ancillary_data = NULL;
int ancillary_data_len = CMSG_SPACE(sizeof(struct sctp_rcvinfo)) + CMSG_SPACE(sizeof(struct sctp_nxtinfo));
if((ancillary_data = malloc(ancillary_data_len)) == NULL) {
    printf("Error allocating memory\n");
    return 1;
}
memset(ancillary_data, 0, ancillary_data_len);

And finally don't forget to set the buffer in struct msghdr:

struct msghdr msg;
memset(&msg, 0, sizeof(struct msghdr));
msg.msg_iov = &io_buf;
msg.msg_iovlen = 1;
msg.msg_name = sender_addr;
msg.msg_namelen = sizeof(struct sockaddr_in);
msg.msg_control = ancillary_data;
msg.msg_controllen = ancillary_data_len;

Now we are ready to accept control messages with recvmsg(). The ancillary data handling is implemented in separate function handle_ancillary_data():

void handle_ancillary_data(struct msghdr* msg)
{
    struct cmsghdr *cmsgptr;
    for(cmsgptr = CMSG_FIRSTHDR(msg); cmsgptr != NULL; cmsgptr = CMSG_NXTHDR(msg, cmsgptr)) {
        if (cmsgptr->cmsg_len == 0) {
            printf("Error handling ancillary data!\n");
            break;
        }
        if(cmsgptr->cmsg_level == PROTO && cmsgptr->cmsg_type == SCTP_RCVINFO) {
            u_char *ptr = CMSG_DATA(cmsgptr);
            struct sctp_rcvinfo* ri = (struct sctp_rcvinfo*)ptr;

            printf("SCTP_RCVINFO: sid: %u, ssn: %u, flags: %u, ppid: %u, tsn: %u, cumtsn: %u, context: %u, assoc_id: %d\n",
                   ri->rcv_sid, ri->rcv_ssn, ri->rcv_flags, ri->rcv_ppid, ri->rcv_tsn, ri->rcv_cumtsn, ri->rcv_context, ri->rcv_assoc_id);
        }
        else if(cmsgptr->cmsg_level == PROTO && cmsgptr->cmsg_type == SCTP_NXTINFO) {
            u_char *ptr = CMSG_DATA(cmsgptr);
            struct sctp_nxtinfo* nri = (struct sctp_nxtinfo*)ptr;

            printf("SCTP_NXTINFO: sid: %u, flags: %u, ppid: %u, length: %u, assoc_id: %d\n",
                   nri->nxt_sid, nri->nxt_flags, nri->nxt_ppid, nri->nxt_length, nri->nxt_assoc_id);
        }
    }
}

We iterate all the messages with a for similar to the one in the code snippets section. Then we check for the two control messages we are interested in and ignore everything else. For each message the stream id, stream sequence number, etc is logged on the screen. You can verify with a packet analyser if the values match with the ones in the DATA chunks. You also see information for the next packet in the buffer, if it is present. There is a little hack for this in the client code, described in the next section.

The client

The client sends two additional control messages with sendmsg() - sctp_initmsg and sctp_sndinfo. No socket options are required to enable this messages so there aren't any major modifications in main(). The only difference is that first all the requests are sent and then their respective answers are received (the two for loops calling send_message() and get_reply()). The server on the other hand responds to each request immediately which adds little delay in the processing. This little cheat allows us to demonstrate how sctp_nxtinfo works, because the server processes the messages slower than the client sends them. get_reply() also hasn't got any modifications so we will skim through to send_message(). The buffer allocation procedure is the same for the control messages sending - CMSG_SPACE is your best friend here:

char* ancillary_data = NULL;
int ancillary_data_len = CMSG_SPACE(sizeof(struct sctp_initmsg)) + CMSG_SPACE(sizeof(struct sctp_sndinfo));
if((ancillary_data = malloc(ancillary_data_len)) == NULL) {
    printf("Error allocating memory\n");
    return 1;
}
memset(ancillary_data, 0, ancillary_data_len);

The address of ancillary_data buffer is saved in struct msghdr's msg_control parameter and the buffer size in msg_controllen. Now all we need to do is to fill in the data structures with the parameters. We use CMSG_FIRSTHDR to access the first data structure and CMSG_NXTHDR for each consequent one:

struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = PROTO;
cmsg->cmsg_type = SCTP_INIT;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct sctp_initmsg));
struct sctp_initmsg* init_dt = (struct sctp_initmsg*) CMSG_DATA(cmsg);
init_dt->sinit_max_attempts = 0;
init_dt->sinit_max_init_timeo = 0;
init_dt->sinit_max_instreams = 5;
init_dt->sinit_num_ostreams = 6;

cmsg = CMSG_NXTHDR(&msg, cmsg);
cmsg->cmsg_level = PROTO;
cmsg->cmsg_type = SCTP_SNDINFO;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct sctp_sndinfo));
struct sctp_sndinfo* sndinfo_dt = (struct sctp_sndinfo*) CMSG_DATA(cmsg);
sndinfo_dt->snd_sid = 3;
sndinfo_dt->snd_flags = 0;
sndinfo_dt->snd_ppid = htonl(8);
sndinfo_dt->snd_context = 3;
sndinfo_dt->snd_assoc_id = 0;

You can verify with a packet analyser or the log from the server that the values are equal to the one we set with the client. Don't forget that these parameters are applied only to the DATA chunks used to transmit the payload in msg_iov buffer of current struct msghdr.

Conclusion

This post is a bit longer than the normal, but I wanted to include all ancillary data control messages for your and mine reference. I hope at this point you agree that working with ancillary data is pretty straight forward. Let me outline again the most important things to remember:

  1. Use CMSG_SPACE to calculate how much space you need for a given control message.
  2. Use CMSG_LEN to fill in the structure length in struct cmsg's cmsg_len field.
  3. Use CMSG_FIRSTHDR to access the first data structure from the control message buffer.
  4. Use CMSG_NEXTHDR to access each consequent data structure in the buffer.
  5. Always iterate the control buffer with a loop and CMSG_FIRSTHDR / CMSG_NEXTHDR macroses.
  6. Don't forget to enable the control messages for recvmsg() with their respective socket options.
  7. Validate your data! Especially if you receive it from your user (upper layer) or the network.
  8. If some control message you need is not supported on your system, check if there is a notification that can do the job for you.

That's all for now. Thank you for reading!

Comments

Comments powered by Disqus