Working with SCTP ancillary data in FreeBSD

Introduction

The socket interface provides good generalized interface for the underlying networking protocol, which is exactly what one programmer needs in most of the time. However for some more specific use cases someone might want to exploit specific protocol features, which can't present in the generalized interface. Ancillary data allows the socket interface to handle such specific protocol parameters/features, otherwise unaccessible from the socket API. Today we will add some ancillary data support to the one-to-many style application from the previous post.

Before digging right into the ancillary data I believe I owe you an explanation. My initial intention was to demonstrate the SCTP interface in Linux and as you have probably noticed, up to now I wrote only for Linux. However it turned out that not all ancillary data control messages are supported out of the box in the Linux distribution I currently use (Ubuntu 14.04.2 LTS with kernel version 3.16.0-36). Because the SCTP stack in Linux was ported from FreeBSD I decided to see what is supported there. Luckily all control messages described in Section 5.3 from RFC 6458 were supported in FreeBSD. That's why for this post I will use this operating system.

If you need to use some SCTP specific features in your Linux powered project I suggest you to use sctp_sendmsg, sctp_connectx, etc.

How ancillary data works

In struct msghdr there are two fields which I ignored up to now - void* msg_control and size_t msg_controllen. The first one points to preallocated buffer, which holds ancillary data structures and the second one is its length. This buffer is filled in with the ancillary data structures when recvmsg() is called. You receive only explicitly enabled (with socket option) parameters. The same buffer is also used to pass additional SCTP parameters to sendmsg() without the requirement to enable additional socket options.

Each ancillary data structure is struct cmsghdr, which is defined in RFC 3542 Section 20.2 like this:

struct cmsghdr {
    socklen_t  cmsg_len;   /* #bytes, including this header */
    int        cmsg_level; /* originating protocol */
    int        cmsg_type;  /* protocol-specific type */
               /* followed by unsigned char cmsg_data[]; */
};

cmsg_len is the size of the structure, including the header. cmsg_level defines the protocol, which in case of SCTP is IPPROTO_SCTP. cmsg_type is the type of the data structure, which contains the parameters. cmsg_data is buffer, which contains the data structure itself. Because of the different size of each ancillary data structure, additional padding might be added before cmsg_data. I strongly advice you to read RFC 3542 Section 20.2, which explains very well how struct cmsghdr is populated.

In both cases (sending and receiving) msg_control can contain more than one control message, which makes memory handling a bit tedious. For convenience there are CMSG_XXX macroses which can help you to manipulate msg_control buffer correctly. They defined in RFC 3542 Section 20.3:

• CMSG_FIRSTHDR - its declaration is equivalent to:

struct cmsghdr *CMSG_FIRSTHDR(const struct msghdr *mhdr);

The input parameter is pointer to struct msghdr and returns pointer to the first struct cmsghdr in msghdr's msg_control buffer.

• CMSG_NXTHDR - its declaration is equivalent to:

struct cmsghdr *CMSG_NXTHDR(const struct msghdr *mhdr, const struct cmsghdr *cmsg);

The input parameters are pointer to struct msghdr and pointer to struct cmsghdr. It returns pointer to the next struct cmsghdr in msg_control buffer.

• CMSG_DATA - its declaration is equivalent to:

unsigned char *CMSG_DATA(const struct cmsghdr *cmsg);

The input parameter is pointer to struct cmsghdr. It returns pointer to the actual data structure in cmsg_data. The return type is unsigned char and it has to be casted to the correct data type.

• CMSG_SPACE - its declaration is equivalent to:

socklen_t CMSG_SPACE(socklen_t length);

The input parameter is the length of the structure that should be kept in cmsg_data field. The return value is the required size of msg_control buffer for this ancillary data type. The length returned includes the space required for the control message, its corresponding struct cmsghdr and the padding. This function is usually used when the size of msg_control buffer is calculated. This function should not be used to fill in cmsg_len in struct cmsghdr.

• CMSG_LEN - its declaration is equivalent to:

socklen_t CMSG_LEN(socklen_t length);

The input parameter is the length of ancillary data structure and it returns the length, that should be saved in cmsg_len (in struct cmsghdr). Please note the difference between this function and CMSG_SPACE. CMSG_LEN returns the length of struct cmsghdr, while CMSG_SPACE returns the required length for struct cmsghdr AND the padding after it. If this is still not completely clear for you, have a look at the diagram in RFC 3542 Section 20.2. It shows what each length represents.

Code snippets showing how to accomplish a few common tasks

Now let's see some examples. All of them are extracted from the sample code that we will review later.

1. Iterating over the buffer msg_control buffer and processing SCTP_RCVINFO (more on this data type later):

struct msghdr* msg;         //Let's pretend that this is the pointer to the struct msghdr, returned from recvmsg()
struct cmsghdr *cmsgptr;
for(cmsgptr = CMSG_FIRSTHDR(msg); cmsgptr != NULL; cmsgptr = CMSG_NXTHDR(msg, cmsgptr)) {
    if (cmsgptr->cmsg_len == 0) {
        //this is error!
        break;
    }
    if(cmsgptr->cmsg_level == IPPROTO_SCTP && cmsgptr->cmsg_type == SCTP_RCVINFO) {
        u_char *ptr = CMSG_DATA(cmsgptr);       //Remember that CMSG_DATA returns unsigned char* ...
        struct sctp_rcvinfo* ri = (struct sctp_rcvinfo*)ptr;    // ... that you cast to the desired data type
        //Process the data here
    }
}

2. Dynamically allocating memory for msg_control buffer for two ancillary data structures (SCTP_RCVINFO and SCTP_NXTINFO)

char* ancillary_data = NULL;
int ancillary_data_len = CMSG_SPACE(sizeof(struct sctp_rcvinfo))        //for SCTP_RCVINFO
                            + CMSG_SPACE(sizeof(struct sctp_nxtinfo));  //for SCTP_NXTINFO

if((ancillary_data = malloc(ancillary_data_len)) == NULL) {
    //this is error
}
memset(ancillary_data, 0, ancillary_data_len);

struct msghdr msg;
memset(&msg, 0, sizeof(struct msghdr));
msg.msg_control = ancillary_data;
msg.msg_controllen = ancillary_data_len;
//continue with the other parameters

3. Prepare msg_control buffer and pass it to sendmsg()

//Let's pretend that this is the pointer to the struct msghdr that we are about to pass to sendmsg()
struct msghdr* msg;

//this is the first data structure. All consequent calls will be CMSG_NXTHDR(msg, cmsg)
struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = IPPROTO_SCTP;
cmsg->cmsg_type = SCTP_INIT;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct sctp_initmsg));
struct sctp_initmsg* init_dt = (struct sctp_initmsg*) CMSG_DATA(cmsg);
//fill in the parameters via init_dt pointer

//for all consequent data structures use CMSG_NXTHDR
cmsg = CMSG_NXTHDR(&msg, cmsg);
cmsg->cmsg_level = PROTO;
cmsg->cmsg_type = SCTP_SNDINFO;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct sctp_sndinfo));
struct sctp_sndinfo* sndinfo_dt = (struct sctp_sndinfo*) CMSG_DATA(cmsg);
//fill in the parameters via sndinfo_dt pointer

Ancillary data in SCTP protocol

SCTP ancillary data structures are defined in Section 5.3. There you can find the cmsg_level (IPPROTO_SCTP each structure), the type name that should be set in cmsg_type and the structure name of the data buffer. In the description you will also find the name of the socket option that should be enabled in order to receive the data with recvmsg(). Please note that there is no guarantee for the order of the control messages, so don't rely on it.

Basically we can divide the ancillary data types in two groups - parameters you can get with recvmsg() and parameters you can set with sendmsg(). The specification lists some deprecated data types that I will not review here.

The code

First of all you need to gain access to a FreeBSD machine. One easy way is to use vagrant. There are ready to use FreeBSD boxes so you will have working virtual machine in minutes. The code for this post is in branch one-to-many_ancillary. You can switch to it with this command:

git checkout one-to-many_ancillary

int enable_ancillary_data(int fd)
{
    int boolean_as_int = 1;
    if(setsockopt(fd, PROTO, SCTP_RECVRCVINFO, &boolean_as_int, sizeof(boolean_as_int))) {
        printf("Error setting SCTP_RECVRCVINFO\n");
        return 1;
    }

    if(setsockopt(fd, PROTO, SCTP_RECVNXTINFO, &boolean_as_int, sizeof(boolean_as_int))) {
        printf("Error setting SCTP_RECVNXTINFO\n");
        return 2;
    }

    return 0;
}

char* ancillary_data = NULL;
int ancillary_data_len = CMSG_SPACE(sizeof(struct sctp_rcvinfo)) + CMSG_SPACE(sizeof(struct sctp_nxtinfo));
if((ancillary_data = malloc(ancillary_data_len)) == NULL) {
    printf("Error allocating memory\n");
    return 1;
}
memset(ancillary_data, 0, ancillary_data_len);

struct msghdr msg;
memset(&msg, 0, sizeof(struct msghdr));
msg.msg_iov = &io_buf;
msg.msg_iovlen = 1;
msg.msg_name = sender_addr;
msg.msg_namelen = sizeof(struct sockaddr_in);
msg.msg_control = ancillary_data;
msg.msg_controllen = ancillary_data_len;

void handle_ancillary_data(struct msghdr* msg)
{
    struct cmsghdr *cmsgptr;
    for(cmsgptr = CMSG_FIRSTHDR(msg); cmsgptr != NULL; cmsgptr = CMSG_NXTHDR(msg, cmsgptr)) {
        if (cmsgptr->cmsg_len == 0) {
            printf("Error handling ancillary data!\n");
            break;
        }
        if(cmsgptr->cmsg_level == PROTO && cmsgptr->cmsg_type == SCTP_RCVINFO) {
            u_char *ptr = CMSG_DATA(cmsgptr);
            struct sctp_rcvinfo* ri = (struct sctp_rcvinfo*)ptr;

            printf("SCTP_RCVINFO: sid: %u, ssn: %u, flags: %u, ppid: %u, tsn: %u, cumtsn: %u, context: %u, assoc_id: %d\n",
                   ri->rcv_sid, ri->rcv_ssn, ri->rcv_flags, ri->rcv_ppid, ri->rcv_tsn, ri->rcv_cumtsn, ri->rcv_context, ri->rcv_assoc_id);
        }
        else if(cmsgptr->cmsg_level == PROTO && cmsgptr->cmsg_type == SCTP_NXTINFO) {
            u_char *ptr = CMSG_DATA(cmsgptr);
            struct sctp_nxtinfo* nri = (struct sctp_nxtinfo*)ptr;

            printf("SCTP_NXTINFO: sid: %u, flags: %u, ppid: %u, length: %u, assoc_id: %d\n",
                   nri->nxt_sid, nri->nxt_flags, nri->nxt_ppid, nri->nxt_length, nri->nxt_assoc_id);
        }
    }
}

char* ancillary_data = NULL;
int ancillary_data_len = CMSG_SPACE(sizeof(struct sctp_initmsg)) + CMSG_SPACE(sizeof(struct sctp_sndinfo));
if((ancillary_data = malloc(ancillary_data_len)) == NULL) {
    printf("Error allocating memory\n");
    return 1;
}
memset(ancillary_data, 0, ancillary_data_len);

struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = PROTO;
cmsg->cmsg_type = SCTP_INIT;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct sctp_initmsg));
struct sctp_initmsg* init_dt = (struct sctp_initmsg*) CMSG_DATA(cmsg);
init_dt->sinit_max_attempts = 0;
init_dt->sinit_max_init_timeo = 0;
init_dt->sinit_max_instreams = 5;
init_dt->sinit_num_ostreams = 6;

cmsg = CMSG_NXTHDR(&msg, cmsg);
cmsg->cmsg_level = PROTO;
cmsg->cmsg_type = SCTP_SNDINFO;
cmsg->cmsg_len = CMSG_LEN(sizeof(struct sctp_sndinfo));
struct sctp_sndinfo* sndinfo_dt = (struct sctp_sndinfo*) CMSG_DATA(cmsg);
sndinfo_dt->snd_sid = 3;
sndinfo_dt->snd_flags = 0;
sndinfo_dt->snd_ppid = htonl(8);
sndinfo_dt->snd_context = 3;
sndinfo_dt->snd_assoc_id = 0;

Conclusion

This post is a bit longer than the normal, but I wanted to include all ancillary data control messages for your and mine reference. I hope at this point you agree that working with ancillary data is pretty straight forward. Let me outline again the most important things to remember:

1. Use CMSG_SPACE to calculate how much space you need for a given control message.

2. Use CMSG_LEN to fill in the structure length in struct cmsg's cmsg_len field.

3. Use CMSG_FIRSTHDR to access the first data structure from the control message buffer.

4. Use CMSG_NEXTHDR to access each consequent data structure in the buffer.

5. Always iterate the control buffer with a loop and CMSG_FIRSTHDR / CMSG_NEXTHDR macroses.

6. Don't forget to enable the control messages for recvmsg() with their respective socket options.

7. Validate your data! Especially if you receive it from your user (upper layer) or the network.

8. If some control message you need is not supported on your system, check if there is a notification that can do the job for you.

That's all for now. Thank you for reading!