SCTP notifications in Linux
Introduction
In the previous post we have discussed the ancillary data - one of the ways to access SCTP specific protocol parameters and events. Today we will review another similar topic - the SCTP notifications. They are also received with recvmsg(), but you receive different kind of information. Ancillary data allowed us to get some SCTP specific parameters for the DATA chunk containing the payload, like stream number, association id, etc. With notifications you can access more general information about the association - e.g. receive event when it is established, when it is teared down, remote errors, etc.
SCTP notifications are described in Section 6 of RFC 6458. The implementation of Linux has got some really small differences from the specification, but they are in terms of structure/parameter names. I will try to list them, but even if I forget something you will easily notice them by yourself.
How notifications work
Each SCTP notification that you want to receive should be explicitly enabled with socket option. There are two ways to do that but more on this in the next section. When SCTP event occurs (and you are subscribed for it) it will be delivered with recvmsg(). The MSG_NOTIFICATION flag will be set in struct msghdr's msg_flags field. As for payload data you can check if the whole notification is delivered by checking if MSG_EOR flag is set. recvmsg() will always deliver only one notification per call.
Enabling SCTP notifications
Section 6 suggests two ways to enable the notifications - SCTP_EVENT and SCTP_EVENTS socket options. The recommended way is to use SCTP_EVENT (SCTP_EVENTS is even considered deprecated), however at the time of writing this option is not supported in Ubuntu 14.04 LTS with kernel 3.16.0-41-generic, which is the operating system I use for testing. For this reason I will explain how both socket options work, but I will use SCTP_EVENTS for the sample code.
SCTP_EVENTS is described in Section 6.2.1. The struct, which is passed to setsockopt(), is struct sctp_event_subscribe:
struct sctp_event_subscribe { uint8_t sctp_data_io_event; uint8_t sctp_association_event; uint8_t sctp_address_event; uint8_t sctp_send_failure_event; uint8_t sctp_peer_error_event; uint8_t sctp_shutdown_event; uint8_t sctp_partial_delivery_event; uint8_t sctp_adaptation_layer_event; uint8_t sctp_authentication_event; uint8_t sctp_sender_dry_event; };
There is one integer for each available notification which should be set to 1 in order to enable the corresponding notification. For now let's ignore what each event means. We will discuss this in detail later.
SCTP_EVENT is specified in Section 6.2.2 like this:
In this structure there are only three integers - se_assoc_id represents the association, se_type is the type of the notification and se_on should be set to 1 to enable it. If more than one notification should be enabled, setsockopt() should be called multiple times.
Now you can see the difference between the two options. SCTP_EVENTS has got predefined list of notifications that you can enable. If the SCTP stack developers decide to add new notification, they need to add another integer to the structure. This change in the API totally breaks the backward compatibility and is kind of problematic. With SCTP_EVENT the notification type is provided as a digit and you can easily add new notifications without changing the interface. The drawback is that you should call setsockopt() multiple times. But this is usually done in the application initialisation and it is not a big problem.
Notification structure
As mentioned when MSG_NOTIFICATION flag is set the msg_iov buffer will contain a notification. It is defined in Section 6.1 like an union of all possible events:
union sctp_notification { struct sctp_tlv { uint16_t sn_type; /* Notification type. */ uint16_t sn_flags; uint32_t sn_length; } sn_header; struct sctp_assoc_change sn_assoc_change; struct sctp_paddr_change sn_paddr_change; struct sctp_remote_error sn_remote_error; struct sctp_send_failed sn_send_failed; struct sctp_shutdown_event sn_shutdown_event; struct sctp_adaptation_event sn_adaptation_event; struct sctp_pdapi_event sn_pdapi_event; struct sctp_authkey_event sn_auth_event; struct sctp_sender_dry_event sn_sender_dry_event; struct sctp_send_failed_event sn_send_failed_event; };
As you see besides the events there is one more struct - sn_header. It is used to identify what notification is received and has got the following fields:
sn_type - the unique id of the event. More on this in the next section.
sn_flags - event-specific flags.
sn_length - the length of the whole sctp_notification structure (including sn_header).
You might be wondering why sn_header is part of the enum and how you can access the sn_header and the event at the same time. To answer this question, let's see how a random event is implemented. For example struct sctp_assoc_change (Section 6.1.1):
struct sctp_assoc_change { uint16_t sac_type; uint16_t sac_flags; uint32_t sac_length; uint16_t sac_state; uint16_t sac_error; uint16_t sac_outbound_streams; uint16_t sac_inbound_streams; sctp_assoc_t sac_assoc_id; uint8_t sac_info[]; };
Notice that the first three elements are exactly the same as in the union's sn_header (two uint16_t and one uint32_t). Each event has got the same three integers in its beginning. The size of the union in C is equal to the size of its biggest element. This means that when you receive a notification you can safely interpret it as struct sctp_tlv and access its type. After that you will know exactly what notification you have got and how to cast it. As you will see in the code section, my approach is a little bit different, but it follows the same idea.
Events
All events that can be received via notifications are listed in Section 6.1. Here, for each event you can find the following information:
Described in - the section in the specification dedicated to this event.
Flag in struct sctp_event_subscribe - Which integer should be set to 1 for SCTP_EVENTS socket option in order to receive this event. This information is taken from Section 6.2.1.
sn_type in union sctp_notification - There is a constant declared for each event type so you can determine the exact event by examining sn_header.sn_type member of the notification structure. The name of the constant is provided. This information can be found in Section 6.1.
Occurs when - What should happen from protocol point of view in order to receive the notification. This information can be found in the corresponding event section.
Parameters - Brief description of each member of the event struct. Also found in the corresponding event section.
SCTP_ASSOC_CHANGE
Described in: Section 6.1.1
Flag in struct sctp_event_subscribe: sctp_association_event
sn_type in union sctp_notification: SCTP_ASSOC_CHANGE
Occurs when: When the association is established or lost/teared down.
struct name in union sctp_notification: struct sctp_assoc_change
Parameters:
uint16_t sac_type - SCTP_ASSOC_CHANGE
uint16_t sac_flags - Not used.
uint32_t sac_length - Total length of the notification data and the header.
- uint16_t sac_state - Can be:
SCTP_COMM_UP - Association is established.
SCTP_COMM_LOST - Association failed.
SCTP_RESTART - Association restart is detected. This might occur if an association is established, the client side crashes, restarts and re-establishes the association using the same port numbers. In this case SCTP_RESTART will be generated on server side.
SCTP_SHUTDOWN_COMP - Graceful association shutdown.
SCTP_CANT_STR_ASSOC - Association establishment failed.
uint16_t sac_error - If the event is generated due to error (e.g. SCTP_COMM_LOST) this field corresponds to the protocol error code.
uint16_t sac_outbound_streams - Maximum outbound streams.
uint16_t sac_inbound_streams - Maximum inbound streams.
sctp_assoc_t sac_assoc_id - Association id.
uint8_t sac_info[] - If state is SCTP_COMM_LOST this field may contain the received ABORT chunk. If state is SCTP_COMM_UP or SCTP_RESTART the field might contain the features supported by the association. Check Section 6.1.1 for more information.
SCTP_PEER_ADDR_CHANGE
Described in: Section 6.1.2
Flag in struct sctp_event_subscribe: sctp_address_event
sn_type in union sctp_notification: SCTP_PEER_ADDR_CHANGE
Occurs when: When address of a multi-homed peer changes state.
struct name in union sctp_notification: struct sctp_paddr_change
Parameters:
uint16_t spc_type - SCTP_PEER_ADDR_CHANGE
uint16_t spc_flags - Not used.
uint32_t spc_length - Total length of the notification data and the header.
struct sockaddr_storage spc_aaddr - The affected remote IP address.
- uint32_t spc_state - Can be:
SCTP_ADDR_AVAILABLE - The address has become reachable.
SCTP_ADDR_UNREACHABLE - The address has become unreachable.
SCTP_ADDR_REMOVED - The address is removed from the association.
SCTP_ADDR_ADDED - The address is added to the association.
SCTP_ADDR_MADE_PRIM - The address is now primary for the association.
uint32_t spc_error - Protocol error code, if the event is generated due to an error.
sctp_assoc_t spc_assoc_id - Association id.
SCTP_REMOTE_ERROR
Described in: Section 6.1.3
Flag in struct sctp_event_subscribe: sctp_peer_error_event
sn_type in union sctp_notification: SCTP_REMOTE_ERROR
Occurs when: When ERR chunk is received from the peer.
struct name in union sctp_notification: struct sctp_remote_error
Parameters:
uint16_t spc_type - SCTP_REMOTE_ERROR
uint16_t spc_flags - Not used.
uint32_t spc_length - Total length of the notification data and the header.
uint16_t sre_error - The error cause from the ERR chunk (in network byte order).
sctp_assoc_t sre_assoc_id - Association id.
uint8_t sre_data[] - The entire ERR chunk.
SCTP_SEND_FAILED
This event is deprecated so I will not discuss it here. Use SCTP_SEND_FAILED_EVENT instead. If you still want to know more about it, check Section 6.1.4.
SCTP_SHUTDOWN_EVENT
Described in: Section 6.1.5
Flag in struct sctp_event_subscribe: sctp_shutdown_event
sn_type in union sctp_notification: SCTP_SHUTDOWN_EVENT
Occurs when: When SHUTDOWN is sent from the peer.
struct name in union sctp_notification: struct sctp_shutdown_event
Parameters:
uint16_t spc_type - SCTP_SHUTDOWN_EVENT
uint16_t spc_flags - Not used.
uint32_t spc_length - Total length of the notification data and the header.
sctp_assoc_t sse_assoc_id - Association id.
SCTP_ADAPTATION_INDICATION
Described in: Section 6.1.6
Flag in struct sctp_event_subscribe: sctp_adaptation_layer_event
sn_type in union sctp_notification: SCTP_ADAPTATION_INDICATION
Occurs when: When Adaptation Layer Indication is sent. Check RFC-5061 for more information.
struct name in union sctp_notification: struct sctp_adaptation_event
Parameters:
uint16_t spc_type - SCTP_ADAPTATION_INDICATION
uint16_t spc_flags - Not used.
uint32_t spc_length - Total length of the notification data and the header.
uint32_t sai_adaptation_ind - The bit array sent in Adaptation Layer Indication parameter.
sctp_assoc_t sai_assoc_id - Association id.
There is a whole document dedicated to this topic, but I will not dig in details. If you need this event please check RFC-5061 for more information.
SCTP_PARTIAL_DELIVERY_EVENT
Described in: Section 6.1.7
Flag in struct sctp_event_subscribe: sctp_partial_delivery_event
sn_type in union sctp_notification: SCTP_PARTIAL_DELIVERY_EVENT
Occurs when: When error occurs (e.g. association goes down) during partial delivery of a message.
struct name in union sctp_notification: struct sctp_pdapi_event
Parameters:
uint16_t spc_type - SCTP_ADAPTATION_INDICATION
uint16_t spc_flags - Not used.
uint32_t spc_length - Total length of the notification data and the header.
- uint32_t pdapi_indication - The indication being sent. The only possible value is:
SCTP_PARTIAL_DELIVERY_ABORTED - The partial delivery of a message is aborted for some reason.
uint32_t pdapi_stream - The stream on which the event happened.
uint32_t pdapi_seq - The stream sequence number that was partially delivered.
sctp_assoc_t pdapi_assoc_id - Association id.
Just a few more words on partial delivery. As you know SCTP is message based and the SCTP stack usually delivers whole messages to the user. There are cases however when the peer sends a message which is bigger than the size of the receiving SCTP buffer (bigger than predefined maximum size, to be precise). In this case partial delivery occurs. This means that the stack starts delivery of the message to the user in order to free up space in the receiving buffer. On FreeBSD you can control when partial delivery should be initiated with the SCTP_PARTIAL_DELIVERY_POINT socket option.
SCTP_AUTHENTICATION_EVENT
Described in: Section 6.1.8
Flag in struct sctp_event_subscribe: sctp_authentication_event
sn_type in union sctp_notification: SCTP_AUTHENTICATION_EVENT
Occurs when: When SCTP authentication event occurs. Check RFC-4895 for details.
struct name in union sctp_notification: struct sctp_authkey_event
Parameters:
uint16_t spc_type - SCTP_AUTHENTICATION_EVENT
uint16_t spc_flags - Not used.
uint32_t spc_length - Total length of the notification data and the header.
uint16_t auth_keynumber - Number of the affected key in the event.
- uint32_t auth_indication - The indication being reported. Possible values are:
SCTP_AUTH_NEW_KEY - Indicates the new active key.
SCTP_AUTH_NO_AUTH - Peer doesn't support authentication.
SCTP_AUTH_FREE_KEY - The key will no longer be used.
sctp_assoc_t auth_assoc_id - Association id.
This event is also specific and I will not dig in the details. Check RFC-4895 for more information.
SCTP_SENDER_DRY_EVENT
Described in: Section 6.1.9
Flag in struct sctp_event_subscribe: sctp_sender_dry_event
sn_type in union sctp_notification: SCTP_SENDER_DRY_EVENT
Occurs when: When the SCTP stack has got no more user data to send.
struct name in union sctp_notification: struct sctp_sender_dry_event
Parameters:
uint16_t spc_type - SCTP_SENDER_DRY_EVENT
uint16_t spc_flags - Not used.
uint32_t spc_length - Total length of the notification data and the header.
sctp_assoc_t sender_dry_assoc_id - Association id.
SCTP_NOTIFICATIONS_STOPPED_EVENT
Described in: Section 6.1.10
Flag in struct sctp_event_subscribe: n/a
sn_type in union sctp_notification: SCTP_NOTIFICATIONS_STOPPED_EVENT
Occurs when: When the SCTP stack disables the notifications due to lack of resources.
struct name in union sctp_notification: n/a
Parameters:
uint16_t spc_type - SCTP_NOTIFICATIONS_STOPPED_EVENT
uint16_t spc_flags - 0.
uint32_t spc_length - Total length of the notification data and the header.
This event is special. You are subscribed for it as soon as you subscribe any event type. It hasn't got any parameters - only the notification header. I am quoting Section 6.1.10 because I feel I can't describe this event better:
SCTP notifications, when subscribed to, are reliable. They are always delivered as long as there is space in the socket receive buffer. However, if an implementation experiences a notification storm, it may run out of socket buffer space. When this occurs, it may wish to disable notifications. If the implementation chooses to do this, it will append a final notification SCTP_NOTIFICATIONS_STOPPED_EVENT.
Remember that if you receive this event - all events you have subscribed for had been disabled. To receive them again you need to resubscribe with SCTP_EVENTS socket option.
SCTP_SEND_FAILED_EVENT
Described in: Section 6.1.11
Flag in struct sctp_event_subscribe: sctp_send_failure_event
sn_type in union sctp_notification: SCTP_SEND_FAILED_EVENT
Occurs when: When the SCTP stack can't deliver a message.
struct name in union sctp_notification: struct sctp_send_failed_event
Parameters:
uint16_t spc_type - SCTP_NOTIFICATIONS_STOPPED_EVENT
- uint16_t spc_flags - There are two flags:
SCTP_DATA_UNSENT - indicates that the data was never put on the wire.
SCTP_DATA_SENT - indicates that the data was put on the wire but this doesn't mean that it is successfully received from the peer.
uint32_t spc_length - Total length of the notification data and the header.
uint32_t ssfe_error - Protocol error code (the error clause in the ERR chunk).
struct sctp_sndinfo ssfe_info - The ancillary data used to send the message (struct sctp_sndinfo). This data is present even if you haven't explicitly enabled ancillary data. More about this topic in my previous post.
sctp_assoc_t ssfe_assoc_id - Association id.
uint8_t ssfe_data[] - The undelivered payload.
The code
As usual you can find the code on GitHub. You need to switch to one-to-many_notif branch.
There are some slight modifications to server and client code and two new functions in common.h. As I wrote, I cut a lot of corners with the sample code because I want to focus on SCTP itself. If you want to use it in your project please take some time to add at least proper error handling.
Enabling notifications
The code which enables notifications is located in separate function in common.h. It is used both by server and client:
int enable_notifications(int fd) { struct sctp_event_subscribe events_subscr; memset(&events_subscr, 0, sizeof(events_subscr)); events_subscr.sctp_association_event = 1; events_subscr.sctp_shutdown_event = 1; return setsockopt(fd, IPPROTO_SCTP, SCTP_EVENTS, &events_subscr, sizeof(events_subscr)); }
Nothing special here. New struct sctp_event_subscribe is created and memset-ed to 0 (to make sure everything is disabled). Then two events are enabled - association and shutdown events. This function is called by the server and the client just after the socket is created.
Detecting notifications
After recvmsg() returns both server and client check if the message is payload or notification. Here is how this is done in the client:
if(msg.msg_flags & MSG_NOTIFICATION) { if(!(msg.msg_flags & MSG_EOR)) { printf("Notification received, but the buffer is not big enough.\n"); continue; } handle_notification((union sctp_notification*)payload, recv_size); }
Notice that I check if the MSG_NOTIFICATION flag is set and if MSG_EOR is cleared and only then call handle_notification() which reads the notification. This check is only to demonstrate how MSG_EOR works and is not sufficient for production code. The correct behavior is to save the partially read notification in another buffer and to call recvmsg() again until MSG_EOR is set.
There is also one side effect. If the buffer is not big enough you will retrieve the notification via two or more recvmsg() calls and you will see two or more warnings. However the last recvmsg() call will have the MSG_EOR flag set and handle notification will be called for the last part of the notification. In this case sn_type will be set to zero and you will get another warning about unhandled notification type. I know this is lame, but again - I want to focus on SCTP protocol right now. The proper C programming part is from you.
Keep this things in mind when you write your code, because this implementation here is not good enough for production systems.
Reading notifications
handle_notification() is in common.h. It handles the enabled notifications:
int handle_notification(union sctp_notification *notif, size_t notif_len) { // http://stackoverflow.com/questions/20679070/how-does-one-determine-the-size-of-an-unnamed-struct int notif_header_size = sizeof( ((union sctp_notification*)NULL)->sn_header ); if(notif_header_size > notif_len) { printf("Error: Notification msg size is smaller than notification header size!\n"); return 1; } switch(notif->sn_header.sn_type) { case SCTP_ASSOC_CHANGE: { if(sizeof(struct sctp_assoc_change) > notif_len) { printf("Error notification msg size is smaller than struct sctp_assoc_change size\n"); return 2; } char* state = NULL; struct sctp_assoc_change* n = ¬if->sn_assoc_change; switch(n->sac_state) { case SCTP_COMM_UP: state = "COMM UP"; break; case SCTP_COMM_LOST: state = "COMM_LOST"; break; case SCTP_RESTART: state = "RESTART"; break; case SCTP_SHUTDOWN_COMP: state = "SHUTDOWN_COMP"; break; case SCTP_CANT_STR_ASSOC: state = "CAN'T START ASSOC"; break; } printf("SCTP_ASSOC_CHANGE notif: state: %s, error code: %d, out streams: %d, in streams: %d, assoc id: %d\n", state, n->sac_error, n->sac_outbound_streams, n->sac_inbound_streams, n->sac_assoc_id); break; } case SCTP_SHUTDOWN_EVENT: { if(sizeof(struct sctp_shutdown_event) > notif_len) { printf("Error notification msg size is smaller than struct sctp_assoc_change size\n"); return 3; } struct sctp_shutdown_event* n = ¬if->sn_shutdown_event; printf("SCTP_SHUTDOWN_EVENT notif: assoc id: %d\n", n->sse_assoc_id); break; } default: printf("Unhandled notification type %d\n", notif->sn_header.sn_type); break; } return 0; }
The function only logs the event on the screen. The outer switch checks the notification type (sn_header.sn_type). For SCTP_ASSOC_CHANGE event there is an inner switch, which checks the state of the association. For each notification type there are checks for the size of the data before each cast. Segmentation faults are not acceptable even for a demonstration project!
Conclusion
In this post we reviewed how SCTP notifications work. In my professional experience I've found them quite useful. As we spoke, SCTP is the preferred transport protocol for telecom applications. There are a lot of cases (for example M3UA protocol) which require you to know what is happening with the association and act accordingly. For example in M3UA you have to notify the local M3UA stack in case the association goes down. It is hard to implement this without notifications.
Thank you for reading and stay tuned for my next post.
The book
This post is part of my "SCTP in Theory and Practice:A quick introduction to the SCTP protocol and its socket interface in Linux" e-book. If you find the content in this post interesting - I think you will like it.
The book covers two topics - how SCTP works in theory and how to use it in Linux.
The best way to learn how SCTP works is to read and understand its specification - RFC 4960. However this document is not an easy read - the purpose of the document is to describe a full SCTP implementation and contains details which you usually don't need, unless you plan to write your own SCTP stack.
The role of the first five chapters of the book is to give you structured and easy to read explanation about how different parts of the protocol work. You will see how an SCTP association is established on network packet level, how data transfer works, how multi-homing is implemented and so on. Additionally each section contains references to specific sections from RFC 4960, which cover the topics in question. This approach will save you a lot of time reading the document.
The rest of the book focuses on SCTP from programmer point of view. You will learn how to write client-server applications in Linux. You will learn the difference between one-to-one and one-to-many style sockets and how to implement multi-homing. Each chapter contains working client and/or server implementation in C and line-by-line code review.
All source code and PCAP files used in the book are available as extra content.
Think you will like it? You can buy it on Leanpub. I really appreciate your support!
Comments
Comments powered by Disqus