Multi-homing in SCTP

Tsvetomir Dimitrov

2016-03-14 21:35

What is multihoming

In the previous posts about the SCTP protocol, I promised a separate article about multi-homing. I think we have covered most of the basic topics and now it is time to review this killer feature. The behaviour of a multi-homed SCTP node is scattered around RFC 4960 and in this post I will present the most important aspects.

I think Section 6.4 has got the best definition for multi-homing:

An SCTP endpoint is considered multi-homed if there are more than one transport address that can be used as a destination address to reach that endpoint.

What does this mean in practice? For example on fig. 1, host A and host B has got two routes between each other. With multi-homing you can add both paths to the association and in case of failure all the traffic will be transparently redirected to the other path.

How multi-homing works

During association initialisation each endpoint may announce list of additional IP address that can be used for communication. Remember that the port number is the same for all addresses in this list. One association can't use more than one port number. One of the paths, between the endpoints, is considered PRIMARY. The others are used either when the primary goes down or when the upper layer implicitly requires the message to be sent to another IP address. In the following sections we will review each phase from the association life and describe what happens when multi-homing is used.

Obtaining peer's IP addresses

If you need a refresher about SCTP association initialisation check this post. Section 5.1.2 explains how a SCTP endpoint can obtain the list with the IP addresses of its peer. There are three possibilities depending on the parameters included in the INIT/INIT ACK chunk:

No Host Name Address, IPv4 Address or IPv6 Address parameters present

There is no multi-homing in this scenario. The SCTP stack saves only the source IP address and port of the sender of the INIT/INIT ACK chunk.

Host Name Address parameter present

In this case there should be only one Host Name Address parameter. More than one is considered an error and if there are any additional IPv4 Address or IPv6 Address parameters they should be ignored. This constraint is defined on multiple places - Section 5.1.2 (subclause B), NOTE 3 in Section 3.3.2 and again NOTE 3 in Section 3.3.3.

Subclause B includes recommendations about when the hostname in Host Name Address parameter should be resolved. The receiver of the INIT chunk should do this after it receives COOKIE ECHO chunk, due to possible resource attack. This means that it should send the INIT ACK chunk to the IP address and the port from which the INIT was received. After the hostname is resolved, no other IP address should be used for data transfer. The receiver of INIT ACK chunk with Host Name Address parameter should resolve the hostname immediately and send COOKIE ECHO chunk to the resolved IP address.

In both cases, if the hostname can't be resolved the association initialisation should be terminated immediately with ABORT chunk. Check subclause B if the security considerations and error handling for this case are important for you.

IPv4 Address or IPv6 Address parameters are present

Unlike the Host Name Address, these parameters can occur more than once in the INIT/INIT ACK chunk. The receiver should record all IPv4/IPv6 addresses AND the IP address from which the INIT/INIT ACK was received. The receiver should use only these IP addresses for any further communication with its peer. However the INIT ACK chunk should always be sent to the IP address and port from which the INIT was received.

The sender of the INIT may additionally include Supported Address Types parameter to specify what IP address types it supports. If the receiver can't satisfy this requirement it should abort the association initialisation immediately.

For more information about Host Name Address, IPv4 Address and IPv6 Address you can also check Section 3.3.2.1.

Path verification

After the association is established, each endpoint knows the IP addresses of its peer. However a misbehaving (or malicious) endpoint may report incorrect IP addresses. To handle this, SCTP has to confirm each address before sending any messages to it. This is accomplished with the path verification procedure, described in Section 5.4. Initially each endpoint has got a set of confirmed addresses:

For the client (the sender of the INIT) these are the addresses, passed from the upper layer.
For the server (the receiver of COOKIE ECHO) this is the address from which the INIT was received.

All other IP addresses are considered unconfirmed. The verification of each address is done with a HEARTBEAT, sent to it. When HEARTBEAT ACK is received, the address is considered confirmed and can be used for data transfer. More details about the path verification can be found in Section 5.4.

Primary path

After all IP addresses are derived, one of them is selected as a PRIMARY and it will be the default for any further messages. This address can be changed by upper layer request, if it becomes unreachable or if the upper layer explicitly request the message to be sent to specific IP address. Usually the primary path is the one which was used to send/receive the INIT chunk (as described earlier).

Each SCTP endpoint should transmit reply chunks (like HEARTBEAT ACK, SACK, etc.) to the address from which the corresponding HEARTBEAT/DATA chunk was received, when this is possible. Exception can be made when for example the stack is sending SACK for multiple DATA chunks, received from different addresses. In this case the reply can be sent to any active IP address. Chunks, which have timed out, might also be retransmitted via different IP addresses. More examples about alternative path usages can be found in Section 6.4.

Each SCTP endpoint should monitor its peer addresses via HEARTBEATs. Once an address becomes unreachable it should be marked as inactive and a notification should be sent to the upper layer. For more details about remote address monitoring check Section 8.2.

Association termination

Association termination hasn't got any specifics related to multi-homing.

Example

Now let's see how multi-homing works in action. We will recreate the sample network setup on fig. 1 and review two cases - normal SCTP operation (when both paths remain available during the association lifetime) and primary path switching (when the link used for primary path goes down). I use VirtualBox and Vagrant for the simulation. You can get the Vagrantfile I use to recreate the network on fig. 1. The only difference is that hosts alpha and beta are directly connected (without routers) to each other, which is irrelevant for our case.

Normal operation

You can find the whole PCAP file here. Below I will use screenshots to show the important things. First let's see the whole communication on fig. 2.

Packets 1-4 are the association initialisation. 5-25 are data transfer and heartbeats and finally 26-28 - association tear down. Pay attention to the source IP addresses of DATA and SACK chunks. They are always the same - 192.168.35.10 and 192.168.35.11. The reason is that 192.168.35.10 <-> 192.168.35.11 is selected for Primary path. The link remains up during the association lifetime, so there is no need to use alternative paths for data transfer. Nevertheless there are HEARTBEAT chunks transferred over the second path (192.168.45.10 <-> 192.168.45.11) to make sure it is up.

Now let's have a look at the INIT chunk on fig. 3 and INIT ACK chunk on fig. 4. They are very similar, so I will review them together. The IP address related parameters are unfolded. The client (the sender of INIT) announces two IP addresses to the server - 192.168.35.11 and 192.168.45.11. However as we discussed in Path verification only 192.168.35.11 is considered verified by the server, because this is the source IP address of the INIT chunk (see the resume for IP protocol on fig. 3). The server announces 192.168.35.10 and 192.168.45.10 as its IP addresses in INIT ACK chunk. The message is sent from 192.168.35.10, so it is the only confirmed address for the client.

The trace confirms the rules described in Path verification. The primary path is 192.168.35.10 <-> 192.168.35.11 and it also is the default choice for data transfer. It is not monitored with HEARTBEATs, because there are acknowledged DATA chunks transferred over it. On the other hand over the second path (192.168.45.10 <-> 192.168.45.11) we can see occasional heartbeats, which assures the endpoint that the path is still available.

Primary path switching

This time I will simulate failure on the link used for the primary path, which will force the SCTP stack to select another path. I simulate the failure by rejecting all incoming SCTP traffic via the Ethernet interface used by the primary path on the client and the server. In my case this is eth1 and I block the traffic with iptables:

iptables -I INPUT -p sctp -i eth1 -j REJECT

On fig. 5 you can see the recorded PCAP file. I have added a few new columns - TSN (for DATA chunks), Cumulative TSN ACK (for SACK) and the payload itself (for DATA chunks). This will help us spot the retransmissions.

I block the SCTP traffic somewhere between packets 11 and 12 and packet 13 (ICMP Destination unreachable) is the first indication that there is something wrong with the link. Immediately a HEARTBEAT (packet 14) is sent on the other link and HEARTBEAT ACK (packet 15) is received. This means that the second link is operational so the server resends the lost DATA chunk (packet 17). Notice that the TSNs of packets 12 and 17 are the same, which indicates retransmission. The client confirms the reception of the DATA chunk with a SACK (packet 18).

Meanwhile the server continues to monitor the first link with HEARTBEATs (packets 19 and 21). The response is still ICMP Destination unreachable, so the communication continues over the second link (packets 23-28). After a while I unblock the link and we can see some acknowledged HEARTBEATs (packets 29-32). This is an indication for the SCTP stack that the PRIMARY link is up again and we can see that the rest of the chunks are transferred over it (packets 33 - 37).

Conclusion

Multi-homing is an interesting SCTP feature which lets you add an additional layer of redundancy for your application. For example this is quite useful for telecom applications, where the typical deployments include primary and backup links. Another option is to use the bandwidth of two links on application level. In this post I mainly demonstrated how the redundancy works. If you want to implement load-balancing via SCTP multi-homing, you will have to handle the routes selection from your application.

Thanks for reading and stay tuned for my next post, which will demonstrate how to use multi-homing in Linux.

The book

This post is part of my "SCTP in Theory and Practice:A quick introduction to the SCTP protocol and its socket interface in Linux" e-book. If you find the content in this post interesting - I think you will like it.

The book covers two topics - how SCTP works in theory and how to use it in Linux.

The best way to learn how SCTP works is to read and understand its specification - RFC 4960. However this document is not an easy read - the purpose of the document is to describe a full SCTP implementation and contains details which you usually don't need, unless you plan to write your own SCTP stack.

The role of the first five chapters of the book is to give you structured and easy to read explanation about how different parts of the protocol work. You will see how an SCTP association is established on network packet level, how data transfer works, how multi-homing is implemented and so on. Additionally each section contains references to specific sections from RFC 4960, which cover the topics in question. This approach will save you a lot of time reading the document.

The rest of the book focuses on SCTP from programmer point of view. You will learn how to write client-server applications in Linux. You will learn the difference between one-to-one and one-to-many style sockets and how to implement multi-homing. Each chapter contains working client and/or server implementation in C and line-by-line code review.

All source code and PCAP files used in the book are available as extra content.

Think you will like it? You can buy it on Leanpub. I really appreciate your support!