Get TCP Infomation Using 'getsockopt'

When troubleshooting network problems, it is very common to need to obtain information about TCP connections, such as amount of data been transmitted, retransmission rates, ACK delay, etc. Although some simple statistical functions can be implemented by hand, for example, recording functions can be added before all send and recv functions calls to calculate the number of bytes sent and received. However, this approach is still not very elegant. If eBPF is used, it is inevitable to involve the kernel space, which is very troublesome. Therefore, if you do not need to consider the portability to other UNIX systems, you can use interface specific to Linux.

Interface

int getsockopt(int sockfd, int level, int optname,
               void *restrict optval, socklen_t *restrict optlen);

When obtaining TCP connection information, fill in level with IPPROTO_TCP and optname with TCP_INFO. Fill in optval with the pointer to the buffer used to store the result. And optlen is both an input and output parameter used to pass the size of the buffer, and return the size of the result.

The Struct

For TCP information, the output result is a struct tcp_info. However, there are two versions of the definition of this structure. The first version is in <netinet/tcp.h>:

struct tcp_info
{
  uint8_t   tcpi_state;
  uint8_t   tcpi_ca_state;
  uint8_t   tcpi_retransmits;
  uint8_t   tcpi_probes;
  uint8_t   tcpi_backoff;
  uint8_t   tcpi_options;
  uint8_t   tcpi_snd_wscale : 4, tcpi_rcv_wscale : 4;

  uint32_t  tcpi_rto;
  uint32_t  tcpi_ato;
  uint32_t  tcpi_snd_mss;
  uint32_t  tcpi_rcv_mss;

  uint32_t  tcpi_unacked;
  uint32_t  tcpi_sacked;
  uint32_t  tcpi_lost;
  uint32_t  tcpi_retrans;
  uint32_t  tcpi_fackets;

  /* Times. */
  uint32_t  tcpi_last_data_sent;
  uint32_t  tcpi_last_ack_sent; /* Not remembered, sorry.  */
  uint32_t  tcpi_last_data_recv;
  uint32_t  tcpi_last_ack_recv;

  /* Metrics. */
  uint32_t  tcpi_pmtu;
  uint32_t  tcpi_rcv_ssthresh;
  uint32_t  tcpi_rtt;
  uint32_t  tcpi_rttvar;
  uint32_t  tcpi_snd_ssthresh;
  uint32_t  tcpi_snd_cwnd;
  uint32_t  tcpi_advmss;
  uint32_t  tcpi_reordering;

  uint32_t  tcpi_rcv_rtt;
  uint32_t  tcpi_rcv_space;

  uint32_t  tcpi_total_retrans;
};

This version is compatible with other UNIX-like systems such as FreeBSD, providing less information.

The second version is Linux-specific and is located in <linux/tcp.h>, providing more comprehensive information. Because the fields in the front part are the same, only the additional fields are listed here.

struct tcp_info {
    // ...
    // Fields above are the same with the first definition

    __u64   tcpi_pacing_rate;
    __u64   tcpi_max_pacing_rate;
    __u64   tcpi_bytes_acked;    /* RFC4898 tcpEStatsAppHCThruOctetsAcked */
    __u64   tcpi_bytes_received; /* RFC4898 tcpEStatsAppHCThruOctetsReceived */
    __u32   tcpi_segs_out;       /* RFC4898 tcpEStatsPerfSegsOut */
    __u32   tcpi_segs_in;        /* RFC4898 tcpEStatsPerfSegsIn */

    __u32   tcpi_notsent_bytes;
    __u32   tcpi_min_rtt;
    __u32   tcpi_data_segs_in;  /* RFC4898 tcpEStatsDataSegsIn */
    __u32   tcpi_data_segs_out; /* RFC4898 tcpEStatsDataSegsOut */

    __u64   tcpi_delivery_rate;

    __u64   tcpi_busy_time;      /* Time (usec) busy sending data */
    __u64   tcpi_rwnd_limited;   /* Time (usec) limited by receive window */
    __u64   tcpi_sndbuf_limited; /* Time (usec) limited by send buffer */

    __u32   tcpi_delivered;
    __u32   tcpi_delivered_ce;

    __u64   tcpi_bytes_sent;     /* RFC4898 tcpEStatsPerfHCDataOctetsOut */
    __u64   tcpi_bytes_retrans;  /* RFC4898 tcpEStatsPerfOctetsRetrans */
    __u32   tcpi_dsack_dups;     /* RFC4898 tcpEStatsStackDSACKDups */
    __u32   tcpi_reord_seen;     /* reordering events seen */

    __u32   tcpi_rcv_ooopack;    /* Out-of-order packets received */

    __u32   tcpi_snd_wnd;        /* peer's advertised receive window after
                      * scaling (bytes)
                      */
};

As you can see, the struct tcp_info in <linux/tcp.h> has some important information that is not present in the first version, such as the number of bytes sent and received, which is very useful for traffic statistics.

Sample Code

#include <linux/tcp.h>

// int fd = ...;
struct tcp_info tcpi;
int bufsz = sizeof(tcpi);
int ret = getsockopt(fd, IPPROTO_TCP, TCP_INFO, &tcpi, &bufsz);
if (ret != 0) {
    // TODO: error handling
}
printf("fd: %d has received %lu bytes and sent %lu bytes.\n",
       fd, tcpi.tcpi_bytes_receiver, tcpi.tcpi_bytes_sent);