Skip to content

Instantly share code, notes, and snippets.

@jfeilbach
Last active August 16, 2024 19:27
Show Gist options
  • Save jfeilbach/b4f30435e7757fde3183ea05a7e997f8 to your computer and use it in GitHub Desktop.
Save jfeilbach/b4f30435e7757fde3183ea05a7e997f8 to your computer and use it in GitHub Desktop.
10/40 Gb NIC Linux Kernel Performance Tuning for samba file server

TCP tuning

The most important TCP tuning areas since kernel 4.9 are:

  • packet pacing
  • dynamic TSO sizing
  • TCP small queues
  • BBR TCP congestion algorithm

Definitions

  • Gb = gigabit
  • GB = gigabyte
  • NIC = Network Interface Card e.g. eth0

Current paramnters

Get current congestion alorithms avaialble

sysctl net.ipv4.tcp_available_congestion_control

Get current congestion alorithm in use

sysctl net.ipv4.tcp_congestion_control

Alternative method

$ grep 'CONFIG_TCP_CONG_BBR' /boot/config-$(uname -r)
$ grep 'CONFIG_NET_SCH_FQ' /boot/config-$(uname -r)
$ egrep 'CONFIG_TCP_CONG_BBR|CONFIG_NET_SCH_FQ' /boot/config-$(uname -r)

Jumbo frames

ip link show | grep mtu

Verify MTU size

ping -s 8972 -M do -c 4 10.100.100.1

Use ethtool to get the number of descriptors for NIC

ethtool -g enp2s0f0

Get driver in use

ethtool -i enp2s0f0

1 Gb NIC

allow testing with buffers up to 16MB

net.core.rmem_max = 16777216 net.core.wmem_max = 16777216

For Samba

net.core.rmem_default = 4194304
net.core.wmem_default = 1048576
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_syncookies = 1

increase Linux autotuning TCP buffer limit to 16MB

net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

To enable Fair Queuing

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

or, if bbr unavailable. TCP BBR is supported by Linux since kernel version 4.9

net.ipv4.tcp_congestion_control=htcp

Jumbo frames

ip link set eth0 mtu 9000

NIC

/etc/modprobe.conf (assuming tg3):
alias eth0 e1000
options tg3 RxDescriptors=4096,4096 TxDescriptors=4096,4096
# or
/etc/rc.local 
ethtool -G ethN rx 4096 tx 4096

To verify

ethtool -g eth0

10 Gb

allow testing with buffers up to 64MB

net.core.rmem_max = 67108864 net.core.wmem_max = 67108864

increase Linux autotuning TCP buffer limit to 32MB

net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432

recommended default congestion control is htcp

net.ipv4.tcp_congestion_control=htcp

recommended for hosts with jumbo frames enabled

net.ipv4.tcp_mtu_probing=1

recommended for newer hosts. TSO is off by default should be disabled when using FQ

net.core.default_qdisc = fq

pace and shape the bandwidth

tc qdisc add dev $ETH root fq maxrate Ngbit

Fair Queuing. Turning on FQ for a 10G data transfer node with 4 parallel streams

tc qdisc add dev $ETH root fq maxrate 2.5gbit

Jumbo frames

ip link set eth0 mtu 9000

40 Gb

allow testing with buffers up to 128MB

net.core.rmem_max = 134217728
net.core.wmem_max = 134217728

increase Linux autotuning TCP buffer limit to 64MB

net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

recommended default congestion control is htcp

net.ipv4.tcp_congestion_control=htcp

recommended for hosts with jumbo frames enabled

net.ipv4.tcp_mtu_probing=1

recommended for CentOS7+/Debian8+ hosts

net.core.default_qdisc = fq

To set the default congestion control algorithm, do:

sysctl -w net.ipv4.tcp_congestion_control=htcp

If jumbo Frames enabled. Do not set tcp_mtu_probing = 2

tcp_mtu_probing = 1

References

Samba Tuning.

Samba on MacOS is pretty terrible by default compared to Windows. It will not autrefresh on remote changes unlike Windows. There are a few changes on the server side that can make life better for MacOS clients. Very large >10,000 directory entries will still be a bit slow but <1000 directory entires should be pretty fast with these changes. Most of the changes are documented here: https://wiki.samba.org/index.php/Configure_Samba_to_Work_Better_with_Mac_OS_X

Version

Check samba version. This is quite important. You can view the version changes at https://wiki.samba.org/index.php/Main_Page. For my Ubuntu server it looks like this:

$ samba --version
Version 4.11.6-Ubuntu
$uname -r
5.4.0-52-generic

/etc/samba/smb.conf

Apple extensions require support for extended attributes(xattr) - defaults to yes in Samba 4.9+ Load the necessary modules; the actual load order is important and also enable AAPL extensions

[global]
    min protocol = SMB2 
    ea support = yes
    vfs objects = catia fruit streams_xattr
    fruit:metadata = stream
    fruit:model = MacSamba
    fruit:veto_appledouble = no
    fruit:posix_rename = yes 
    fruit:zero_file_id = yes
    fruit:wipe_intentionally_left_blank_rfork = yes 
    fruit:delete_empty_adfiles = yes 
    unix charset = utf8
    vfs objects = fruit
    fruit:aapl = yes
    fruit:encoding = native
    fruit:locking = none
    fruit:resource = file
    fruit:time machine = no
    fruit:posix_rename = yes
    readdir_attr:aapl_rsize = yes
    readdir_attr:aapl_finder_info = yes
    readdir_attr:aapl_max_access = no
    mdns name = mdns

Number of open files.

MacOS benefits from increasing this value. The client can make less directory listing requests.

Get current values

Check both the hard limit and the soft limit.

ulimit -Hn -Sn or ulimit -a or cat /proc/sys/fs/file-max

Set it higher

Open files example

In the file /etc/security/limits.conf add the following line with a value that makes sense. This will change the value for all users on the system. * - nofile 16385

Example for all users with just hard limit

* hard nofile 4096

Example soft limit for max opened files * is for all users, You can also specifiy specific users by username

* soft nofile 1024

or in /etc/sysctl.conf add the following line:

fs.file-max=500000

and activate the recent changes:

sysctl -p

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment