VirtualBoxにLagopus switchを入れてみる。

Posted on 2014/10/13(Mon) 01:50 in technical

Summary

Intel-DPDK 1.7.1 + Lagopus 0.1.1 on VirtualBoxです。

とりあえずLagopusの起動まで。

基本的には公式のQUICKSTART に従います。

VirtualBox configuration

Virtual machine configuration
OS	Ubuntu(64bit)
CPU	2 core（Lagopusを動かすには2コア以上必須）
Memory	768MB（とりあえずギリギリを狙う）
NIC1	インターネットに抜ける用: Intel PRO/1000 MT Desktop (82540EM)
NIC2	Lagopus用 1ポート目: Intel PRO/1000 MT Server (82545EM)　/　プロミスキャスモード: すべて許可
NIC3	Lagopus用 2ポート目: Intel PRO/1000 MT Server (82545EM)　/　プロミスキャスモード: すべて許可

警告

プロミスキャスモードが無いとVirtualBoxが生成したMAC以外はOSに届く前に破棄されるので、当然必要です。

NIC1は、aptとかを使うためにインターネット抜ける用なので、正直何でもいいです。

Lagopus用のポートは、 Intel-DPDKのサポートNIC　に合うようにすれば何でもいいと思います。

Initial setup

とりあえず Ubuntu-14.04.1-amd64-server を入れます。その辺は割愛します。適当に入れましょう。

Hugepages

Ubuntu-14.04.1であればカーネルサポートは既に入っているので、設定して再起動します。:

# vi /etc/default/grub
GRUB_CMDLINE_LINUX="hugepages=128"
# update-grub
# mkdir /mnt/huge
# vi /etc/fstab
nodev /mnt/huge hugetlbfs defaults 0 0
# reboot

警告

この時、搭載メモリ量を超える割り当てを設定すると起動時にPanicするので、メモリ量は手元の環境と相談してください。

一応、128ブロック（128*2MB=256MB）確保しておけば、Lagopusの起動確認は可能です。

再起動後、こんな感じになればOK:

$ grep -i huge /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:     128
HugePages_Free:      128
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Intel-DPDK install

Lagopusで使用するIntel-DPDKは個別にインストールする必要があります。

公式のQUICKSTART が1.6なので、最新版っぽい1.7.1を使ってみます。:

$ cd
$ sudo apt install make coreutils gcc binutils
$ wget http://dpdk.org/browse/dpdk/snapshot/dpdk-1.7.1.tar.gz
$ tar zxf dpdk-1.7.1.tar.gz
$ cd dpdk-1.7.1
$ export RTE_SDK=`pwd`
$ export RTE_TARGET="x86_64-native-linuxapp-gcc"
$ make config T=${RTE_TARGET}
$ make install T=${RTE_TARGET}

警告

Intel-DPDK 1.6の頃は "x86_64-default-linuxapp-gcc" でしたが、1.7では "x86_64-native-linuxapp-gcc" です。

出来上がったカーネルドライバをロードして、Intel-DPDKのテストアプリを動かして一応の確認をします。:

$ sudo modprobe uio
$ sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/igb_uio.ko
$ sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/rte_kni.ko
$ ./tools/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller' if=eth0 drv=e1000 unused=igb_uio *Active*
0000:00:08.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth1 drv=e1000 unused=igb_uio
0000:00:09.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio

Other network devices
=====================
<none>
$ sudo ./tools/dpdk_nic_bind.py -b igb_uio 0000:00:08.0
$ sudo ./tools/dpdk_nic_bind.py -b igb_uio 0000:00:09.0

テスト用アプリケーションの起動（動けばDPDK的にはOK）:

$ sudo ./x86_64-native-linuxapp-gcc/app/testpmd -c3 -n3 -- -i --nb-cores=1 --nb-ports=1
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Support maximum 64 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL:   cannot open VFIO container, error 2 (No such file or directory)
EAL: VFIO support could not be initialized
EAL: Setting up memory...
EAL: Ask a virtual area of 0xc00000 bytes
EAL: Virtual area found at 0x7f805e400000 (size = 0xc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f805e000000 (size = 0x200000)
EAL: Ask a virtual area of 0xdc00000 bytes
EAL: Virtual area found at 0x7f8050200000 (size = 0xdc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f804fe00000 (size = 0x200000)
EAL: Ask a virtual area of 0x1200000 bytes
EAL: Virtual area found at 0x7f804ea00000 (size = 0x1200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f804e600000 (size = 0x200000)
EAL: Requesting 128 pages of size 2MB from socket 0
EAL: TSC frequency is ~2631404 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: Master core 0 is ready (tid=6008e840)
EAL: Core 1 is ready (tid=4ddfe700)
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 8086:100e rte_em_pmd
EAL:   0000:00:03.0 not managed by UIO driver, skipping
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL:   probe driver: 8086:100f rte_em_pmd
EAL:   PCI memory mapped at 0x7f8060038000
EAL: PCI device 0000:00:09.0 on NUMA socket -1
EAL:   probe driver: 8086:100f rte_em_pmd
EAL:   PCI memory mapped at 0x7f8060018000
Interactive-mode selected
Configuring Port 0 (socket 0)
Port 0: 08:00:27:69:B8:82
Configuring Port 1 (socket 0)
Port 1: 08:00:27:3B:CE:77
Checking link statuses...
Port 0 Link Up - speed 1000 Mbps - full-duplex
Port 1 Link Up - speed 1000 Mbps - full-duplex
Done
testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: 08:00:27:69:B8:82
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 1000 Mbps
Link duplex: full-duplex
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 15
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
  strip on
  filter on
  qinq(extend) off

********************* Infos for port 1  *********************
MAC address: 08:00:27:3B:CE:77
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 1000 Mbps
Link duplex: full-duplex
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 15
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
  strip on
  filter on
  qinq(extend) off
testpmd> quit
Stopping port 0...done
Stopping port 1...done
bye...

通報

DPDKアプリはHugepagesの確保のためにroot権限が要るので、基本的に全部sudoです。

権限が無いとこうなります。:

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Support maximum 64 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL:   cannot open VFIO container, error 2 (No such file or directory)
EAL: VFIO support could not be initialized
EAL: Setting up memory...
EAL: map_all_hugepages(): open failed: Permission denied
EAL: Failed to mmap 2 MB hugepages
PANIC in rte_eal_init():
Cannot init memory
6: [./x86_64-native-linuxapp-gcc/app/testpmd() [0x4245c3]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f35fefabec5]]
4: [./x86_64-native-linuxapp-gcc/app/testpmd(main+0x24) [0x423e64]]
3: [./x86_64-native-linuxapp-gcc/app/testpmd(rte_eal_init+0x1c7a) [0x4800ea]]
2: [./x86_64-native-linuxapp-gcc/app/testpmd(__rte_panic+0xc1) [0x423d2f]]
1: [./x86_64-native-linuxapp-gcc/app/testpmd(rte_dump_stack+0x18) [0x487068]]
Aborted

さて、次からようやくLagopusです。

Lagopus switch install

とりあえず手順に従ってインストール:

$ cd
$ sudo apt install build-essential libexpat-dev libgmp-dev libncurses-dev \
  libssl-dev libpcap-dev byacc flex libreadline-dev \
  python-dev python-pastedeploy python-paste python-twisted git
$ git clone https://github.com/lagopus/lagopus
$ cd lagopus
$ git show
commit cb6cb41d86f677654e6f67c877b4af81bdd66846
Merge: be60ac5 33e1ab5
Author: Yoshihiro Nakajima <nakajima.yoshihiro@lab.ntt.co.jp>
Date:   Wed Oct 8 21:04:36 2014 +0900

    Lagopus 0.1.1

$ ./configure --with-dpdk-dir=${RTE_SDK}
$ make
$ sudo make install
$ sudo mkdir /etc/lagopus
$ sudo cp samples/lagopus.conf /etc/lagopus/
$ sudo cp src/config/operational.xml /etc/lagopus/
$ sudo cp src/config/configuration.xml /etc/lagopus/
$ cd /etc/lagopus

通報

コンパイルは「何かすげー警告出てるんだけど本当に大丈夫なのコレ...」って思ってると出来上がります。

警告

ちゃんとは調べてないんだけど、operational.xmlとconfiguration.xmlも無いと動かないはずなんだけど、公式QuickStart だとlagopus.confしかコピーしてないんだよな？

でも、 lagopus/mk/pkg_param.conf を見ると、明らかに要るっぽいんだよな？

で、今のところは全部同じディレクトリに置いて動かせばいいか、とか言って棚上げにしてる。

フォアグラウンドで動かすとき:

$ sudo lagopus -d -l /tmp/lagopus.log -- -c3 -n1 -- -p3
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Support maximum 64 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL:   cannot open VFIO container, error 2 (No such file or directory)
EAL: VFIO support could not be initialized
EAL: Setting up memory...
EAL: Ask a virtual area of 0xc00000 bytes
EAL: Virtual area found at 0x7f8747a00000 (size = 0xc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f8747600000 (size = 0x200000)
EAL: Ask a virtual area of 0xdc00000 bytes
EAL: Virtual area found at 0x7f8739800000 (size = 0xdc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f8739400000 (size = 0x200000)
EAL: Ask a virtual area of 0x1200000 bytes
EAL: Virtual area found at 0x7f8738000000 (size = 0x1200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f8737c00000 (size = 0x200000)
EAL: Requesting 128 pages of size 2MB from socket 0
EAL: TSC frequency is ~2630470 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: Master core 0 is ready (tid=4ca91840)
EAL: Core 1 is ready (tid=373fe700)
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 8086:100e rte_em_pmd
EAL:   0000:00:03.0 not managed by UIO driver, skipping
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL:   probe driver: 8086:100f rte_em_pmd
EAL:   PCI memory mapped at 0x7f874ca3b000
EAL: PCI device 0000:00:09.0 on NUMA socket -1
EAL:   probe driver: 8086:100f rte_em_pmd
EAL:   PCI memory mapped at 0x7f874ca1b000
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 8086:100e rte_em_pmd
EAL:   0000:00:03.0 not managed by UIO driver, skipping
Initializing NIC port 0 ...
Initializing NIC port 0 RX queue 0 ...
Initializing NIC port 0 TX queue 0 ...

Checking link statusPort 0 Link Up - speed 1000 Mbps - full-duplex
Initializing NIC port 1 ...
Initializing NIC port 1 RX queue 0 ...
Initializing NIC port 1 TX queue 0 ...

Checking link statusPort 1 Link Up - speed 1000 Mbps - full-duplex
Initialization completed.
NIC RX ports:
  port 0 (queue 0)
  port 1 (queue 0)

I/O lcore 1 (socket 0):
 RX ports:
  port 0 (queue 0)
  port 1 (queue 0)
 Output rings:
  0x7f87477daac0

Worker 0: lcore 1 (socket 0):
 Input rings:
  0x7f87477daac0
 Output rings per TX port
  port 0 (0x7f87477dcb40)
  port 1 (0x7f87477debc0)

NIC TX ports:
  0  1

I/O lcore 1 (socket 0):
 Input rings per TX port
 port 0
  worker 0, 0x7f87477dcb40
 port 1
  worker 0, 0x7f87477debc0

Ring sizes:
  NIC RX     = 1024
  Worker in  = 1024
  Worker out = 1024
  NIC TX     = 1024
Burst sizes:
  I/O RX (rd = 144, wr = 144)
  Worker (rd = 144, wr = 144)
  I/O TX (rd = 144, wr = 144)

Logical core 1 (io-worker 0) main loop.
Adding Physical Port 0
08:00:27:69:b8:82:
Adding Physical Port 1
08:00:27:3b:ce:77:
Assigning port id 0 to bridge br0
Assigning port id 1 to bridge br0
^C

バックグラウンドで動かすとき:

$ sudo lagopus -l /tmp/lagopus.log -- -c3 -n1 -- -p3
$ ps ax | grep [l]agopus
18208 ?        Ssl    0:07 lagopus -l /tmp/lagopus.log -- -c3 -n1 -- -p3

lagosh

Lagopus switchには操作用のCLI lagoshが付いてくる。

こんな感じで使う。（今回は雰囲気だけ）:

$ sudo lagosh
ubuntu1404-1> show bridge-domains
bridge: br0
  datapnath id: 12374.a9:e2:21:a7:fb:9e
  max packet buffers: 65535, number of tables: 255
  capabilities: flow_stats on, table_stats on, port_stats on, group_stats on
                ip_reasm off, queue_stats on, port_blocked off
  fail-mode: standalone-mode (default)
port: eth0: ifindex 0, OpenFlow Port 1
port: eth1: ifindex 1, OpenFlow Port 2
ubuntu1404-1> show interface eth0
eth0:
 Description:
  OpenFlow Port: 1
  Hardware Address: 08:00:27:69:b8:82
  PCI Address: 0000:00:08.0
  Config: no restricted
  State: LINK UP, LIVE
 Statistics:
  rx_packets: 0
  tx_packets: 0
  rx_bytes:   0
  tx_bytes:   0
  rx_dropped: 0
  tx_dropped: -1
  rx_error:   0
  tx_error:   0
ubuntu1404-1> show flow
Bridge: br0
 Table id: 0
ubuntu1404-1> show flowcache
Bridge: br0
  nentries: 0
  hit:      0
  miss:     0
ubuntu1404-1> configure
Entering configuration mode
[edit]
ubuntu1404-1# show
interface {
    ethernet {
        eth0;
        eth1;
    }
}
bridge-domains {
    br0 {
        port {
            eth0;
            eth1;
        }
        controller {
            127.0.0.1;
        }
    }
}
[edit]
ubuntu1404-1# exit
ubuntu1404-1> exit

警告

ちなみに、これもroot権限が無いとアクセス出来ないっぽい。

一般ユーザだとこうなる:

$ lagosh
ubuntu1404-1> show bridge-domains

% Can't open connection to lagopus

ubuntu1404-1> exit

ものぐさな人向けコピペシート

自動的にモジュールの読み込み、DPDKへのNICの登録をしていない時用のコピペシート（1行目はsudoのキャッシュ作る用）:

$ sudo ls
$ cd ~/dpdk-1.7.1
$ export RTE_SDK=`pwd`
$ export RTE_TARGET="x86_64-native-linuxapp-gcc"
$ sudo modprobe uio
$ sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/igb_uio.ko
$ sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/rte_kni.ko
$ sudo ./tools/dpdk_nic_bind.py -b igb_uio 0000:00:08.0
$ sudo ./tools/dpdk_nic_bind.py -b igb_uio 0000:00:09.0
$ cd /etc/lagopus
$ sudo lagopus -l /tmp/lagopus.log -- -c3 -n1 -- -p3

おしまい

とりあえずインストールしただけなので、特に面白いことは無いと思うので、まぁその、そういうことです。

上手く起動したら、今度はコントローラと接続して実際にSDNスイッチとして動かしてみるのが良いでしょう。

おわり。

Previous Post Next Post

Summary

VirtualBox configuration

Initial setup

Hugepages

Intel-DPDK install

Lagopus switch install

lagosh

ものぐさな人向けコピペシート

おしまい

Related articles: