Permalink: 2014-10-14 00:30:00+09:00 by ruy@ainoniwa.net in technical tags: lagopus virtualbox vswitch social:

Summary

前回 Lagopusを入れたので、とりあえずryu辺りを使ってL2SWとして動くところまでを確認します。

相変わらず 公式のQUICKSTART に従います。

構成図

こういう感じでひとつお願いします。

lagopus_test_run_001.png

ryuのインストール

Lagopusが入っているのと同じマシンに楽ちんインストールです。:

$ sudo apt-get install python-setuptools python-pip python-dev libxml2-dev libxslt-dev
$ sudo pip install ryu

と思いきや:

$ ryu-manager --version
Traceback (most recent call last):
  File "/usr/local/bin/ryu-manager", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2749, in <module>
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 446, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 459, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 632, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (six 1.5.2 (/usr/lib/python2.7/dist-packages), Requirement.parse('six>=1.7.0'))

ぎゃあ。:

$ sudo pip install -U six
$ ryu-manager --version
ryu-manager 3.13

おっけー。

コントローラの起動

こんな感じで。:

$ ryu-manager --log-file=/tmp/ryu.log /usr/local/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py &

Lagopusの起動

ものぐさコピペシートー:

cd
cd ~/dpdk-1.7.1
export RTE_SDK=`pwd`
export RTE_TARGET="x86_64-native-linuxapp-gcc"
sudo modprobe uio
sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/igb_uio.ko
sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/rte_kni.ko
sudo ./tools/dpdk_nic_bind.py -b igb_uio 0000:00:08.0
sudo ./tools/dpdk_nic_bind.py -b igb_uio 0000:00:09.0
cd /etc/lagopus
sudo lagopus -l /tmp/lagopus.log -- -c3 -n1 -- -p3

何か適当なコマンドでsudoでパスワード聞かれなくしたら、コピペで起動する。

ping打ってみる

node-01:

# ifconfig em1 192.168.0.1/24 up

node-02:

# ifconfig em1 192.168.0.2/24 up
# ping -c4 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=33.564 ms
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=13.000 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=13.890 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=14.771 ms

--- 192.168.0.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 13.000/18.806/33.564/8.543 ms

疎通は問題なさそうだ。

ryuのログも見てみる。:

$ cat /tmp/ryu.log
loading app /usr/local/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py
loading app ryu.controller.ofp_handler
instantiating app /usr/local/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py of SimpleSwitch13
instantiating app ryu.controller.ofp_handler of OFPHandler
packet in 5803077010208869558 08:00:27:e2:53:5a ff:ff:ff:ff:ff:ff 2
packet in 5803077010208869558 08:00:27:3b:72:00 08:00:27:e2:53:5a 1
packet in 5803077010208869558 08:00:27:e2:53:5a 08:00:27:3b:72:00 2

ふむ、よろしい。

では、lagopusのフローテーブルも確認しよう。:

$ sudo lagosh
ubuntu1404-1> show flow
Bridge: br0
 Table id: 0
  priority=1,idle_timeout=0,hard_timeout=0,flags=0,cookie=0,packet_count=4,byte_count=392,in_port=1,eth_dst=08:00:27:e2:53:5a actions=output:2
  priority=1,idle_timeout=0,hard_timeout=0,flags=0,cookie=0,packet_count=3,byte_count=294,in_port=2,eth_dst=08:00:27:3b:72:00 actions=output:1
  priority=0,idle_timeout=0,hard_timeout=0,flags=0,cookie=0,packet_count=3,byte_count=218 actions=output:controller

ちゃんとフローエントリに登録されているようだ。

netmapで負荷をかけてみる

BSDRPは最初からnetmap搭載なので、pkt-genを使用することが出来る。

もちろん、これは仮想環境なので十分な測定器としての性能は出ないのだが、pingだけでは少々物足りないので。

送信側はフレーム長512Byteで出せるだけ。

node-02(TX):

# pkt-gen -i em1 -f tx -l 512 -D 08:00:27:3b:72:00 -s 192.168.0.2 -d 192.168.0.1
582.957774 main [1705] interface is em1
582.960888 extract_ip_range [289] range is 192.168.0.2:0 to 192.168.0.2:0
582.960902 extract_ip_range [289] range is 192.168.0.1:0 to 192.168.0.1:0
583.000689 main [1896] mapped 94548KB at 0x801c00000
Sending on netmap:em1: 1 queues, 1 threads and 1 cpus.
192.168.0.2 -> 192.168.0.1 (00:00:00:00:00:00 -> 08:00:27:3b:72:00)
583.000766 main [1980] Sending 512 packets every  0.000000000 s
583.000773 main [1982] Wait 2 secs for phy reset
585.024614 main [1984] Ready...
585.025767 nm_open [456] overriding ifname em1 ringid 0x0 flags 0x1
585.026074 sender_body [1064] start, fd 4 main_fd 3
585.325608 sender_body [1133] drop copy
586.037434 main_thread [1502] 379033 pps (383345 pkts in 1011375 usec)
587.044306 main_thread [1502] 423423 pps (426329 pkts in 1006862 usec)
588.058373 main_thread [1502] 410826 pps (416610 pkts in 1014078 usec)
589.094597 main_thread [1502] 420336 pps (435546 pkts in 1036185 usec)
590.130362 main_thread [1502] 395297 pps (409444 pkts in 1035787 usec)
591.137019 main_thread [1502] 437230 pps (440148 pkts in 1006674 usec)
592.146696 main_thread [1502] 418719 pps (422748 pkts in 1009622 usec)
593.154509 main_thread [1502] 409806 pps (413019 pkts in 1007840 usec)
594.168906 main_thread [1502] 424308 pps (430423 pkts in 1014411 usec)
595.201132 main_thread [1502] 404090 pps (417117 pkts in 1032239 usec)
596.240903 main_thread [1502] 393294 pps (408926 pkts in 1039747 usec)
597.255694 main_thread [1502] 411032 pps (417116 pkts in 1014803 usec)
598.263879 main_thread [1502] 434547 pps (438099 pkts in 1008173 usec)
599.279539 main_thread [1502] 434879 pps (441687 pkts in 1015654 usec)
600.313926 main_thread [1502] 418084 pps (432469 pkts in 1034407 usec)
^C600.976396 sigint_h [326] received control-C on thread 0x801806800
600.976470 sender_body [1162] flush tail 2041 head 1018 on thread 0x801806800
601.325873 main_thread [1502] 261473 pps (264600 pkts in 1011958 usec)
Sent 6597626 packets, 512 bytes each, in 15.95 seconds.
Speed: 413.61 Kpps Bandwidth: 1.69 Gbps (raw 1.77 Gbps)

node-01(RX):

# pkt-gen -i em1 -f rx
578.422659 main [1705] interface is em1
578.423529 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0
578.423539 extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0
578.463447 main [1896] mapped 94564KB at 0x801c00000
Receiving from netmap:em1: 1 queues, 1 threads and 1 cpus.
578.463500 main [1982] Wait 2 secs for phy reset
580.499898 main [1984] Ready...
580.500224 nm_open [456] overriding ifname em1 ringid 0x0 flags 0x1
580.500377 receiver_body [1238] reading from netmap:em1 fd 4 main_fd 3
581.512074 receiver_body [1245] waiting for initial packets, poll returns 0 0
581.512113 main_thread [1502] 0 pps (0 pkts in 1011760 usec)
582.520785 main_thread [1502] 0 pps (0 pkts in 1008668 usec)
582.520843 receiver_body [1245] waiting for initial packets, poll returns 0 0
583.529179 receiver_body [1245] waiting for initial packets, poll returns 0 0
583.529499 main_thread [1502] 0 pps (0 pkts in 1008717 usec)
584.575076 main_thread [1502] 0 pps (0 pkts in 1045578 usec)
584.575122 receiver_body [1245] waiting for initial packets, poll returns 0 0
585.606672 main_thread [1502] 66382 pps (68479 pkts in 1031595 usec)
586.623319 main_thread [1502] 206749 pps (210191 pkts in 1016647 usec)
587.630316 main_thread [1502] 209486 pps (210952 pkts in 1006997 usec)
588.646089 main_thread [1502] 206478 pps (209735 pkts in 1015773 usec)
589.685415 main_thread [1502] 204498 pps (212540 pkts in 1039327 usec)
590.689263 main_thread [1502] 208670 pps (209473 pkts in 1003848 usec)
591.702000 main_thread [1502] 199973 pps (202520 pkts in 1012736 usec)
592.710373 main_thread [1502] 203329 pps (205028 pkts in 1008356 usec)
593.724733 main_thread [1502] 204628 pps (207570 pkts in 1014377 usec)
594.756339 main_thread [1502] 201510 pps (207879 pkts in 1031605 usec)
595.795702 main_thread [1502] 205698 pps (213795 pkts in 1039364 usec)
596.811223 main_thread [1502] 204670 pps (207847 pkts in 1015521 usec)
597.827079 main_thread [1502] 199453 pps (202615 pkts in 1015855 usec)
598.871887 main_thread [1502] 206255 pps (215494 pkts in 1044793 usec)
599.905812 main_thread [1502] 195547 pps (202184 pkts in 1033941 usec)
600.910418 main_thread [1502] 199519 pps (200438 pkts in 1004606 usec)
^C601.804637 sigint_h [326] received control-C on thread 0x801806800
601.920027 main_thread [1502] 63721 pps (64333 pkts in 1009602 usec)
Received 3251073 packets, in 16.53 seconds.
Speed: 196.62 Kpps

おおよその値だが、送信側は400Kpps、受信側は200Kpps程度出ている。

Open vSwitchと比べてみる

では、ここでLagopusを入れた仮想マシンにOpen vSwitchを入れて比較してみる。

Open vSwitchでポートを接続し、先程と同じようにryu-managerでsimple_switch_13.pyを起動する。:

$ sudo apt install openvswitch-switch
$ sudo reboot
$ sudo ovs-vsctl add-br br0
$ sudo ovs-vsctl add-port br0 eth1
$ sudo ovs-vsctl add-port br0 eth2
$ sudo ovs-vsctl set bridge br0 protocols=OpenFlow13
$ sudo ovs-vsctl set-controller br0 tcp:127.0.0.1:6633
$ ryu-manager --log-file=/tmp/ryu.log /usr/local/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py &

pingが飛ぶことを確認したらpkt-genで送信開始。

node-02(TX):

# ping -c4 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=16.935 ms
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=1.491 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=1.041 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.864 ms

--- 192.168.0.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.864/5.083/16.935/6.847 ms
# pkt-gen -i em1 -f tx -l 512 -D 08:00:27:3b:72:00 -s 192.168.0.2 -d 192.168.0.1
374.356559 main [1705] interface is em1
374.357108 extract_ip_range [289] range is 192.168.0.2:0 to 192.168.0.2:0
374.357126 extract_ip_range [289] range is 192.168.0.1:0 to 192.168.0.1:0
374.396577 main [1896] mapped 94548KB at 0x801c00000
Sending on netmap:em1: 1 queues, 1 threads and 1 cpus.
192.168.0.2 -> 192.168.0.1 (00:00:00:00:00:00 -> 08:00:27:3b:72:00)
374.397217 main [1980] Sending 512 packets every  0.000000000 s
374.397226 main [1982] Wait 2 secs for phy reset
376.423024 main [1984] Ready...
376.424440 nm_open [456] overriding ifname em1 ringid 0x0 flags 0x1
376.424628 sender_body [1064] start, fd 4 main_fd 3
376.651937 sender_body [1133] drop copy
377.437845 main_thread [1502] 547529 pps (554780 pkts in 1013244 usec)
378.443151 main_thread [1502] 572720 pps (575762 pkts in 1005312 usec)
379.460618 main_thread [1502] 577950 pps (588045 pkts in 1017467 usec)
380.499787 main_thread [1502] 567368 pps (589582 pkts in 1039153 usec)
381.539151 main_thread [1502] 573155 pps (595719 pkts in 1039368 usec)
382.546350 main_thread [1502] 566560 pps (570645 pkts in 1007211 usec)
383.552021 main_thread [1502] 572517 pps (575764 pkts in 1005672 usec)
384.562003 main_thread [1502] 575141 pps (580881 pkts in 1009981 usec)
385.572339 main_thread [1502] 567857 pps (573720 pkts in 1010325 usec)
386.611927 main_thread [1502] 573032 pps (595723 pkts in 1039599 usec)
387.621921 main_thread [1502] 572097 pps (577815 pkts in 1009994 usec)
388.631956 main_thread [1502] 565493 pps (571168 pkts in 1010035 usec)
389.642583 main_thread [1502] 565682 pps (571677 pkts in 1010598 usec)
390.689010 main_thread [1502] 568310 pps (594702 pkts in 1046439 usec)
391.728145 main_thread [1502] 568358 pps (590605 pkts in 1039142 usec)
392.731926 main_thread [1502] 572061 pps (574230 pkts in 1003791 usec)
393.743127 main_thread [1502] 564364 pps (570649 pkts in 1011136 usec)
394.759490 main_thread [1502] 574519 pps (583949 pkts in 1016414 usec)
395.762549 main_thread [1502] 570940 pps (572689 pkts in 1003063 usec)
396.798457 main_thread [1502] 574078 pps (594698 pkts in 1035918 usec)
397.837743 main_thread [1502] 564837 pps (587027 pkts in 1039286 usec)
398.845684 main_thread [1502] 574269 pps (578829 pkts in 1007941 usec)
399.854181 main_thread [1502] 569405 pps (574234 pkts in 1008480 usec)
400.861927 main_thread [1502] 567264 pps (571668 pkts in 1007763 usec)
^C401.038285 sigint_h [326] received control-C on thread 0x801806800
401.038691 sender_body [1162] flush tail 1432 head 1432 on thread 0x801806800
401.909521 main_thread [1502] 94776 pps (99287 pkts in 1047593 usec)
Sent 14013848 packets, 512 bytes each, in 24.61 seconds.
Speed: 569.32 Kpps Bandwidth: 2.33 Gbps (raw 2.44 Gbps)

node-01(RX):

# pkt-gen -i em1 -f rx
369.668412 main [1705] interface is em1
369.670635 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0
369.670690 extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0
369.711000 main [1896] mapped 94564KB at 0x801c00000
Receiving from netmap:em1: 1 queues, 1 threads and 1 cpus.
369.711552 main [1982] Wait 2 secs for phy reset
371.719284 main [1984] Ready...
371.719996 nm_open [456] overriding ifname em1 ringid 0x0 flags 0x1
371.720143 receiver_body [1238] reading from netmap:em1 fd 4 main_fd 3
372.732654 receiver_body [1245] waiting for initial packets, poll returns 0 0
372.732711 main_thread [1502] 0 pps (0 pkts in 1012597 usec)
373.740524 main_thread [1502] 0 pps (0 pkts in 1007812 usec)
373.740585 receiver_body [1245] waiting for initial packets, poll returns 0 0
374.756275 receiver_body [1245] waiting for initial packets, poll returns 0 0
374.756943 main_thread [1502] 0 pps (0 pkts in 1016420 usec)
375.795651 main_thread [1502] 0 pps (0 pkts in 1038707 usec)
375.795711 receiver_body [1245] waiting for initial packets, poll returns 0 0
376.799290 main_thread [1502] 5243 pps (5262 pkts in 1003641 usec)
377.811330 main_thread [1502] 51692 pps (52314 pkts in 1012039 usec)
378.819327 main_thread [1502] 51138 pps (51544 pkts in 1007936 usec)
379.866574 main_thread [1502] 51460 pps (53892 pkts in 1047259 usec)
380.905832 main_thread [1502] 51469 pps (53492 pkts in 1039307 usec)
381.913870 main_thread [1502] 51530 pps (51944 pkts in 1008037 usec)
382.921554 main_thread [1502] 52032 pps (52432 pkts in 1007686 usec)
383.929619 main_thread [1502] 52024 pps (52443 pkts in 1008061 usec)
384.945221 main_thread [1502] 52014 pps (52826 pkts in 1015606 usec)
385.979527 main_thread [1502] 52176 pps (53966 pkts in 1034306 usec)
386.992471 main_thread [1502] 51964 pps (52637 pkts in 1012943 usec)
387.999305 main_thread [1502] 52365 pps (52723 pkts in 1006834 usec)
389.016092 main_thread [1502] 51963 pps (52835 pkts in 1016787 usec)
390.019465 main_thread [1502] 52144 pps (52320 pkts in 1003373 usec)
391.029548 main_thread [1502] 51940 pps (52464 pkts in 1010085 usec)
392.055595 main_thread [1502] 52015 pps (53370 pkts in 1026045 usec)
393.089165 main_thread [1502] 52265 pps (54020 pkts in 1033572 usec)
394.099234 main_thread [1502] 52175 pps (52700 pkts in 1010067 usec)
395.111112 main_thread [1502] 51414 pps (52025 pkts in 1011879 usec)
396.119496 main_thread [1502] 51971 pps (52402 pkts in 1008293 usec)
397.165867 main_thread [1502] 52218 pps (54641 pkts in 1046407 usec)
398.199329 main_thread [1502] 52243 pps (53994 pkts in 1033516 usec)
399.210557 main_thread [1502] 51914 pps (52492 pkts in 1011135 usec)
400.220779 main_thread [1502] 51815 pps (52350 pkts in 1010316 usec)
401.236538 main_thread [1502] 52317 pps (53142 pkts in 1015760 usec)
402.240178 main_thread [1502] 3153 pps (3164 pkts in 1003633 usec)
403.276884 main_thread [1502] 0 pps (0 pkts in 1036710 usec)
^C403.698121 sigint_h [326] received control-C on thread 0x801806800
404.279287 main_thread [1502] 0 pps (0 pkts in 1002403 usec)
Received 1277394 packets, in 27.00 seconds.
Speed: 47.31 Kpps

送信が560Kpps、受信が50Kppsと言ったところ。

DPDKアプリは基本的に割り当てられたコアのCPU占有率を100%にしようとするので、 仮想マシンで全部を賄っているこの環境においてはホストのCPUを取り合ってジェネレータの性能が出ない。

ので、この結果は値としての参考にはならないのだが、Lagopusがある程度の性能を出せそうな感触はつかめることだろう。

KVM用の仮想スイッチとして動かした場合、これくらいの転送性能は出ることが期待できる。(他の仮想マシンの処理性能への影響を無視すれば、だが)

とは言え、pingの値だけを見るとLagopus側もしかして少し遅延乗る?的な意見もあるかもしれなくて、どう使うかによって確認したい項目は変わってくると思います。

誤解を招くのでおまけはありません。