Lagopus switchをryuと一緒に動かしてみるなの？
Posted on 2014/10/14(Tue) 00:30 in technical
構成図
こういう感じでひとつお願いします。
ryuのインストール
Lagopusが入っているのと同じマシンに楽ちんインストールです。:
$ sudo apt-get install python-setuptools python-pip python-dev libxml2-dev libxslt-dev $ sudo pip install ryu
と思いきや:
$ ryu-manager --version
Traceback (most recent call last):
File "/usr/local/bin/ryu-manager", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2749, in <module>
working_set = WorkingSet._build_master()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 446, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 459, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 632, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (six 1.5.2 (/usr/lib/python2.7/dist-packages), Requirement.parse('six>=1.7.0'))
ぎゃあ。:
$ sudo pip install -U six $ ryu-manager --version ryu-manager 3.13
おっけー。
コントローラの起動
こんな感じで。:
$ ryu-manager --log-file=/tmp/ryu.log /usr/local/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py &
Lagopusの起動
ものぐさコピペシートー:
cd
cd ~/dpdk-1.7.1
export RTE_SDK=`pwd`
export RTE_TARGET="x86_64-native-linuxapp-gcc"
sudo modprobe uio
sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/igb_uio.ko
sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/rte_kni.ko
sudo ./tools/dpdk_nic_bind.py -b igb_uio 0000:00:08.0
sudo ./tools/dpdk_nic_bind.py -b igb_uio 0000:00:09.0
cd /etc/lagopus
sudo lagopus -l /tmp/lagopus.log -- -c3 -n1 -- -p3
何か適当なコマンドでsudoでパスワード聞かれなくしたら、コピペで起動する。
ping打ってみる
node-01:
# ifconfig em1 192.168.0.1/24 up
node-02:
# ifconfig em1 192.168.0.2/24 up # ping -c4 192.168.0.1 PING 192.168.0.1 (192.168.0.1): 56 data bytes 64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=33.564 ms 64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=13.000 ms 64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=13.890 ms 64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=14.771 ms --- 192.168.0.1 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 13.000/18.806/33.564/8.543 ms
疎通は問題なさそうだ。
ryuのログも見てみる。:
$ cat /tmp/ryu.log loading app /usr/local/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py loading app ryu.controller.ofp_handler instantiating app /usr/local/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py of SimpleSwitch13 instantiating app ryu.controller.ofp_handler of OFPHandler packet in 5803077010208869558 08:00:27:e2:53:5a ff:ff:ff:ff:ff:ff 2 packet in 5803077010208869558 08:00:27:3b:72:00 08:00:27:e2:53:5a 1 packet in 5803077010208869558 08:00:27:e2:53:5a 08:00:27:3b:72:00 2
ふむ、よろしい。
では、lagopusのフローテーブルも確認しよう。:
$ sudo lagosh ubuntu1404-1> show flow Bridge: br0 Table id: 0 priority=1,idle_timeout=0,hard_timeout=0,flags=0,cookie=0,packet_count=4,byte_count=392,in_port=1,eth_dst=08:00:27:e2:53:5a actions=output:2 priority=1,idle_timeout=0,hard_timeout=0,flags=0,cookie=0,packet_count=3,byte_count=294,in_port=2,eth_dst=08:00:27:3b:72:00 actions=output:1 priority=0,idle_timeout=0,hard_timeout=0,flags=0,cookie=0,packet_count=3,byte_count=218 actions=output:controller
ちゃんとフローエントリに登録されているようだ。
netmapで負荷をかけてみる
BSDRPは最初からnetmap搭載なので、pkt-genを使用することが出来る。
もちろん、これは仮想環境なので十分な測定器としての性能は出ないのだが、pingだけでは少々物足りないので。
送信側はフレーム長512Byteで出せるだけ。
node-02(TX):
# pkt-gen -i em1 -f tx -l 512 -D 08:00:27:3b:72:00 -s 192.168.0.2 -d 192.168.0.1 582.957774 main [1705] interface is em1 582.960888 extract_ip_range [289] range is 192.168.0.2:0 to 192.168.0.2:0 582.960902 extract_ip_range [289] range is 192.168.0.1:0 to 192.168.0.1:0 583.000689 main [1896] mapped 94548KB at 0x801c00000 Sending on netmap:em1: 1 queues, 1 threads and 1 cpus. 192.168.0.2 -> 192.168.0.1 (00:00:00:00:00:00 -> 08:00:27:3b:72:00) 583.000766 main [1980] Sending 512 packets every 0.000000000 s 583.000773 main [1982] Wait 2 secs for phy reset 585.024614 main [1984] Ready... 585.025767 nm_open [456] overriding ifname em1 ringid 0x0 flags 0x1 585.026074 sender_body [1064] start, fd 4 main_fd 3 585.325608 sender_body [1133] drop copy 586.037434 main_thread [1502] 379033 pps (383345 pkts in 1011375 usec) 587.044306 main_thread [1502] 423423 pps (426329 pkts in 1006862 usec) 588.058373 main_thread [1502] 410826 pps (416610 pkts in 1014078 usec) 589.094597 main_thread [1502] 420336 pps (435546 pkts in 1036185 usec) 590.130362 main_thread [1502] 395297 pps (409444 pkts in 1035787 usec) 591.137019 main_thread [1502] 437230 pps (440148 pkts in 1006674 usec) 592.146696 main_thread [1502] 418719 pps (422748 pkts in 1009622 usec) 593.154509 main_thread [1502] 409806 pps (413019 pkts in 1007840 usec) 594.168906 main_thread [1502] 424308 pps (430423 pkts in 1014411 usec) 595.201132 main_thread [1502] 404090 pps (417117 pkts in 1032239 usec) 596.240903 main_thread [1502] 393294 pps (408926 pkts in 1039747 usec) 597.255694 main_thread [1502] 411032 pps (417116 pkts in 1014803 usec) 598.263879 main_thread [1502] 434547 pps (438099 pkts in 1008173 usec) 599.279539 main_thread [1502] 434879 pps (441687 pkts in 1015654 usec) 600.313926 main_thread [1502] 418084 pps (432469 pkts in 1034407 usec) ^C600.976396 sigint_h [326] received control-C on thread 0x801806800 600.976470 sender_body [1162] flush tail 2041 head 1018 on thread 0x801806800 601.325873 main_thread [1502] 261473 pps (264600 pkts in 1011958 usec) Sent 6597626 packets, 512 bytes each, in 15.95 seconds. Speed: 413.61 Kpps Bandwidth: 1.69 Gbps (raw 1.77 Gbps)
node-01(RX):
# pkt-gen -i em1 -f rx 578.422659 main [1705] interface is em1 578.423529 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0 578.423539 extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0 578.463447 main [1896] mapped 94564KB at 0x801c00000 Receiving from netmap:em1: 1 queues, 1 threads and 1 cpus. 578.463500 main [1982] Wait 2 secs for phy reset 580.499898 main [1984] Ready... 580.500224 nm_open [456] overriding ifname em1 ringid 0x0 flags 0x1 580.500377 receiver_body [1238] reading from netmap:em1 fd 4 main_fd 3 581.512074 receiver_body [1245] waiting for initial packets, poll returns 0 0 581.512113 main_thread [1502] 0 pps (0 pkts in 1011760 usec) 582.520785 main_thread [1502] 0 pps (0 pkts in 1008668 usec) 582.520843 receiver_body [1245] waiting for initial packets, poll returns 0 0 583.529179 receiver_body [1245] waiting for initial packets, poll returns 0 0 583.529499 main_thread [1502] 0 pps (0 pkts in 1008717 usec) 584.575076 main_thread [1502] 0 pps (0 pkts in 1045578 usec) 584.575122 receiver_body [1245] waiting for initial packets, poll returns 0 0 585.606672 main_thread [1502] 66382 pps (68479 pkts in 1031595 usec) 586.623319 main_thread [1502] 206749 pps (210191 pkts in 1016647 usec) 587.630316 main_thread [1502] 209486 pps (210952 pkts in 1006997 usec) 588.646089 main_thread [1502] 206478 pps (209735 pkts in 1015773 usec) 589.685415 main_thread [1502] 204498 pps (212540 pkts in 1039327 usec) 590.689263 main_thread [1502] 208670 pps (209473 pkts in 1003848 usec) 591.702000 main_thread [1502] 199973 pps (202520 pkts in 1012736 usec) 592.710373 main_thread [1502] 203329 pps (205028 pkts in 1008356 usec) 593.724733 main_thread [1502] 204628 pps (207570 pkts in 1014377 usec) 594.756339 main_thread [1502] 201510 pps (207879 pkts in 1031605 usec) 595.795702 main_thread [1502] 205698 pps (213795 pkts in 1039364 usec) 596.811223 main_thread [1502] 204670 pps (207847 pkts in 1015521 usec) 597.827079 main_thread [1502] 199453 pps (202615 pkts in 1015855 usec) 598.871887 main_thread [1502] 206255 pps (215494 pkts in 1044793 usec) 599.905812 main_thread [1502] 195547 pps (202184 pkts in 1033941 usec) 600.910418 main_thread [1502] 199519 pps (200438 pkts in 1004606 usec) ^C601.804637 sigint_h [326] received control-C on thread 0x801806800 601.920027 main_thread [1502] 63721 pps (64333 pkts in 1009602 usec) Received 3251073 packets, in 16.53 seconds. Speed: 196.62 Kpps
おおよその値だが、送信側は400Kpps、受信側は200Kpps程度出ている。
Open vSwitchと比べてみる
では、ここでLagopusを入れた仮想マシンにOpen vSwitchを入れて比較してみる。
Open vSwitchでポートを接続し、先程と同じようにryu-managerでsimple_switch_13.pyを起動する。:
$ sudo apt install openvswitch-switch $ sudo reboot $ sudo ovs-vsctl add-br br0 $ sudo ovs-vsctl add-port br0 eth1 $ sudo ovs-vsctl add-port br0 eth2 $ sudo ovs-vsctl set bridge br0 protocols=OpenFlow13 $ sudo ovs-vsctl set-controller br0 tcp:127.0.0.1:6633 $ ryu-manager --log-file=/tmp/ryu.log /usr/local/lib/python2.7/dist-packages/ryu/app/simple_switch_13.py &
pingが飛ぶことを確認したらpkt-genで送信開始。
node-02(TX):
# ping -c4 192.168.0.1 PING 192.168.0.1 (192.168.0.1): 56 data bytes 64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=16.935 ms 64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=1.491 ms 64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=1.041 ms 64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.864 ms --- 192.168.0.1 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.864/5.083/16.935/6.847 ms # pkt-gen -i em1 -f tx -l 512 -D 08:00:27:3b:72:00 -s 192.168.0.2 -d 192.168.0.1 374.356559 main [1705] interface is em1 374.357108 extract_ip_range [289] range is 192.168.0.2:0 to 192.168.0.2:0 374.357126 extract_ip_range [289] range is 192.168.0.1:0 to 192.168.0.1:0 374.396577 main [1896] mapped 94548KB at 0x801c00000 Sending on netmap:em1: 1 queues, 1 threads and 1 cpus. 192.168.0.2 -> 192.168.0.1 (00:00:00:00:00:00 -> 08:00:27:3b:72:00) 374.397217 main [1980] Sending 512 packets every 0.000000000 s 374.397226 main [1982] Wait 2 secs for phy reset 376.423024 main [1984] Ready... 376.424440 nm_open [456] overriding ifname em1 ringid 0x0 flags 0x1 376.424628 sender_body [1064] start, fd 4 main_fd 3 376.651937 sender_body [1133] drop copy 377.437845 main_thread [1502] 547529 pps (554780 pkts in 1013244 usec) 378.443151 main_thread [1502] 572720 pps (575762 pkts in 1005312 usec) 379.460618 main_thread [1502] 577950 pps (588045 pkts in 1017467 usec) 380.499787 main_thread [1502] 567368 pps (589582 pkts in 1039153 usec) 381.539151 main_thread [1502] 573155 pps (595719 pkts in 1039368 usec) 382.546350 main_thread [1502] 566560 pps (570645 pkts in 1007211 usec) 383.552021 main_thread [1502] 572517 pps (575764 pkts in 1005672 usec) 384.562003 main_thread [1502] 575141 pps (580881 pkts in 1009981 usec) 385.572339 main_thread [1502] 567857 pps (573720 pkts in 1010325 usec) 386.611927 main_thread [1502] 573032 pps (595723 pkts in 1039599 usec) 387.621921 main_thread [1502] 572097 pps (577815 pkts in 1009994 usec) 388.631956 main_thread [1502] 565493 pps (571168 pkts in 1010035 usec) 389.642583 main_thread [1502] 565682 pps (571677 pkts in 1010598 usec) 390.689010 main_thread [1502] 568310 pps (594702 pkts in 1046439 usec) 391.728145 main_thread [1502] 568358 pps (590605 pkts in 1039142 usec) 392.731926 main_thread [1502] 572061 pps (574230 pkts in 1003791 usec) 393.743127 main_thread [1502] 564364 pps (570649 pkts in 1011136 usec) 394.759490 main_thread [1502] 574519 pps (583949 pkts in 1016414 usec) 395.762549 main_thread [1502] 570940 pps (572689 pkts in 1003063 usec) 396.798457 main_thread [1502] 574078 pps (594698 pkts in 1035918 usec) 397.837743 main_thread [1502] 564837 pps (587027 pkts in 1039286 usec) 398.845684 main_thread [1502] 574269 pps (578829 pkts in 1007941 usec) 399.854181 main_thread [1502] 569405 pps (574234 pkts in 1008480 usec) 400.861927 main_thread [1502] 567264 pps (571668 pkts in 1007763 usec) ^C401.038285 sigint_h [326] received control-C on thread 0x801806800 401.038691 sender_body [1162] flush tail 1432 head 1432 on thread 0x801806800 401.909521 main_thread [1502] 94776 pps (99287 pkts in 1047593 usec) Sent 14013848 packets, 512 bytes each, in 24.61 seconds. Speed: 569.32 Kpps Bandwidth: 2.33 Gbps (raw 2.44 Gbps)
node-01(RX):
# pkt-gen -i em1 -f rx 369.668412 main [1705] interface is em1 369.670635 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0 369.670690 extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0 369.711000 main [1896] mapped 94564KB at 0x801c00000 Receiving from netmap:em1: 1 queues, 1 threads and 1 cpus. 369.711552 main [1982] Wait 2 secs for phy reset 371.719284 main [1984] Ready... 371.719996 nm_open [456] overriding ifname em1 ringid 0x0 flags 0x1 371.720143 receiver_body [1238] reading from netmap:em1 fd 4 main_fd 3 372.732654 receiver_body [1245] waiting for initial packets, poll returns 0 0 372.732711 main_thread [1502] 0 pps (0 pkts in 1012597 usec) 373.740524 main_thread [1502] 0 pps (0 pkts in 1007812 usec) 373.740585 receiver_body [1245] waiting for initial packets, poll returns 0 0 374.756275 receiver_body [1245] waiting for initial packets, poll returns 0 0 374.756943 main_thread [1502] 0 pps (0 pkts in 1016420 usec) 375.795651 main_thread [1502] 0 pps (0 pkts in 1038707 usec) 375.795711 receiver_body [1245] waiting for initial packets, poll returns 0 0 376.799290 main_thread [1502] 5243 pps (5262 pkts in 1003641 usec) 377.811330 main_thread [1502] 51692 pps (52314 pkts in 1012039 usec) 378.819327 main_thread [1502] 51138 pps (51544 pkts in 1007936 usec) 379.866574 main_thread [1502] 51460 pps (53892 pkts in 1047259 usec) 380.905832 main_thread [1502] 51469 pps (53492 pkts in 1039307 usec) 381.913870 main_thread [1502] 51530 pps (51944 pkts in 1008037 usec) 382.921554 main_thread [1502] 52032 pps (52432 pkts in 1007686 usec) 383.929619 main_thread [1502] 52024 pps (52443 pkts in 1008061 usec) 384.945221 main_thread [1502] 52014 pps (52826 pkts in 1015606 usec) 385.979527 main_thread [1502] 52176 pps (53966 pkts in 1034306 usec) 386.992471 main_thread [1502] 51964 pps (52637 pkts in 1012943 usec) 387.999305 main_thread [1502] 52365 pps (52723 pkts in 1006834 usec) 389.016092 main_thread [1502] 51963 pps (52835 pkts in 1016787 usec) 390.019465 main_thread [1502] 52144 pps (52320 pkts in 1003373 usec) 391.029548 main_thread [1502] 51940 pps (52464 pkts in 1010085 usec) 392.055595 main_thread [1502] 52015 pps (53370 pkts in 1026045 usec) 393.089165 main_thread [1502] 52265 pps (54020 pkts in 1033572 usec) 394.099234 main_thread [1502] 52175 pps (52700 pkts in 1010067 usec) 395.111112 main_thread [1502] 51414 pps (52025 pkts in 1011879 usec) 396.119496 main_thread [1502] 51971 pps (52402 pkts in 1008293 usec) 397.165867 main_thread [1502] 52218 pps (54641 pkts in 1046407 usec) 398.199329 main_thread [1502] 52243 pps (53994 pkts in 1033516 usec) 399.210557 main_thread [1502] 51914 pps (52492 pkts in 1011135 usec) 400.220779 main_thread [1502] 51815 pps (52350 pkts in 1010316 usec) 401.236538 main_thread [1502] 52317 pps (53142 pkts in 1015760 usec) 402.240178 main_thread [1502] 3153 pps (3164 pkts in 1003633 usec) 403.276884 main_thread [1502] 0 pps (0 pkts in 1036710 usec) ^C403.698121 sigint_h [326] received control-C on thread 0x801806800 404.279287 main_thread [1502] 0 pps (0 pkts in 1002403 usec) Received 1277394 packets, in 27.00 seconds. Speed: 47.31 Kpps
送信が560Kpps、受信が50Kppsと言ったところ。
DPDKアプリは基本的に割り当てられたコアのCPU占有率を100%にしようとするので、 仮想マシンで全部を賄っているこの環境においてはホストのCPUを取り合ってジェネレータの性能が出ない。
ので、この結果は値としての参考にはならないのだが、Lagopusがある程度の性能を出せそうな感触はつかめることだろう。
KVM用の仮想スイッチとして動かした場合、これくらいの転送性能は出ることが期待できる。（他の仮想マシンの処理性能への影響を無視すれば、だが）
とは言え、pingの値だけを見るとLagopus側もしかして少し遅延乗る？的な意見もあるかもしれなくて、どう使うかによって確認したい項目は変わってくると思います。
誤解を招くのでおまけはありません。