モンストを支えるインフラの今とこれから

Transcript

1. ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β 2016.3.1 dots. Conference Spring 2016 ήʔϜ։ൃͷཪଆ mixi, Inc. XFLAG™

   STUDIO @isaoshimizu

2. About Me • ਗ਼ਫ ܄ SHIMIZU ISAO @isaoshimizu • 2011೥8݄ΑΓגࣜձࣾϛΫγΟ

   • ݱࡏ͸ΤοΫεϑϥοάελδΦ γεςϜ։ൃ෦ ॴଐ • ໿2೥લʹιʔγϟϧɾωοτϫʔΩϯά αʔϏε mixi ͷӡ༻͔Β
 εϚʔτϑΥϯήʔϜʮϞϯελʔετϥΠΫʯͷӡ༻΁ 2

3. ϞϯελʔετϥΠΫ • 2013೥10݄10೔ ਖ਼ࣜϦϦʔε • iOS൛, Android൛Λఏڙ • શੈքͰ3,000ສਓҎ্͕ϓϨΠʢಉҰ୺຤Ͱॏෳμ΢ϯϩʔυ͞Εͨ਺͸ؚ·ͣʣ •

   ೔ຊɺ୆࿷ɺؖࠃɺ๺ถɺ߳ߓɾϚΧΦͰల։ • YouTubeͰΞχϝͷ഑৴ 2,500ສճͷ࠶ੜɺχϯςϯυʔ3DS൛ ग़ՙ਺100ສຊಥഁ 3

4. ͔͜͜Β͸ϞϯελʔετϥΠΫ ೔ຊ൛ʹݶఆͨ͠಺༰ʹͳΓ·͢ 4

5. ਺஋ͰݟΔϞϯελʔετϥΠΫͷΠϯϑϥ 5

6. 6 Application 300+ Servers Resque 50 Servers Redis 30 Servers

   Memcached 200+ Servers CDN Traffic Max 270Gbps API Traffic Avg 1Gbps TURN 40 Servers MariaDB 250+ Servers Internal Traffic Max 20Gbps

7. Πϯϑϥߏ੒ 7

8. 8 DC1 DC2 Main Main/Backup Main/Dev

9. ෳ਺DCͱAmazon VPC • DC 1ʢϝΠϯʣɺDC 2ʢϝΠϯͱόοΫΞοϓʣɺAWSʢϝΠϯͱ։ൃ؀ڥʣͰߏ੒ • ֤DC͸TY2,4ܦ༝ͰAmazon VPCͱDirectConnectͰϓϥΠϕʔτ઀ଓ •

   ֤DCؒ͸ยܥো֐࣌ʹ͓͍ͯ΋20Gbps·Ͱ଱͑ΒΕΔઃܭ • ৄ͍͠ղઆ
 “ڌ఺৑௕ͨ͠࿩ʢ಺෦τϥϑΟοΫϘϦϡʔϜฤʣ”
 http://xflag.com/blog/infradb/internal_traffic_volume.html 9

10. ֤ڌ఺ͷ໾ׂ 10

11. 11 DC1 DC2 Main Main/Backup Main/Dev Application MariaDB Redis Resque

    Memcached MariaDB Redis Memcached TURN Server ։ൃ༻Πϯελϯε܈ Application Resque

12. ֤ڌ఺ͷ໾ׂ • ຊ൪ͷApplicationɺResque͸DC1ɺDC2ͷ྆ํͰՔಇ • ຊ൪ͷRedisɺMariaDBɺMemcached • DC1ʹMaster/Backupͷηοτ • DC2ʹBackup •

    Backup͸DC1ͱDC2߹Θͤͯ2ηοτ͋Δ • AWSͰ͸ຊ൪ͷTURN ServerʢϚϧνϓϨΠ༻ʣɺ։ൃ༻Πϯελϯε܈ͳͲ͕Քಇɻ 12

13. ΫϥΠΞϯτΞϓϦ͔ΒݟͨΠϯϑϥ 13

14. 14 DC1 DC2 AWS TURN Server (EC2) CDN (CloudFront, Akamai)

    API API LB LB Internet Client

15. ΫϥΠΞϯτΞϓϦ͔ΒͷΞΫηεઌ • DC1ͱDC2ʹ͋ΔϩʔυόϥϯαʔʢHAߏ੒ʣ΁ • ϩʔυόϥϯαʔ΁ͷΞΫηε͸DNSϥ΢ϯυϩϏϯʢ2ͭͷԾ૝IPʣ • ϩʔυόϥϯαʔ͔Β֤ڌ఺ͷApplicationαʔό΁ৼΓ෼͚ • AWSͰ͸ϚϧνϓϨΠ༻ʹTURN ServerʢEC2

    + Elastic IPʣ • ήʔϜσʔλɾϦιʔεσʔλ༻ʹCDNʢCloudFront + S3ʣɻAkamai΋ซ༻ɻ 15

16. σϓϩΠ 16

17. 17 DC1 DC2 Deploy Server S3 Bucket tarball S3 PUT

    GitHub ssh Code Internet

18. σϓϩΠ • Capistrano 2 • ඞཁͳͱ͖ʹඞཁͳ͚ͩσϓϩΠ͢Δ • capistrano-s3copy-awscli https://github.com/bacter1a/capistrano-s3copy-awscli •

    GitHub͔Β࠷৽ͷίʔυΛऔಘͯ͠S3Ξοϓϩʔυ༻ʹtarballΛ࡞੒ • ApplicationαʔόͰ͸S3ʹΞοϓϩʔυ͞ΕͨtarballΛμ΢ϯϩʔυɺల։ • GitHub΁ͷ઀ଓ਺੍ݶରࡦʢgithub.com΁େྔʹ઀ଓ͕ཁٻ͞ΕΔͱڋ൱͞ΕΔͨΊʣ 18

19. • capistrano-rbenv https://github.com/yyuu/capistrano-rbenv • όʔδϣϯࢦఆͰRubyͷ഑෍ɺΞοϓσʔτָ͕ʹ • Ruby 2.2.4ಋೖࡁΈ 19

20. ϓϩϏδϣχϯά 20

21. 21 DC1 DC2 Deploy Server S3 Bucket GitHub ssh Chef

    Cookbook Ubuntu Package Mirror Internet deb Package deb Package

22. ϓϩϏδϣχϯά • ChefʹΑΔϓϩϏδϣχϯά͕جຊ • capistrano-paratrooper-chef
 https://github.com/tk0miya/capistrano-paratrooper-chef • ෺ཧϚγϯͷॳظηοτΞοϓʢ࠷௿ݶʣ͸AnsibleΛ࢖͏ • OSΫϦʔϯΠϯετʔϧ௚ޙͷঢ়ଶͰ͸ɺChef͕Πϯετʔϧ͞Ε͍ͯͳ͍ͨΊ

    22

23. • Community CookbookΛincludeͨ͠ಠࣗCookbookΛ࡞੒ɺར༻͍ͯ͠Δ • ಠࣗCookbook͸ɺXFLAGશମͷΠϯϑϥͰ΋ڞ௨ͯ͠࢖͑ΔΑ͏ʹ • ͢΂ͯͷαʔό͕ChefͰߏஙՄೳ • Քಇதͷຊ൪DBɺMemcachedʹ࠶ద༻ͯ͠΋໰୊͕ى͖ͳ͍Α͏ʹ 23

24. • S3ʹMariaDBͷϨϙδτϦͷϛϥʔʢຊՈͷো֐ରࡦʣ • ioDriveͳͲͷυϥΠό΋ஔ͍ͯ͋Δ • aptlyͰߏங http://www.aptly.info/ • ݸਓͷ։ൃϚγϯͰ΋Vagrant +

    VirtualBox + ChefͰߏ੒Ͱ͖ΔΑ͏ʹ • Test Kitchenͷ׆༻ https://github.com/test-kitchen/test-kitchen 24

25. ෛՙରࡦ 25

26. ࠓ೥ʹೖ͔ͬͯΒ΋ߴෛՙ͕ଓ͘… 26

27. ෛՙରࡦ • DBͷγϟʔσΟϯάɺςʔϒϧ෼ׂΛܧଓ࣮ࢪ • Applicationαʔόͷ૿ઃ
 12-20 core ͳCPU x 2ʢHyper-Threading׵ࢉͰ24-40

    coreʣͷϚγϯ͕جຊߏ੒ • ΫΤϦվળ΍ΩϟογϡରԠ
 ioDrive(ioMemory)͔ΒSSD΁ͷεέʔϧμ΢ϯͷ࣮ݱ • Memcachedͷ૿ઃ 27

28. Memcachedͷߏ੒ 28

29. Memcachedͷߏ੒ • Memcached͸2ϓʔϧߏ੒ • 1ϓʔϧ͋ͨΓ໿100୆ • Ϛγϯ1୆͋ͨΓͷϝϞϦׂΓ౰ͯ26GBʢϝϞϦ32GB౥ࡌͷϚγϯʣ • 1ϓʔϧશମͰ໿2.6TBͷ༰ྔΛ΋ͭ •

    ྆ํͷϓʔϧʹಉ࣌ॻ͖ࠐΈ
 DoubleWriteCacheStores https://github.com/hirocaster/double_write_cache_stores • ϝϯςφϯεແ͠Ͱϓʔϧͷ੾Γସ͕͑ՄೳɻϓʔϧαΠζͷ૿ڧ͕Մೳʹɻ 29

30. 2ϓʔϧߏ੒ͷMemcached
 ೖΕସ͑ํ๏ 30

31. 31 App M App App M M M M M

    M M M M M M set get set Pool 1 Pool 2 App App App M M M M M M set get Pool 1 Pool 2 ௨ৗ࣌ ఀࢭɾ૿ઃ M M શΫϦΞ͓ͯ͘͠ M M M M M M Pool2΁ͷsetΛఀࢭ M Memcachedϗετ

32. 32 App M App App M M M M M

    set get set Pool 1 Pool 2 ੾Γସ͑ M M M M M M M M App App App set get Pool 1 Pool 2 Ωϟογϡ͋ͨͨΊ set M M M M M M M M M M M M M M ͠͹Β͋ͨͨ͘ΊΔ

33. ϩʔυόϥϯαʔ 33

34. ϩʔυόϥϯαʔ • Applicationαʔόͷલஈ͸ΞϓϥΠΞϯεͷϩʔυόϥϯαʔΛར༻ • A10 Networks AX2500 • RESTful APIʹΑΔૢ࡞͕Մೳ

    • ࣗࣾ։ൃͷGo੡ͷCLIπʔϧʢa10-cliʣ • Applicationαʔόͷ௥Ճɺ࡟আ࣌ʹίϚϯυ1ͭͰαʔϏεΠϯɾΞ΢τ͕Ͱ͖Δ 34

35. 35 nginx unicorn nginx unicorn nginx unicorn nginx unicorn Memcached΍MariaDBͳͲ

    Internet A10 ϩʔυόϥϯαʔ Applicationαʔό

36. ϞχλϦϯάɾ؂ࢹ 36

37. 37 CloudForecast • ͢΂ͯͷαʔόͷϝτϦΫεऔಘ • ໿1,000୆ͷαʔό͕ܭଌର৅ • ༷ʑͳಠࣗͷϝτϦΫεΛ௥Ճ https://github.com/kazeburo/cloudforecast

38. 38 GrowthForecast • DCϥοΫͷిྗ஋ͷάϥϑԽ • Fluentd Monitoring AgentͷάϥϑԽ http://kazeburo.github.io/GrowthForecast/

39. 39 • ͢΂ͯͷαʔόͷ؂ࢹ • 12,000Λ௒͑Δ؂ࢹ߲໨ • ಠࣗͷϓϥάΠϯΛ௥Ճ • PagerDutyͱͷ࿈ܞ •

    ౰൪ͷεέδϡʔϦϯά • ΤεΧϨʔγϣϯϙϦγʔ • Slack΁ͷ௨஌ https://www.nagios.org/

40. 40 • nginx, ApplicationͷϩάΛ
 FluentdΛ࢖ͬͯElasticsearch΁సૹ • APIຖͷॲཧ࣌ؒ΍ϨεϙϯελΠϜͷܭଌ • Τϥʔස౓ͷܭଌ •

    ಛఆϗετ͔ΒͷΞΫηεͷௐࠪ • Slow QueryͷՄࢹԽ • CloudTrailϩάͷՄࢹԽ https://www.elastic.co/

41. 41 nginx unicorn Amazon S3 GrowthForecast LTSV Log Application Servers

    ϩάసૹϑϩʔ Monitoring Agent Response Time Status Code Monitoring Agent Log Log td-agent 2.3.x Max 500Mbps Max 500Mbps

42. ೔ৗͷπʔϧ 42

43. 43 • ίϛϡχέʔγϣϯ͸SlackͰ౷Ұ • Hubot https://hubot.github.com ར༻ʹΑΔChatOpsͷ࣮ݱ • ։ൃ؀ڥ΁ͷσϓϩΠͳͲΛSlack͔Β࣮ߦՄೳ •

    ৄ͘͠͸Software Design 2016೥1݄߸ͷChatOpsಛूʹ
 http://gihyo.jp/magazine/SD/archive/2016/201601 • ΞϓϦέʔγϣϯίʔυ΋Πϯϑϥߏங༻ͷCookbookͳͲ
 ͢΂ͯGitHub্Ͱ؅ཧ • pull requestsϕʔεͷ։ൃɺϨϏϡʔ • GitHub্Ͱͷ׆ಈ͸Slack΁௨஌͞ΕΔ • GitHubͰpull requests͕࡞੒͞ΕΔͱࣗಈςετ࣮ߦ • Dockerίϯςφͷੜ੒ɺςετ࣮ߦɺഁغ • Hubotͱͷ࿈ܞ

44. ͜Ε͔Β΍͍͖͍ͬͯͨ͜ͱ 44

45. • ίʔυͷվળɺεέʔϧΞ΢τɺΩϟογϡར༻ͳͲ • εέʔϧΞοϓ͸Ͱ͖Δ͚ͩ΍Γͨ͘ͳ͍ • ݁Ռతʹεέʔϧμ΢ϯʹͭͳ͕Δͱϕετ • ͜ΕΒ͸ैདྷ͔Βܧଓ࣮ͯ͠ࢪ͍ͯ͠Δ • ιϑτ΢ΣΞͷܧଓతΞοϓσʔτ

    • Ubuntu Server 14.04 or 16.04 LTSʢ4/21ϦϦʔε༧ఆʣ΁ͷҠߦ • Kernelɺϛυϧ΢ΣΞͷΞοϓσʔτ͸ੵۃతʹ࣮ࢪ 45

46. • MemcachedͷSlab Rebalancingʢ slab_reassign, slab_automove ʣ • ·ͩௐࠪɺݕূஈ֊ͳͷͰӡ༻ࣄྫ͕͋Ε͹஌Γ͍ͨ • ؂ࢹɺϞχλϦϯά؀ڥͷվળ

    • ෛՙ΍ҟৗʹΑΓૉૣ͘ؾ͚ͮΔӡ༻ʹ • ΋͏গࣗ͠ಈԽ͍ͨ͠ͱ͜Ζ΋ • Consulͱͷ࿈ܞ 46

47. • ࣾ಺ͷ৽نҊ݅Ͱ͸Hashicorpࣾ੡πʔϧ Terraformͷར༻͕࢝·͍ͬͯΔ • Ϟϯετʹޙ͔Βಋೖ͢Δͷ͸ίετ΍ϦεΫ͕ߴ͍ͷͰಋೖ༧ఆ͸ͳ͠ • ෦෼తʹπʔϧར༻ʹΑΔޮ཰Խ͸͓͜ͳ͍͖͍ͬͯͨ • Πϯϑϥςετͷޮ཰Խ •

    Docker΍ServerspecͳͲΛ͞Βʹ׆༻͍͖͍ͯͨ͠ • Chef CookbookͷϦϑΝΫλϦϯάɺ΋ͬͱγϯϓϧʹ 47

48. ࠷ޙʹ 48

49. ΤϯδχΞืूதʂ 49 http://xflag.com/recruit/career/engineer.html

50. ͋Γ͕ͱ͏͍͟͝·ͨ͠ 50