Long road to microservices architecture at LINE messaging platform.

Masahiro Ide
LINE Z Part Team
https://linedevday.linecorp.com/jp/2019/sessions/D1-6

Transcript

1. 2019 DevDay Long Road to Microservices Architecture at LINE Messaging Platform. > Masahiro Ide > LINE Z Part Team
2. Agenda > How did we scale our services > Weapons we prepared for our service microservices > How do we growth our architecture in future
3. When we launch LINE service LINE Messaging Platform (2011) > A Monolithic Service • Everything are handled at talk-server > Team • 1 team, 6 members LEGY talk-server Redis MySQL Server team
4. LINE Messaging Platform (2011~2019) Social Graph VoIP Message Sticker Theme LINE@ Group Search LINE NEWS Recommendation Game Timeline Image Video Payment Wallet Clova OpenChat Social Login Emoji Settings
5. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ...
6. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok, let’s focus on Video share feature
7. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok, let’s focus on Video share feature Oh, wait. I want to focus on sticker
8. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok, let’s focus on Video share feature Oh, wait. I want to focus on sticker Waaait!! VoIP has highest priority!!
9. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Image/Video Delivery
10. LINE Messaging Platform (2011~2019) LEGY talk-server ... Image/Video Delivery Stickershop Want to make more new type of sticker!! Let’s consider next-gen protocol Let’s improve img/video quality Let’s focus on new messaging feature
11. Now LINE Messaging Platform (2019) > Simple app server became a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
12. Now LINE Messaging Platform (2019) > Simple app server became a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop x8 x25 x10 x31 x6 x9 Service x1 Service >20 Service x1 Service >40 Service >20 Service x7
13. Microservice architecture solves everything? > Good ! • Conflicts between features ↓ • Development speed ↑ ↑ > Not good " • Network failure ↑ • Potential Cascading failure ↑ • Manage more service configs ↑
14. Concepts need to build our microservices architecture Armeria Connectivity LEGY Routing CentralDogma Directory Service
15. Connectivity Armeria - Our RPC layer > Asynchronous RPC/REST library • built on top of Java 8, Netty, HTTP/2, Thrift and gRPC • https://github.com/line/armeria > Take care common functionality for microservice • Logging, Client side Load Balancing, Monitoring integration, Tracing • Retry, Throttling, Circuit Breaker, Health Check
16. Structured Logging Client Connectivity Armeria - Our RPC layer gRPC/Thrift Client HTTP/2 Client Client side Load Balancer Circuit Breaker Metrics Integration Structured Logging Service gRPC/Thrift Service HTTP/2 Service Zipkin Service Rate Limiting Service Metrics Integration
17. Directory Service Central Dogma > Repository service for textual configurations • Highly available, version controlled, advanced query mechanism > Stores start-time/run-time configuration safely > https://github.com/line/centraldogma/ JSON, YAML, XML, JS…
18. Directory Service Central Dogma TKSVR123 Find talk-server-release servers TKSVR124 ... [{hostname:TKSVR123, port: 8080}, {hostname:TKSVR124, port: 8080,},... ] Armeria Client http://group:talk-server-relase/api group:talk-server-relase → TKSVR123:8080
19. Routing - LEGY DataCenter 1 Main DataCenter App Server LEGY DataCenter 2 LEGY LEGY App Server > LINE Event Delivery Gateway – API Gateway written in Erlang • SPDY (& HTTP/2) • Global POP • Routing DSL
20. Routing - LEGY > LINE Event Delivery Gateway – API Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf
21. Routing - LEGY > LINE Event Delivery Gateway – API Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf
22. talk-server and stickershop example Case study > Making sticker ownership validation feature as microservices > Service redesign project between talk-server & Sticker Shop at 2017 • Rearranges service boundary
23. Motivation for reconstruction > Increase the number of stickers sold due to Creators Market launch > Increasing number of users who have used LINE since long times ago • Some users have 20,000 stickers. # of user # of owned stickers
24. How to send a sticker at 2017 > Save sticker ownership data to Redis for each user as a Set • Refresh all user ownership when a user purchase a sticker > Check Redis data when a user sends a sticker Stickershop LEGY talk-server Redis
25. What happens if services are not separated > Talk-server stops every time a mass-sticker owned user buy a sticker LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
26. What happens if services are not separated > In worst case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
27. What happens if services are not separated > In worst case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
28. How to solve this? > Separate responsibilities and services not to block each development > Make feature fast-fail-able not to affect each other • Circuit breaker + automatic request retry Stickershop LEGY talk-server Sticker Capability service Sticker Capability service
29. Retrospective > Rebuilt with microservices architecture • without degrading service quality thanks to our weapons > Require a mechanism that can flexibly respond to • Scale • Speed
30. Our Futures > Make our platform more fine-grained microservices Feature Feature Feature To: separating by Function Function Function Function Function Function Function Function Function Function Function Function Function From: separating by Feature
31. Wrap up of this talk > We adopt microservices architecture to speed up our development > Prepared our weapons for microservices architecture • Armeria, CentralDogma, LEGY > We’re continuing to transform our service to microservices architecture