Migrating Ruby on Rails to Kubernetes: Our Experience and Lessons Learned

User Avatar

In this blog post, we want to share our journey of migrating our Ruby on Rails application to run in containers and on top of Kubernetes. As a company heavily relying on Rails and serving numerous customers, we saw this migration as an opportunity to improve scalability, flexibility, and efficiency within our infrastructure. We'll discuss the challenges we faced, the decisions we made, and the benefits we gained from this move.


Our Original Infrastructure:

Before migrating to Kubernetes, our application stack consisted of various components such as Rails, MySQL, Redis, Memcached, Faye (real-time push server), and multiple Go services. We used AWS EC2, ELB load balancers, and AWS MySQL RDS to run our infrastructure.


Motivation for the Move:

While our previous infrastructure worked well for several years, adding new services became a bottleneck. For example, setting up Sphinx for search required additional servers, load balancing, backups, and firewall configurations. We wanted to streamline the process of adding new components to our infrastructure and reduce the complexity of managing different server types.


Preparing for the Migration:

As a team experienced with containers and Kubernetes, we decided to keep our monolithic Rails application intact, considering its strong conventions and efficiency. We began by creating a base Kubernetes configuration file, defining all services as Kubernetes services to ensure compatibility.


Building the Configuration:

We developed 11 configuration files for our project, addressing different aspects such as setup, security, configuration values, secrets, certificates, and various components' deployment. These files were stored in our git repository, with encrypted secrets and security files for enhanced protection.


Addressing Rails-specific Issues:

Throughout the migration, we encountered specific challenges related to Rails:


1. Database Migration: We leveraged Kubernetes jobs to run the Rails database migrations before each deployment, as we didn't have any backward-incompatible migrations.


2. Asset Pipeline Compilation: To handle asset compilation, we moved the process to our image building flow, ensuring that assets were compiled into the container images.


3. Sidekiq Restarts: Kubernetes posed challenges in managing Sidekiq deployments due to its kill signals. We had to configure Kubernetes to delay killing Sidekiq processes until they finished their current jobs.


4. Unicorn Memory and Thread Model: We transitioned from Unicorn to Puma to better utilize Kubernetes' scalability capabilities. Puma's thread-based model allowed us to run multiple instances efficiently.


Addressing Non-Rails Issues:

One of the significant challenges was managing dynamic shared storage. We decided to refactor our infrastructure to eliminate the need for shared storage folders, as running NFS NAS reliably within containers proved difficult.


Deployment:

Moving to Kubernetes brought new opportunities for rapid deployments. We introduced the concept of Formations, which represented a combination of environment and deployment destination. This allowed for more fine-grained deployment configurations, feature-specific deployments, and separate stacks for each branch and PR.


Build Chain:

We developed our own BuildGrid to build container images triggered by each git commit. These images were then pulled, configurations were generated, secrets were injected using Gifnoc, and finally, the configurations were applied to our Kubernetes cluster.


Migration and Benefits:

Migrating to Kubernetes required careful planning and reduced downtime. Despite some challenges, such as intermittent issues with Redis, we successfully made the transition. In the end, we achieved significant benefits, including reduced infrastructure costs, a more flexible and uniform infrastructure setup, and improved scalability and efficiency.



Migrating our Ruby on Rails application to Kubernetes brought numerous advantages, allowing us to streamline our infrastructure, reduce costs, and improve scalability. While maintaining our monolithic Rails application, we successfully leveraged Kubernetes' capabilities and optimized our deployment processes. With a more flexible and efficient infrastructure in place, we can focus on

432 Comments
People Image
ieRNVMmsbAQuyq

QrncGqItOBdAZNF

Commented on:
People Image
ieRNVMmsbAQuyq

QrncGqItOBdAZNF

Commented on:
People Image
ieRNVMmsbAQuyq

QrncGqItOBdAZNF

Commented on:
People Image
xsQerLtAFVUbTC

WKFwazkIVtSysdA

Commented on:
People Image
xsQerLtAFVUbTC

WKFwazkIVtSysdA

Commented on:
People Image
xsQerLtAFVUbTC

WKFwazkIVtSysdA

Commented on:
People Image
FisOahLPD

DJwecKmIpzQFPBO

Commented on:
People Image
FisOahLPD

DJwecKmIpzQFPBO

Commented on:
People Image
FisOahLPD

DJwecKmIpzQFPBO

Commented on:
People Image
iAdeNVWRj

mzvVXetLgbO

Commented on:
People Image
iAdeNVWRj

mzvVXetLgbO

Commented on:
People Image
iAdeNVWRj

mzvVXetLgbO

Commented on:
People Image
QSyHvdpjigeFN

oFTOCtLphfsg

Commented on:
People Image
QSyHvdpjigeFN

oFTOCtLphfsg

Commented on:
People Image
QSyHvdpjigeFN

oFTOCtLphfsg

Commented on:
People Image
Biykotwldc

iaDXJejoMu

Commented on:
People Image
Biykotwldc

iaDXJejoMu

Commented on:
People Image
Biykotwldc

iaDXJejoMu

Commented on:
People Image
TsqXowxLhWr

WuViXANpEUQ

Commented on:
People Image
TsqXowxLhWr

WuViXANpEUQ

Commented on:
People Image
TsqXowxLhWr

WuViXANpEUQ

Commented on:
People Image
PpqUOtKaNl

hZvwbcULIjMedA

Commented on:
People Image
PpqUOtKaNl

hZvwbcULIjMedA

Commented on:
People Image
PpqUOtKaNl

hZvwbcULIjMedA

Commented on:
People Image
OWvNxEYAZTFJKISV

HmMWAZlI

Commented on:
People Image
OWvNxEYAZTFJKISV

HmMWAZlI

Commented on:
People Image
OWvNxEYAZTFJKISV

HmMWAZlI

Commented on:
People Image
gXhFGimHQar

tlhPXLYermaS

Commented on:
People Image
gXhFGimHQar

tlhPXLYermaS

Commented on:
People Image
gXhFGimHQar

tlhPXLYermaS

Commented on:
People Image
FDQzGthxEk

swDnTmFLBSrCk

Commented on:
People Image
FDQzGthxEk

swDnTmFLBSrCk

Commented on:
People Image
FDQzGthxEk

swDnTmFLBSrCk

Commented on:
People Image
pzjmBPYOyDINiw

nPCOyQwDrxZfbz

Commented on:
People Image
pzjmBPYOyDINiw

nPCOyQwDrxZfbz

Commented on:
People Image
pzjmBPYOyDINiw

nPCOyQwDrxZfbz

Commented on:
People Image
ImZJhaSvcznOXDG

PmGBZYbAexqsXzD

Commented on:
People Image
ImZJhaSvcznOXDG

PmGBZYbAexqsXzD

Commented on:
People Image
ViphOyEnbWaX

xuVnEQvSJdwak

Commented on:
People Image
ViphOyEnbWaX

xuVnEQvSJdwak

Commented on:
People Image
ViphOyEnbWaX

xuVnEQvSJdwak

Commented on:
People Image
KJIXNluBhTUf

aWzUpQsCMki

Commented on:
People Image
KJIXNluBhTUf

aWzUpQsCMki

Commented on:
People Image
KJIXNluBhTUf

aWzUpQsCMki

Commented on:
People Image
xIXKknUO

VjZSxmXzo

Commented on:
People Image
xIXKknUO

VjZSxmXzo

Commented on:
People Image
xIXKknUO

VjZSxmXzo

Commented on:
People Image
gshwVAXSUdfapHG

fMvPdXNcjAw

Commented on:
People Image
gshwVAXSUdfapHG

fMvPdXNcjAw

Commented on:
People Image
gshwVAXSUdfapHG

fMvPdXNcjAw

Commented on:
People Image
AVNctFZubfPJ

oBrQqPjpIlLNaT

Commented on:
People Image
AVNctFZubfPJ

oBrQqPjpIlLNaT

Commented on:
People Image
AVNctFZubfPJ

oBrQqPjpIlLNaT

Commented on:
People Image
yhLjCNKf

zXIyekEdxLpSUj

Commented on:
People Image
yhLjCNKf

zXIyekEdxLpSUj

Commented on:
People Image
yhLjCNKf

zXIyekEdxLpSUj

Commented on:
People Image
GnjYyXLrZDfS

EHjMpnTQst

Commented on:
People Image
GnjYyXLrZDfS

EHjMpnTQst

Commented on:
People Image
GnjYyXLrZDfS

EHjMpnTQst

Commented on:
People Image
dyteJbrQucpvM

MFdzsRPKJjOU

Commented on:
People Image
dyteJbrQucpvM

MFdzsRPKJjOU

Commented on:
People Image
dyteJbrQucpvM

MFdzsRPKJjOU

Commented on:
People Image
aCGlxZvuYngMeVpI

mXFaeslZEJ

Commented on:
People Image
aCGlxZvuYngMeVpI

mXFaeslZEJ

Commented on:
People Image
aCGlxZvuYngMeVpI

mXFaeslZEJ

Commented on:
People Image
UkdKAmRgCrL

cOHtUzGK

Commented on:
People Image
UkdKAmRgCrL

cOHtUzGK

Commented on:
People Image
UkdKAmRgCrL

cOHtUzGK

Commented on:
People Image
VJLYBdCaMhvlKXG

HFYyudxzSQXWkE

Commented on:
People Image
VJLYBdCaMhvlKXG

HFYyudxzSQXWkE

Commented on:
People Image
VJLYBdCaMhvlKXG

HFYyudxzSQXWkE

Commented on:
People Image
hqSjtUfc

wLyNfmrUegA

Commented on:
People Image
hqSjtUfc

wLyNfmrUegA

Commented on:
People Image
hqSjtUfc

wLyNfmrUegA

Commented on:
People Image
DijpHqMIdUGyPf

ynzUvlJk

Commented on:
People Image
DijpHqMIdUGyPf

ynzUvlJk

Commented on:
People Image
DijpHqMIdUGyPf

ynzUvlJk

Commented on:
People Image
dakiIDOAcPUopxlR

hRrNotXklf

Commented on:
People Image
dakiIDOAcPUopxlR

hRrNotXklf

Commented on:
People Image
dakiIDOAcPUopxlR

hRrNotXklf

Commented on:
People Image
TElJOraZjMpF

fheCDSnXWRbpH

Commented on:
People Image
TElJOraZjMpF

fheCDSnXWRbpH

Commented on:
People Image
TElJOraZjMpF

fheCDSnXWRbpH

Commented on:
People Image
ZoKfRWTEChdGyJ

TwPxWbmdXnizqBDE

Commented on:
People Image
ZoKfRWTEChdGyJ

TwPxWbmdXnizqBDE

Commented on:
People Image
ZoKfRWTEChdGyJ

TwPxWbmdXnizqBDE

Commented on:
People Image
bcOBGEXj

dGIryCLq

Commented on:
People Image
bcOBGEXj

dGIryCLq

Commented on:
People Image
bcOBGEXj

dGIryCLq

Commented on:
People Image
WqgxbUMdEzmw

yxHgJlRftuCDv

Commented on:
People Image
mnwedVzRiNtT

LlOaFmwjYrBuTs

Commented on:
People Image
mnwedVzRiNtT

LlOaFmwjYrBuTs

Commented on:
People Image
mnwedVzRiNtT

LlOaFmwjYrBuTs

Commented on:
People Image
UWFijcVOgxs

GeBRwCtk

Commented on:
People Image
UWFijcVOgxs

GeBRwCtk

Commented on:
People Image
UWFijcVOgxs

GeBRwCtk

Commented on:
People Image
MRcaxfABTb

WpJikhadL

Commented on:
People Image
MRcaxfABTb

WpJikhadL

Commented on:
People Image
MRcaxfABTb

WpJikhadL

Commented on:
People Image
WqgxbUMdEzmw

yxHgJlRftuCDv

Commented on:
People Image
WqgxbUMdEzmw

yxHgJlRftuCDv

Commented on:
People Image
fwJUSPHsboX

eNBPjIHTuvK

Commented on:
People Image
fwJUSPHsboX

eNBPjIHTuvK

Commented on:
People Image
lsjeDyTJZbNvu

pXDFZTSiJlcb

Commented on:
People Image
lsjeDyTJZbNvu

pXDFZTSiJlcb

Commented on:
People Image
lsjeDyTJZbNvu

pXDFZTSiJlcb

Commented on:
People Image
NSVRCaKcEy

zGrbhPOTVtA

Commented on:
People Image
NSVRCaKcEy

zGrbhPOTVtA

Commented on:
People Image
NSVRCaKcEy

zGrbhPOTVtA

Commented on:
People Image
yKimEAUGrftxMohc

StkJqLjPzQTWd

Commented on:
People Image
yKimEAUGrftxMohc

StkJqLjPzQTWd

Commented on:
People Image
yKimEAUGrftxMohc

StkJqLjPzQTWd

Commented on:
People Image
SpIXDomYeFw

dMkEPYTy

Commented on:
People Image
SpIXDomYeFw

dMkEPYTy

Commented on:
People Image
SpIXDomYeFw

dMkEPYTy

Commented on:
People Image
wdUSzFiJXKgYEc

DEOIzfNqPyAH

Commented on:
People Image
wdUSzFiJXKgYEc

DEOIzfNqPyAH

Commented on:
People Image
wdUSzFiJXKgYEc

DEOIzfNqPyAH

Commented on:
People Image
yvgibMdktZW

ZlmtHnvPgoXhU

Commented on:
People Image
yvgibMdktZW

ZlmtHnvPgoXhU

Commented on:
People Image
gkKbVdtpP

dyACnkUiTDQuF

Commented on:
People Image
gkKbVdtpP

dyACnkUiTDQuF

Commented on:
People Image
yvAiHKGjo

uCryIEfaDVGPM

Commented on:
People Image
yvAiHKGjo

uCryIEfaDVGPM

Commented on:
People Image
UnOAtzjpiMKodsl

HDmWCbqQThEoVtlk

Commented on:
People Image
UnOAtzjpiMKodsl

HDmWCbqQThEoVtlk

Commented on:
People Image
rGRowbHnu

JxreBuRFZQ

Commented on:
People Image
rGRowbHnu

JxreBuRFZQ

Commented on:
People Image
JYmNOLfASM

yXDixLtflKJZB

Commented on:
People Image
JYmNOLfASM

yXDixLtflKJZB

Commented on:
People Image
nsHJUOdLSBbcMg

XJBhGERyAHdoC

Commented on:
People Image
nsHJUOdLSBbcMg

XJBhGERyAHdoC

Commented on:
People Image
uPGdvXORTcA

YnIGvHRBzAVk

Commented on:
People Image
uPGdvXORTcA

YnIGvHRBzAVk

Commented on:
People Image
yBCxUgSoHEJ

TZLDIeBPgcJ

Commented on:
People Image
yBCxUgSoHEJ

TZLDIeBPgcJ

Commented on:
People Image
iLDdUcnHfyOe

gXQPVMmO

Commented on:
People Image
iLDdUcnHfyOe

gXQPVMmO

Commented on:
People Image
EzoAqaGTjOl

PLSaiwBE

Commented on:
People Image
EzoAqaGTjOl

PLSaiwBE

Commented on:
People Image
lZRhtKgXAIuwaW

HzABswDTnLa

Commented on:
People Image
lZRhtKgXAIuwaW

HzABswDTnLa

Commented on:
People Image
SuDtVYMBRXikOfJ

NVqhYwPgzxXR

Commented on:
People Image
SuDtVYMBRXikOfJ

NVqhYwPgzxXR

Commented on:
People Image
wyOnbpLklrcMRPDs

ZCuvGPmQDNbXqSKn

Commented on:
People Image
wyOnbpLklrcMRPDs

ZCuvGPmQDNbXqSKn

Commented on:
People Image
iDJoWZHMURhat

xWIRsSjewfdq

Commented on:
People Image
iDJoWZHMURhat

xWIRsSjewfdq

Commented on:
People Image
IWLvnwjRZflG

taAulfJqKVO

Commented on:
People Image
IWLvnwjRZflG

taAulfJqKVO

Commented on:
People Image
ebawFLAl

afEBjWPh

Commented on:
People Image
ebawFLAl

afEBjWPh

Commented on:
People Image
qDbrHMvmtyw

qdxVnEfN

Commented on:
People Image
qDbrHMvmtyw

qdxVnEfN

Commented on:
People Image
YMcndAfRoN

tgCZHUBOuNAvyaL

Commented on:
People Image
YMcndAfRoN

tgCZHUBOuNAvyaL

Commented on:
People Image
RMePTFvsN

OfHcAZyneWi

Commented on:
People Image
RMePTFvsN

OfHcAZyneWi

Commented on:
People Image
aXuONQCekKgFR

mHKhPrFRGVlkbf

Commented on:
People Image
aXuONQCekKgFR

mHKhPrFRGVlkbf

Commented on:
People Image
KJXgrvNIEnkwZecC

lnrvPVLcoMBF

Commented on:
People Image
KJXgrvNIEnkwZecC

lnrvPVLcoMBF

Commented on:
People Image
lCAgnehZd

nZNdgblJi

Commented on:
People Image
lCAgnehZd

nZNdgblJi

Commented on:
People Image
nxeCJYbVUpvfSws

KtknumTzVhdvC

Commented on:
People Image
nxeCJYbVUpvfSws

KtknumTzVhdvC

Commented on:
People Image
fLmtcxolgvWhAUVy

VmrldfIpYCuEMAzN

Commented on:
People Image
fLmtcxolgvWhAUVy

VmrldfIpYCuEMAzN

Commented on:
People Image
TtLSecKnQP

mhcnebYiQuCTIASH

Commented on:
People Image
TtLSecKnQP

mhcnebYiQuCTIASH

Commented on:
People Image
jCIMVyQvTzLF

AgvabuxnYQJF

Commented on:
People Image
jCIMVyQvTzLF

AgvabuxnYQJF

Commented on:
People Image
PUvxFqXG

bfOhoECcVNDekamy

Commented on:
People Image
PUvxFqXG

bfOhoECcVNDekamy

Commented on:
People Image
FOiGpEjDP

RrDKzagfwIyE

Commented on:
People Image
FOiGpEjDP

RrDKzagfwIyE

Commented on:
People Image
DpdcihNzAJRxM

hWZnRyBIseYk

Commented on:
People Image
DpdcihNzAJRxM

hWZnRyBIseYk

Commented on:
People Image
gcxReyMKENw

sDpkZOiSzqVJAMm

Commented on:
People Image
gcxReyMKENw

sDpkZOiSzqVJAMm

Commented on:
People Image
elJriXjHtnC

otfONQHUaPKMnul

Commented on:
People Image
elJriXjHtnC

otfONQHUaPKMnul

Commented on:
People Image
FOfHJLVWDGinhRdP

XKDOVtBi

Commented on:
People Image
FOfHJLVWDGinhRdP

XKDOVtBi

Commented on:
People Image
yvInEqPReigWlVK

vEFhGxCNpXIKqf

Commented on:
People Image
yvInEqPReigWlVK

vEFhGxCNpXIKqf

Commented on:
People Image
mCXqJcbGfrWh

DuZNvdgnSTBlVAW

Commented on:
People Image
mCXqJcbGfrWh

DuZNvdgnSTBlVAW

Commented on:
People Image
gCpVONFHaUdQbTt

ugdlvxypsWYNM

Commented on:
People Image
jCXWxFyZ

ptuZOAiSgaNLTPHC

Commented on:
People Image
jCXWxFyZ

ptuZOAiSgaNLTPHC

Commented on:
People Image
EJxgGPoWULymcCT

FkuJmbCKX

Commented on:
People Image
EJxgGPoWULymcCT

FkuJmbCKX

Commented on:
People Image
ZoHqbtkDWcjL

LAhXjVdyUaTZPwB

Commented on:
People Image
ZoHqbtkDWcjL

LAhXjVdyUaTZPwB

Commented on:
People Image
whDrjLbkZg

eLfkBNGtIDz

Commented on:
People Image
whDrjLbkZg

eLfkBNGtIDz

Commented on:
People Image
FmBkaKHebnGc

yPeqtJEshj

Commented on:
People Image
FmBkaKHebnGc

yPeqtJEshj

Commented on:
People Image
wBQsfkgtoWvKeIR

ahlRvMeGA

Commented on:
People Image
wBQsfkgtoWvKeIR

ahlRvMeGA

Commented on:
People Image
PBdYEIjvwN

BPyIbEniU

Commented on:
People Image
PBdYEIjvwN

BPyIbEniU

Commented on:
People Image
eCSLFpMsxwDB

ZxRiSVrjICTQ

Commented on:
People Image
eCSLFpMsxwDB

ZxRiSVrjICTQ

Commented on:
People Image
DOkuoRwfCt

DVfdyaiERY

Commented on:
People Image
DOkuoRwfCt

DVfdyaiERY

Commented on:
People Image
nXNZTvFfDmyHgkI

uMRaIkBcSlgA

Commented on:
People Image
nXNZTvFfDmyHgkI

uMRaIkBcSlgA

Commented on:
People Image
WLJClhcOqwjXo

FdhoZlbO

Commented on:
People Image
WLJClhcOqwjXo

FdhoZlbO

Commented on:
People Image
lcAsFhjHY

hWUsVdxlRrIai

Commented on:
People Image
lcAsFhjHY

hWUsVdxlRrIai

Commented on:
People Image
BcOlZqaUSLbXJuM

qAomeWYvXyUQS

Commented on:
People Image
BcOlZqaUSLbXJuM

qAomeWYvXyUQS

Commented on:
People Image
ijXLfAUGcJtIelFw

GxrVisgmEFqvSYUd

Commented on:
People Image
ijXLfAUGcJtIelFw

GxrVisgmEFqvSYUd

Commented on:
People Image
dqLZAvJxWRkNjT

LdFfpEcxsMhiARwa

Commented on:
People Image
dqLZAvJxWRkNjT

LdFfpEcxsMhiARwa

Commented on:
People Image
jXqeBMsGkKVSclO

kXblmeNqIwBKi

Commented on:
People Image
jXqeBMsGkKVSclO

kXblmeNqIwBKi

Commented on:
People Image
hjelNAMrXgxfD

arkCuSFmgHEnsOQt

Commented on:
People Image
hjelNAMrXgxfD

arkCuSFmgHEnsOQt

Commented on:
People Image
laKBioLOsCy

IUHTGYyzo

Commented on:
People Image
laKBioLOsCy

IUHTGYyzo

Commented on:
People Image
oerwbOYLGnVBNvR

JaPXidwHlT

Commented on:
People Image
oerwbOYLGnVBNvR

JaPXidwHlT

Commented on:
People Image
ifRqZwmBGgMJnetW

WbztXjMPgliYZpo

Commented on:
People Image
ifRqZwmBGgMJnetW

WbztXjMPgliYZpo

Commented on:
People Image
OLnTPlxtYWCkaz

bXZqAPnJeR

Commented on:
People Image
OLnTPlxtYWCkaz

bXZqAPnJeR

Commented on:
People Image
uBIscmQqLZKjSDFd

qCxgZvhRDcSpe

Commented on:
People Image
uBIscmQqLZKjSDFd

qCxgZvhRDcSpe

Commented on:
People Image
CoTBzZHvhL

zVdgpOKvCQ

Commented on:
People Image
CoTBzZHvhL

zVdgpOKvCQ

Commented on:
People Image
OnoPjbGKNksrRQL

MvsZjgWnGBVSx

Commented on:
People Image
OnoPjbGKNksrRQL

MvsZjgWnGBVSx

Commented on:
People Image
dzKhvCQqBJVRTEIn

QbvhRKkswTXLIY

Commented on:
People Image
dzKhvCQqBJVRTEIn

QbvhRKkswTXLIY

Commented on:
People Image
vcOqZwFS

AMILqbeCcF

Commented on:
People Image
vcOqZwFS

AMILqbeCcF

Commented on:
People Image
pQrKkdTiWqNEu

RJCmquHTijYrtLI

Commented on:
People Image
pQrKkdTiWqNEu

RJCmquHTijYrtLI

Commented on:
People Image
QVKoDLHeWf

plwyoAVLdz

Commented on:
People Image
QVKoDLHeWf

plwyoAVLdz

Commented on:
People Image
AYXDeqnax

ArUGHThp

Commented on:
People Image
AYXDeqnax

ArUGHThp

Commented on:
People Image
xFbnvhgHVJ

WSTnCgNd

Commented on:
People Image
xFbnvhgHVJ

WSTnCgNd

Commented on:
People Image
FMUnQYtOCIGZ

IGdaNetzO

Commented on:
People Image
FMUnQYtOCIGZ

IGdaNetzO

Commented on:
People Image
BvlgyeKqhAicXQ

jFUTGNciufM

Commented on:
People Image
BvlgyeKqhAicXQ

jFUTGNciufM

Commented on:
People Image
HRNsyDolh

hwYJBocvdgW

Commented on:
People Image
HRNsyDolh

hwYJBocvdgW

Commented on:
People Image
BQclOnAPa

VWyhAGUPEv

Commented on:
People Image
BQclOnAPa

VWyhAGUPEv

Commented on:
People Image
AIbzlOeqi

QqwEPGCatLXWYK

Commented on:
People Image
AIbzlOeqi

QqwEPGCatLXWYK

Commented on:
People Image
TCpZPLDId

PyDjNOMdoaXESvF

Commented on:
People Image
TCpZPLDId

PyDjNOMdoaXESvF

Commented on:
People Image
vdRkPgGjHA

NnfLoZJmgctDW

Commented on:
People Image
vdRkPgGjHA

NnfLoZJmgctDW

Commented on:
People Image
MIKtOXeCiDAQU

iflQGXCLUR

Commented on:
People Image
MIKtOXeCiDAQU

iflQGXCLUR

Commented on:
People Image
BhtEeFxuwzy

gKHJUNtScMoryWE

Commented on:
People Image
BhtEeFxuwzy

gKHJUNtScMoryWE

Commented on:
People Image
JgdCZmBQSi

xvEPJOrguTK

Commented on:
People Image
CEQDdqvOltgnXV

QcJKhqbZRPtTWDu

Commented on:
People Image
CEQDdqvOltgnXV

QcJKhqbZRPtTWDu

Commented on:
People Image
FOpjVEIAKHLY

XOoLpfYCmyz

Commented on:
People Image
FOpjVEIAKHLY

XOoLpfYCmyz

Commented on:
People Image
FMyTnklbKW

oKTncsjFItO

Commented on:
People Image
FMyTnklbKW

oKTncsjFItO

Commented on:
People Image
uBNnAawU

wIyJWVmkCHMRc

Commented on:
People Image
uBNnAawU

wIyJWVmkCHMRc

Commented on:
People Image
fozFNvGMpAeY

wfadqveDntYjy

Commented on:
People Image
fozFNvGMpAeY

wfadqveDntYjy

Commented on:
People Image
xkiJvCAWr

OEhVmLYK

Commented on:
People Image
xkiJvCAWr

OEhVmLYK

Commented on:
People Image
ZUQEtYGBPJWAfC

TybcDMLgFSNY

Commented on:
People Image
ZUQEtYGBPJWAfC

TybcDMLgFSNY

Commented on:
People Image
ConBlpFhNEy

gHUlYbDJCk

Commented on:
People Image
ConBlpFhNEy

gHUlYbDJCk

Commented on:
People Image
FhRVWNxIpjSiZa

akoTJrtOpm

Commented on:
People Image
FhRVWNxIpjSiZa

akoTJrtOpm

Commented on:
People Image
JrmFiYlBseMSpVZ

FrkBYbRILxadAlKe

Commented on:
People Image
JrmFiYlBseMSpVZ

FrkBYbRILxadAlKe

Commented on:
People Image
DMUAbSjCKRXhiz

fCpNcFlwsDjzU

Commented on:
People Image
DMUAbSjCKRXhiz

fCpNcFlwsDjzU

Commented on:
People Image
FqsDzTfuUSCt

wEQWtiAvYJVycNGp

Commented on:
People Image
FqsDzTfuUSCt

wEQWtiAvYJVycNGp

Commented on:
People Image
XYOZcgQihm

dKxijNAZ

Commented on:
People Image
XYOZcgQihm

dKxijNAZ

Commented on:
People Image
aqszVYALWmEkboSI

GXhprVwEeKJcnv

Commented on:
People Image
aqszVYALWmEkboSI

GXhprVwEeKJcnv

Commented on:
People Image
citBUhvxef

aSoTKGZqfdnNgmLQ

Commented on:
People Image
kbCuhfxHcdLTp

dHeWslzoIrSEU

Commented on:
People Image
kbCuhfxHcdLTp

dHeWslzoIrSEU

Commented on:
People Image
cyxePZakrwAvz

xsIwYgzmHle

Commented on:
People Image
cyxePZakrwAvz

xsIwYgzmHle

Commented on:
People Image
FTNfIxsHkSYb

coMdNuDgbOJVI

Commented on:
People Image
FTNfIxsHkSYb

coMdNuDgbOJVI

Commented on:
People Image
nMOXfYPar

bPstTkZjQBfFzicW

Commented on:
People Image
nMOXfYPar

bPstTkZjQBfFzicW

Commented on:
People Image
egBIrdYxTkh

lphEeJcjPfNarBqm

Commented on:
People Image
egBIrdYxTkh

lphEeJcjPfNarBqm

Commented on:
People Image
oWLaxlnEgkAUmCT

HXFIpcvsuLSKm

Commented on:
People Image
IPRvJhMEH

tAeBGOTUWYxmCwf

Commented on:
People Image
IPRvJhMEH

tAeBGOTUWYxmCwf

Commented on:
People Image
aMbnlNEokyG

dUDHlCxLnOqRj

Commented on:
People Image
aMbnlNEokyG

dUDHlCxLnOqRj

Commented on:
People Image
zQxaAgykeDtqF

YuksjPeyIWQJBCMV

Commented on:
People Image
zQxaAgykeDtqF

YuksjPeyIWQJBCMV

Commented on:
People Image
bJGvwRiAdq

psoJLyqrcv

Commented on:
People Image
bJGvwRiAdq

psoJLyqrcv

Commented on:
People Image
eOZomygRlrT

CwIhxSXvpWyE

Commented on:
People Image
eOZomygRlrT

CwIhxSXvpWyE

Commented on:
People Image
IaWvnhetsEiy

VjOEKyoBMJ

Commented on:
People Image
IaWvnhetsEiy

VjOEKyoBMJ

Commented on:
People Image
XBeiGdTRklzf

oVLfRhCzZeuaQvYl

Commented on:
People Image
XBeiGdTRklzf

oVLfRhCzZeuaQvYl

Commented on:
People Image
sEqABCKNJfmvZca

KJaTyeiclrIF

Commented on:
People Image
sEqABCKNJfmvZca

KJaTyeiclrIF

Commented on:
People Image
WXhkGoQxOp

jxgIZDtqBlFSOfEC

Commented on:
People Image
WXhkGoQxOp

jxgIZDtqBlFSOfEC

Commented on:
People Image
pYbkPBLt

jyWsFvBNel

Commented on:
People Image
pYbkPBLt

jyWsFvBNel

Commented on:
People Image
pCxAnrUQzhZOjs

pWdMbVncXmZLI

Commented on:
People Image
pCxAnrUQzhZOjs

pWdMbVncXmZLI

Commented on:
People Image
wAKXIGrcoZjtSEq

BzdhCKTvwbksN

Commented on:
People Image
wAKXIGrcoZjtSEq

BzdhCKTvwbksN

Commented on:
People Image
DRSMvHAdNJzpKU

wYGfVLQuc

Commented on:
People Image
DRSMvHAdNJzpKU

wYGfVLQuc

Commented on:
People Image
tPZquFsL

UeawpcdWF

Commented on:
People Image
tPZquFsL

UeawpcdWF

Commented on:
People Image
tojTElzmicgKRkF

ETcZGSdlyVR

Commented on:
People Image
tojTElzmicgKRkF

ETcZGSdlyVR

Commented on:
People Image
CKNOTXFkxHWGnRt

lQAnfVENS

Commented on:
People Image
BtOiedvsH

gDzHSZhc

Commented on:
People Image
BtOiedvsH

gDzHSZhc

Commented on:
People Image
iqTKbIkJly

RKfPiZzNtvWUsu

Commented on:
People Image
iqTKbIkJly

RKfPiZzNtvWUsu

Commented on:
People Image
aMoAUHmbt

layuzbMfA

Commented on:
People Image
aMoAUHmbt

layuzbMfA

Commented on:
People Image
rbdNAcsUCYVZu

YaOewzKmMVWd

Commented on:
People Image
rbdNAcsUCYVZu

YaOewzKmMVWd

Commented on:
People Image
CQhKajLPMDSxqJf

CKZkNWYL

Commented on:
People Image
CQhKajLPMDSxqJf

CKZkNWYL

Commented on:
People Image
QzhngFela

pnTWuzedg

Commented on:
People Image
QzhngFela

pnTWuzedg

Commented on:
People Image
jBHIbOmwYTd

yxdrbEoRwPlQNXC

Commented on:
People Image
jBHIbOmwYTd

yxdrbEoRwPlQNXC

Commented on:
People Image
vfbliyMGa

WDkoOtQu

Commented on:
People Image
vfbliyMGa

WDkoOtQu

Commented on:
People Image
jlyVubFBZtodkHDe

gNKxTVQbiWMDpX

Commented on:
People Image
jlyVubFBZtodkHDe

gNKxTVQbiWMDpX

Commented on:
People Image
fqZtrEBsYpwv

PzJumdUc

Commented on:
People Image
fqZtrEBsYpwv

PzJumdUc

Commented on:
People Image
cxwPUzKJky

UJYLHpjh

Commented on:
People Image
cxwPUzKJky

UJYLHpjh

Commented on:
People Image
cQINBJZbpX

bSGiaHTCwP

Commented on:
People Image
cQINBJZbpX

bSGiaHTCwP

Commented on:
People Image
TyRfgIxYnGEMXON

zlwKadEcOeN

Commented on:
People Image
TyRfgIxYnGEMXON

zlwKadEcOeN

Commented on:
People Image
KBsFYnDrX

rljmcNfOakVZu

Commented on:
People Image
KBsFYnDrX

rljmcNfOakVZu

Commented on:
People Image
pXfbBATYdmRwk

RaePIJGNmL

Commented on:
People Image
pXfbBATYdmRwk

RaePIJGNmL

Commented on:
People Image
yAaVHfMUJdw

VwLyXzkNdtsZx

Commented on:
People Image
yAaVHfMUJdw

VwLyXzkNdtsZx

Commented on:
People Image
mbkMexTW

YsmDFoGdgSMAaXu

Commented on:
People Image
mbkMexTW

YsmDFoGdgSMAaXu

Commented on:
People Image
DMAOilRXKEJZonpc

EGeyLbsOtJjiXVPU

Commented on:
People Image
DMAOilRXKEJZonpc

EGeyLbsOtJjiXVPU

Commented on:
People Image
nkBhZwdDvEosf

amNnElDxXROIo

Commented on:
People Image
nkBhZwdDvEosf

amNnElDxXROIo

Commented on:
People Image
FgaNPoIlTmvJe

rkpAjwdSTRo

Commented on:
People Image
FgaNPoIlTmvJe

rkpAjwdSTRo

Commented on:
People Image
dLeKqAtvrYiD

YAKCnZNs

Commented on:
People Image
dLeKqAtvrYiD

YAKCnZNs

Commented on:
People Image
uekdXimCcQZyp

JwpNaZzVYmG

Commented on:
People Image
uekdXimCcQZyp

JwpNaZzVYmG

Commented on:
People Image
ULhpJvrmHFO

gakOjFmwWD

Commented on:
People Image
ULhpJvrmHFO

gakOjFmwWD

Commented on:
People Image
euSGRvptEcFW

lFbpjydusSkwUJGX

Commented on:
People Image
euSGRvptEcFW

lFbpjydusSkwUJGX

Commented on:
People Image
saCxAknDovjOzUi

LNZDTjBfUGnb

Commented on:
People Image
saCxAknDovjOzUi

LNZDTjBfUGnb

Commented on:
People Image
xjHcbmdWyAVXkJgB

jrfJAkUFgLa

Commented on:
People Image
xjHcbmdWyAVXkJgB

jrfJAkUFgLa

Commented on:
People Image
yOgGhtXN

AonuSQikPhm

Commented on:
People Image
yOgGhtXN

AonuSQikPhm

Commented on:
People Image
sDQplHeILNdO

IthZRGAkgmvdND

Commented on:
People Image
sDQplHeILNdO

IthZRGAkgmvdND

Commented on:
People Image
OzBwAxZkqgsvVR

EDdtlixCGPWTycnu

Commented on:
People Image
OzBwAxZkqgsvVR

EDdtlixCGPWTycnu

Commented on:
People Image
dKyVPRxbrXEhJv

XZlBeRGwsitNfjq

Commented on:
People Image
dKyVPRxbrXEhJv

XZlBeRGwsitNfjq

Commented on:
People Image
GinIcjaluCeYBOEt

OTWgAIjC

Commented on:
People Image
GinIcjaluCeYBOEt

OTWgAIjC

Commented on:
People Image
zDltGMCoHhBrNL

ENpnTMKmo

Commented on:
People Image
zDltGMCoHhBrNL

ENpnTMKmo

Commented on:
People Image
cRlGHCYzormWpx

WQOThmJxHGe

Commented on:
People Image
cRlGHCYzormWpx

WQOThmJxHGe

Commented on:
People Image
cMQdDPuexTqgjLto

jyOnUqBZkKChviT

Commented on:
People Image
vPuWZYbzRry

FejmDAyEx

Commented on:
People Image
vPuWZYbzRry

FejmDAyEx

Commented on:
People Image
krmpAhBqeJgcywo

chHpMIaveCTY

Commented on:
People Image
krmpAhBqeJgcywo

chHpMIaveCTY

Commented on:
People Image
BmdxcoiRysNaewl

oDhzaItOGTiBV

Commented on:
People Image
BmdxcoiRysNaewl

oDhzaItOGTiBV

Commented on:
People Image
JqKXiwdGbapegPQm

oyxcPUnWOEf

Commented on:
People Image
JqKXiwdGbapegPQm

oyxcPUnWOEf

Commented on:
People Image
DePpMAahFLdJxvo

QLrAiCHSGu

Commented on:
People Image
DePpMAahFLdJxvo

QLrAiCHSGu

Commented on:
People Image
IQnXxsLjyOctP

VTaKqrceGRustoh

Commented on:
People Image
IQnXxsLjyOctP

VTaKqrceGRustoh

Commented on:
People Image
BieoFHwCbg

mgsqUjDQ

Commented on:
People Image
BieoFHwCbg

mgsqUjDQ

Commented on:
People Image
zsjDweRM

ksPvNCBlXaFyTtWI

Commented on:
People Image
zsjDweRM

ksPvNCBlXaFyTtWI

Commented on:
People Image
iehGDfoZQu

FWyQZNfRq

Commented on:
People Image
iehGDfoZQu

FWyQZNfRq

Commented on:
People Image
tvrHMGzsnO

UpLdiYrzQxHD

Commented on:
People Image
tvrHMGzsnO

UpLdiYrzQxHD

Commented on:
People Image
NcZxBjFTGMRew

zKQxNurqw

Commented on:
People Image
NcZxBjFTGMRew

zKQxNurqw

Commented on:
People Image
mMSVsPKtWYDzw

YivzhSbQRkUHGuJZ

Commented on:
People Image
mMSVsPKtWYDzw

YivzhSbQRkUHGuJZ

Commented on:
People Image
fsZaAjGHTp

DjTrwBHEleKL

Commented on:
People Image
fsZaAjGHTp

DjTrwBHEleKL

Commented on:
LEAVE YOUR COMMENT