From 80c4aa0a8696714e8d5960982bbd690811e3dc30 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 16 Nov 2021 10:25:58 +0000 Subject: [PATCH 1/7] Move and update Database Load Balancing docs --- .../img/db_load_balancing_postgres_stats.png | Bin 21543 -> 0 bytes doc/administration/index.md | 4 +- doc/administration/logs.md | 2 +- .../database_load_balancing.md | 296 ++++++++---------- .../licensed_feature_availability.md | 2 +- .../merge_request_performance_guidelines.md | 4 +- doc/development/scalability.md | 3 +- doc/development/sidekiq_style_guide.md | 2 +- 8 files changed, 142 insertions(+), 171 deletions(-) delete mode 100644 doc/administration/img/db_load_balancing_postgres_stats.png rename doc/administration/{ => postgresql}/database_load_balancing.md (51%) diff --git a/doc/administration/img/db_load_balancing_postgres_stats.png b/doc/administration/img/db_load_balancing_postgres_stats.png deleted file mode 100644 index 8b311616e7b600ed5191a13821faf25eaafbdf97..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 21543 zcmeAS@N?(olHy`uVBq!ia0y~yV1B{C!1#rOnSp^}=C|+i3=9mM1s;*b3=G`DAk4@x zYmNj112aQ_Pl&6$tgM2ZtiGO}ny#L*g1n-$nuxNL%xeNdFAiVmD2 z@A?kU>K1QfWtYMV6W^fu_4>Ehn4dd;A@}_KxROu@J3D6!rAkk&m)k8j^%}0}FxZlA zZ0O;=eWKC+0|y;4@+Xz*gj=X~=j*g&Y0s|KYY)_!9-$Z5H)HqiJ%MH_8b&6myH2%-<_EzMr!i)KH9b>ivFFGi(@sCwjR1wWtkbOo|c;0+uIvg z8g?kpR9#Ii`{a$hNDXsMh5jO)(?w?H1`1E=tw*F&tO!S>x zPn|x!W$RWe50mC%=g=arU{{sptp<@1k>SOp^oDawj#$SgW&n$z}-eEEm*y)$~~buBriAD#@f1cN!t0vrs|r?NmY?M>uhpo z^lU3Nu3pf-bm_A78#b8eD(v02KfkM3OJ8kKwfCB}>mEFKFtO2gYJvU1T1zJfhv}WR z8*=TF101IAm^iOLHGRRnU{A;TWBW3q9GiEqDi3v-zM`Tv#$j=W{l*oMGa3^sCpToT zTR*8RynRyMxhDJ4Ec@+EA$zB1*Hs5Kt(~=fx`Tgi{)!IUC6h~TO|Y1GXl_DDSk;8W zlN`QZ97`ctnM7-&A^Wv1N3ctJAA)O$G)922U5qkcwMx=DznA`6+umJ-n_) zJn2BH;xPtoz17(!RS!z_y-KvF?QRNK$(om~VR+N*RBV&bg{fNMtyRZ8w^YT-%?aAE zAbrmhmMVcskL5D-GNm;6D*~KnfAL#<{@0UvU-)1AxfNS}G_-$j^tacq|0bjzpZUJ# z-oDSZZ)?)aW(10~2sm*lq7g5rXdn3~zc*v~3bo3XNh;F?81vs>@i=4ut?S@>&Ry;m zS6|jD{_$S3g=?bdzc=EK>=r10dEB_}q$%SIj*~pQj+;Fy=UgAU(@eamT3l^PUi06H zFC~|szG}U?+0Oa}@7HLNrF(9@E{l5~Hbt=i@Y@|5Lj&JUYrpfk?zGT?=XEbi+S!&V z-}Za4yYyJ&yqD|CtJAN@x_-ZF`=ICsqwK;ZP3!#9L#K%DPhM6Nx1;>lvlt~4^XbO# zqBQ0UCpEL}G8egI78v6^Pjb6Nz!XKpc)RDj7WY&f-E&;kYVm^GIok?(OQP3vT-n{C ztiI>%j|l=-i|1KfU~iql-+ADO@Nc<*{d_TT4(=K!7xZ(KZkU(-a^AkzqTfDT*n2ze zwB$@dbCV0tb0oPBE$rK;YkB$ap}7mU?Ov9b7;3rpd_a_of;!JG&HY6&9b!-G9KN`3 zi_LmrCo2%_Rlapb_d9F#1=F^-F~}~IXZrFnPWIwCgBRDT8FMS+_Oe z-caRR*6HU%U-@oOpU1Q}c!%z-b!kes^;oOA1s-=9EzOg@x!vkWxcjH44SfNV9KUC4 zSi8LcIBmkQ9gM#kIL>an6HwXRw9k>{SM;)$6M?(ZZ9g*xu4{_BzdrMM@bede7Y^T6 zW3%1#p7ZZ~HU`;+DNT8CJ)5(xUt62k`%&?&Pvr5tm$Do3`0g&>lAUSwLh43YX=Z>$ z(bdAAg-VcXqXob@7K3cu#qDO;8FzloOJ^?PbhmV>)S=K}Y-xYc5xzs0@% z6Ja}xt?k`Pp6cJWjLao}ljiW{r!S9YzWZ{|^N*_*Hwv!js1)1tZn@)iy<=Zri9bGc zTg&gE?5qBUb(tDfhI+f+7YS;-=Vh>RI>}=DVD0~a-HFfs%~@ULyp`?O7vAX{n%9c* zGqMiI?Bai6zdrJpvtECE^TzrUugmzNdgou4G5_P=Ecdrkchg&!z0tA8LZ#{b)d?Z(Xx0+Rxta;v@2?VD@4w~Wcr!zuRj58n&6 zT2CVVgd9EWZoYCqw7`13=Be0=zG@tv%jk6Wqvf-=9jlVD$i{7 z)vWjpT9((1r|2AyeLZ3BgO?j3rW=^M%I`|&zWx2L?A!;Met9n$b~$}0iCDvIb?|cH zM!pv;{)I)%B~8mMb~0MYPP?PQ_o5`CXtBkD%ZVp@?=Z-|e&Uwb;MacCYJsqgtk~-Y zzYir5aPKg5Z=8|7NF9Gf*h)6Y9Fpm34V>t>? z&htf1GA-GD?!KX;dWDtFwQ#SG`%}_@ILF{$-w#Rciz0dXU#1${}ZA1tD^+WQcpcxw=G==6vf&LDmEymZQRmp{dj|4 z-X~H0-0r9U#6U6oL|bm}_p1^o%NeY$o%G%JNOZDcH*0M}Uun@~fiLT399?u7%wAvd z<$Z8zstjAnn#r?kjvlYsc)A^wkksNo=APYny8Sq#?CU3Lu^-Q-ZalsGKl6(Sb@P{9 z?($;kEBHYv=aZ=Zx$f9^3=rL`KP6AU{f`x*X#4hKdf%azZ{51_^kk@g+0}1~#33%6 z7caWo=9lh*&v-afj|Kh*Th?|+@cQgML1UqEg zn+wb4&*!*z>6z>M-^Z>z%lG2kRjD2G!$YoeUUh{AKiJw{`HbM(oR1HB?f<_kXY;K; z;rXX$M2Fs;yhqzTD=GXrqt&&iXWyRD%ANJI?c<)Qo2PTlzVa-8uWEAMv$X|FZP)Iq zl1dJL{$J?AM!np;{29iNE4Aa_tXv&5d;i~MJv&?TTUXy$qy4U7-4-URYfI1OUX@I) zOPc*}&(z8G`MKe<*H07r@+11;qy6mvxL!m|H{H%}Hfw?T%l^cOziGK^Z~ghBDOudT zUsTWUa6?~dX7TNfCv8IYugM=ScvVxq@75o4`^kpeLHhk4$}?MCyKKMR^m)l&6PvX4 zYu4y!YVO=|pXKkuI^ADilaE$reg6JWB{}co_CrTka$VRs@88a~whLZJsdA0{D0rMCU5d=zbXGqb-VhawNDQ`dVeqY z>(_O;sac*@$=+qU*^eFKP1c7@-IlrFy=QLTvzsM{*-Ha@l@aAGcj;8i~Vzl!{X7#9{pludI)kJM|w; zFMGkiU?pS8n&pLg;onr*_A1vft==2*<;~J7&(h+|vfp@SeLO758J@25%Kz+X>D2}- zFCzR8-#H_lI&UF2ztZEVx3SIfOC?G^Z;H%}HtJ57soQw1v^6y~_3T;O+YGX=UrxXM zZ!4$c|92&gKLe|;KUmjaC-FMHJoDGr;@S6&^kz$*-{>9oanWV{t5*AEFKnDww^KG| z{)&_0_g#-|y|tUI{H{+{;c4qOyPY*(Up+lNT|d=sePz4h9*bMw*IdqYcGaJH+xYvo zlGkAl$E&2Cl@-Q1UHD(C@n_@fM1zXoX$CWo`u}_Q%ze7u?cKL)<{b6snsqLRd9mVr`I0wFi@3}7O!;~%(cyU2uczBGj~n-w9DHyw`G5NRTgi(~o%^zIP+&hPAdXWK^K*JiWN+4ft1pYxk-v$4waKgPxS&mY%( zKmF(3&wcM}`b!TsoPC*shP?-mdRKLvzQ-Q*V_$b6e2#(_{g*^Od|OeJ{8B2VH)9+0^aJTmSp}_Wn=)@b}9<`L{ox+I?$Ib>L_I zI+5vC!M3QiX`8Qdy?J=m{>ITK>6<1!y7%BwpXj3^qv+;4EsX056xz0AtxQ|5Wqvle za+1~YAFK0@i|=Qi|DMfp-)Dv22aOiAGrtI5A^lujd~e)ewf!>(&CkUpm#7=6dm zHAe4i<^*o7-SPKkxz`y5_bTqpJa#kCLEre>gJreh_5Y6YefavIU8cPJx+};{|KF`l zbQim*@M`Uvb@HNoq1bgTl zEuF3UG6NKkZXT~px(iR%f6_kaW`8!PgzZZ~z+t)UMUEn2uWo+mIO)6dgzH|W&!L~) zoVW@W*B{s&@i{B!?E&U4bq76Z(-7FpFbuaP{{J7)e z1;>iWzN=YZeM?eV?APp6i|u|sX<@-SE8PX6Qh=gzDyGv)0+Yu;JQy(RDHzhCh4?5$R_R%=-+xtVE+npNIc)t9pj4jGO zf1~G4m&66h*SG!iE*^hr-JRqg_U3g+lg(+K{&j5$!e)VWRu)s8x9?nP*D0#Ix#GL+ zr>M29$_nCsdqku39^9UEP5V>PWeu*C4z-Iq8Ru@?$NMBrURKuE*VmqD@7vwhFCtq` z@A_40#reh9AWU`Jjai4%)Aw(3>HM0o=9G1qO4df6*Cm%$N2QxP#9cX}sW9`-N!7Dg zo;+O^C=$f^Vl&T+p0BwI+oM@|d3mSnn;#Ymntyjux}aD2*VCItEaJ17Rn1 zY1zw{`Px4-Zt33mY3|Ks>A`Ooa|N&b+SsXhCb@N6N5o=%?@OxY?{}W{uw6VC;>v3u z1&{BZ8MwS{yWYjk9b8%Orn_@|x|g?ux9r%h1Dcm-{xaE@(z3Xh`$JX1(XCfL74;wg z{Ppi9ncQnHXKl)QtH`kP!H2MnlAfy^R;hnNYO8~T>+IIP|9y7$#9a>rQykLQxNo{M z;XCiwn8j6V3q$^_`294I>&1r`S$kMpYnN^NAI42XKRwG(w?=8P@NH@H>{NT;J?De|5d{#?j-4CR&ocP`AvN6M^YnGYn zR}HWQ6J|e1%lF$`_jg6@mN~(}8Lc9PzYQJcnIBdC`1tBi{v*|^*R$-C|MYILUtUda z%t`t`@zZ+Zu?$IIJn zjk?yAaUyjiY{+h{hV(v2JT$4%zhI4Nwv#?>gG2}Z9`b}7SCP2uHf~f;-d=>SkD!5Up_z3 z+NSK1bE(s$qw97*$NLlYK2}d5}NY9EzF^hYwN7mW`m5``qqg? zw+}^LdGIpB-l^5V*JZ6~ic_M$?c?5e3{&gfDz7hEd5ky0e7hq%<640i@1{u{ZhiE@ zI(eySn&bt^*AvyZv+ypMtf#eR)#u9BhKi{+BDa|O@BTiPaOI`#od+lDcos7CKG}a$ zSV-vF{^i~NGY`ycynk-Wv}yB}-$|HK-dlF=;UfR3VjAYBj$FU{V$O0omnb(^)9o|P zG@Og*uUSxiMSWu7Bc=;GA72ES9m2xJ){^Fb=_nebM|MI^%p~}m-BO0O|1Q-w&|9( z?3uns`(P>O8*ef+rmUPeJz8f~x_I=O-B)BFW^=F4W|;fIwcW~Oi`YM9L&L(wecKjg zFFMG_fA~VXztRHnCLOaLzIAWc_x>;08KZpstD)D|6z^> zw_c`hWNhrbcE+AeS6A0-&w23;~nc1(XPi$LLJX2+Qkb2}JooQ2M zY5gm(wA!9*e(S&Ex*h?A$5r<#X8)h^B_m9_nNP4Kb^e^9<(ldmPdIlTkJsPiOZ?8bw0I0x`n&;TydYI{eoGe;h0sAmcQG6_CrO_3cemc6J~!x z#5Q`tmK>w_eTK~+(z~-{q3*+zMAauq30#Dm6d>TdSbqG&kAv;J8F6dr+m2L(i&>d5%EJ%z!Q=OVj_3Xf~z0YXK3yIA@%v@HFNFl@80M){});Q zn)Par$h~NL+vA|@lCsk@n_=#jYIV&8+Op>k`aS3W*UWwC;F|CMHK@w5bZoSZh}Xdl-@=Zq^7G5dTXJlG;Ll{Ef5?Cfvb!0Gs7RwKt`J)s8HuN6VWg9V(r4!H~UO7>xyqod-Qu* zVb00e2j7ld&0i~gxu2<#;qB&`MUP|ZS4`?)o;!P;`~HirSFd!t+zB?H@94BBZN|xY zC)!`FDw>@jI%)mIrt7~=RhJyLG5b{bdvWEN^D`v%ggu%HPF?SL@~k20;PpSxjywLU z*~3$MBP-D7*aLIHn_h2I&A$b#IM4twJaqX(_G?#L8yj0gR~M{VzTrdqqB9RCh|lxY z&tWNXud3R0I6J<&Eo^^j)Z;Eqji(Q)|Gp4q`Ldz?M@ki=>Dq$qWX*p)FIRqgH^uIsP6ndA;&~}Si@Z?HqQetp$@Nv-#Ejv#!*j{;(#p&f_&h$Et zr{q+y=o^;6FU9VcZ5=x+NT6$>w+deNKo!gWpfuSF%ym=$+M`)b#wSuCw{ek2RX{28l&kUfB7v>YMai z33c{#zvLs|vQiJPu-u{^Ug)!H|CDF_pUV}zZBpk5EuXyig;Vg=yKCE}&d-Z`^YW!) zB~!&Vai(DJxzEhoP1k#icADJiTYUe|U5D9|k~GpCKq1c8SjV>@<3XBB(B*(HJ`H}; zbJRHWe*~w;-TU*$Cb-O9LdB)z!R+qlGQ$Vu6?@GtJvmn7XmHi4`Ahwl#d8F2Ot*4u zxM#ib$+RN^;`^^oV|?2Zvo+b`!p@h!K1(lcQ}DiYB;m;FBM}Qub61)@nNh;)J-cw{ zwz*t-Gn{8O)IZ#7J^4B-!w=}&J-sJUHTvg~6`}ryKnU=cG?-g9L zPAdEqKD63C)ULl$ljY#Y=hv6s`I|DEC&Qrn#1}LFc^wNIt}n|fGF4MNDEQyu=tGZ} zt-4;@MKfm4N#J@6N+wtGr${BKJnYzETem!a;+1ay-7)K$n9PMI_8B-#-<)~Iab|sj zsd;(nf%SWj%Et)3Vz}_~_>)*iQKL_x=RVx~{PMI&?D_jYGn-*yUUWnE3ww3NvYk6EkN5Mho<7~rBd8$M_}eu0 z>G^rdTIoI>zgd32RzLnBnAPd1{DIq_Pq@hoo3U$7m)Y|@CXY3?a)h2H9=VA{RL}-%?gf-}38N|NTD?*Vi@w|9_G%@XR0f(~bM)-kQ8EJt)*_VS{1c z0}*r8EY<4@8}Ck6cC*+q$LxMbvJm&m1+rzFy8H*Yr^dBk&$FK8bnd*TwDsksN|z6X z+u9_A<%_eN&e*E4c=1ce64%olJ?vAizA*gZ`=ujt@667U!^X)^)}Ph?lk{Je=gW5; zvF^tI36>YO2k$;{E~P(kY0vx*rh-~C^VEKnNgZPR>tvEV>DR_@G7ot$pRAOnS_a#{jX%$pPR%&) z-(`GxXYv19pF7(*D--|s8(z3y|F+rxXVRd2anyi{o?9I7vX=~NPAL&u+FH410?f7q%;r=;z+lxiZmqy*0^hL_D=GpNif5|yV z9+-a@xu{lBv0d=O)~dwA`IrCy@J>2*&^v$cf99;k)mMsOYXO35*St+ z!}YoSCm-`YT`m6K*|)w*{N}v8@3-EX&3-FcXJtA!;%4WuQ?2uBUx&r~F)N&&@%-WO z?fDAdbD3TotNkr9v(NwEv2eA0i}IJ3ozl1*em17Qq2->c2g{Q=Dqp8oeo9a_ikw>0 z#}VPJx~Mcz;?dKo@j+GdH)A_9rj;kS<6F>eKdWg?&_@Nevrt- zUGh^QN7*Ry>iM}mab1$r`Ab#Q)ZX2_?VNQxbGvqqyD;1GWy_{bpFa1be}CX@Ube}d z)1IF+TO*R1o^HRJ_cB}Tyd6JsmE^N0=lfZ|KfdUy`JV5;!*#?afB5vi=TP+YILjZm zZiBMRzqgEkYZ&)+Gzg|G=ALwV(ymq}+hh&1%d@?gW>3uUaoqW2OJz*4y#3w9A)55=zA~gFO$k3QEZC?weOr!;+HT9s zFY~JVpDp!z$Nzg)qm^aZ@)XO6Wn8}ws?}Cz7S|c`rYd=wZsar6KFq;W@>J(g<2m2! z`WKGMu6g!#|Ac^t&ls#N_7-dEIRBpJv%Iuy)$f3bEO{nsp0BQZ?eg|IW^1YL!7jJw z!OK}8rc@Vs0Quq?zRi!-T`&1YOK}Fs4X7Te{Tm5^vXY~lbo3qJ? z({IPjrQy6^7KGi9S^aX0Yunz3cdi5}p1ME(i=^sT*6ix)>hSQsb_FT9)$#Mrm87J2 zWPe(6>RZMix7Y7xXr9SDaBB-cJ9F&S38z6WKKZ~|(knN6`-Td)$Fb(SYwYf?tU6yJ zvEXvj)r&ir=3a_WF+KFWTwG54>2v=$r&mip)ZK91;IWun=dGnv;YYo|pxYt0`!fD4 zJM~m~qDRr0Q`6tC>HM6qUArArqn|&x*=Xvmr>2KqKKqcb=5n^{bkGdF>l?PO$UXVx z0;9vo9-Quv()S{(7w!3wACKu>gmntX_{M)*D`Ic za?&imdVRiC=wc?TrE@m>=QY2)npt9fabiSxprLW|W{$Pl&s9oydb`eE-&=A;`xA%D zr_-nNuTJr2H@kD|!TzGD>m8D%HWwYN-w19vJwIsNHsRaWb$apB&u_n?H>c-{LFTHJ zn)>?d_loho*!Gs8%I(_AX?G{d=&}5MvUU9~y|SOjG=BG*E#Gx#?>V!b4-cB#2uz+_ zpm1DW_LAb|mz)V3CWB({`N8z=iDh}dYdapyDq8&VNf_Jm^y?pZto@dWpO>51clp(; zR|ywS{W*Fodh64#B}qyuk_$gb&D|w>B6nrh?K54wZyoN8d0lRz9qDy%@{jtS8Gg#u z+cf8aOkRIbKViY?Cjp+l(UNo8-jt;FuYRc$E;99;LcsieJn?eU`r5DG=}ipgd*!mw zvg=jJ&p&%Kl#P{>ul)-z{I#QQZ%yY6z3r!-&IGlVtmOXOjrE9DYxjC_Tj)afq1NMt z2C=2d!A5#v!W&tII=_C1J@q<8C2;=7*^4uDTG`v$>hGWXSaHnz7<=c-5^nu7?4UNH zmE0a9$~G1>A7=);cP|G-EW&-{ZD?M1De{h zsB=$~ImWAL<<$|`64uVjqoA+a&8M%er6##KRG1e<*^{qNWnvW(Pn$H%@%1K z^jgw*Ea6hiu1;w?liBau-?rywyC`)M!# zjN#$T=O51h+3|hO?>Wy`O22#cYSz4IAHMD0`u|c%)9N6mFFpTL&)EH*TlMM^v2oe5u30l#cX&Tx3rXQ!Q}_7Fl?xY)&nCHWTiF}Q|1nc>V1IG?<=xHqRvvM_ z+*NXL!RDzs7ovj_ub(}4l2_x(j$)VNx4-S0#{H^Vq|D2=Pj@YQ(>Vo(7YwzQdxJG! zd^&jfRrf-c=hCsP6R-D{PvpFjcDZC?;=T2+r!D(={i_f2t}T5R);wjjT9&uwf#l93 zyF+hHzV%IE$^%oS7ktbO=ge;}-@Fi?)5|&dyuCHv{GMxJ)NHQcNp({rdTvGEj1A4@eX`x#WaZlP=Q8qAuX%eb zH1w^Mwy3aTjoo+tZn?V3g`-PXSt;5W?fS^Fvmx#5WYa4)leO|w%R{F=op;j1_wCDz zQ?}M#UDl@?dm>v1>fF{<4<%?ThpZpDe$%Wm&}o-#yjq z&wRdSU3&Y()BC~4r%ik2Yq=ymRn_76m0rm^#ZxOz&s>nK*m%z9P_z9S$-280#anl9 z|I+KyiG37Xl{|0zy9;mip6Ul2pJsNuc=f7P=PE8iY<(K0Gjr=gmh_Vv2dBq#?9cmi zL+kw?nIG({_jE6NUYV`AGc?cLD<y6Gwr@t+IjI+HmwNBdIkWxr zGK;ldd&C^96<&@oh!+GVIcAE93;HPWfOkWpw`t{Q- zWz|1h<7~dNUob44%%Wc)kiYV5^~BV){6|8l;H^f-<+Y>(6@57#*Z_$ulDZuvk03zWlBiw+?3M075SpQ z+v9gvE}#AR*yrrS2Bn~!bJ2R4ua9)6va4e5#Ran$Et+!4dO`8*ymPbfJm6fA{3?wr zE>cyksZQXFWy$7O%U=9m#5C8o?7ftn-lr;WbtzL>$+~;@@7u?i++e-HxU19gy|rwK z+nh{C|Epi0sV-IP^YOhqV^`tAm|)SOJ(XQE7%njG+SxRJjc?@kyt~)`pIef?L+^Ng zztCN;nqBucuMyW?ZSmQ2|FUySw;dD)2j8ENQ#Wp`i}jC_sa2X_JojjF;G)@!CeOY! z+dMYS^vSNhv#h2!-n|Y=M{N(bXK>vKJ-?3q%lmz+h77L`9pEXU_aFs*B$8LT&2qS0{g&Sz1;u_Afns z?)t0#xK*d`-~9VSNolIcB}mU^yBPe3-nvduU)QJ6I%V}|J2KCmd_4Z z8MbEH@tyN$1g$+99l!GQ{g)=mF;}j{?k`yPHm{nR(mYjf1dLFae(Q|(huu? zCha~{eQ(k6YnOW`pOw@3q}TN)UiSUbZ(gxGEf-9VmT-vQ_*B;_Z`SVC|0dtkL7G%Q zocmGLe5g7}tEwdV{MjPo5Z2k1otD!+zq`;-x77U2g4^|<=lwXkb#3v5p{DicuAFYeZ#=PYY^C1;Vj_55?Mc6;xBUH`lL z)JjYJJ;B>+f1Eat(QDuEYF%u^&Tp4RuWx+KcVT+k(pi(1vFtTGRkHc!j`BtMrW@=3 zw*UL{xvcg=X_T2>)~nS?oqM${l4C3S_%2L8SG{SeS@1*K`soFyUq3r>)8ox%tzDYe zEp9vPjQ{JzyUXviTgtJ-SI79{B15g0vHbme^zc)JDm}%;;U`>x6b)*-aWs2_ikPFmeh(dj@BL}s<=bxC z6|yd1E(vCR@A>wn=UI>SduAN5yMLiq!n^iz+5z8c?v*c>@G+J6+&yuya`nQ+B43nK zFZk}6^8NC|P0Mz!vh9y#a$tY4jO$!9@7?9Dk6x8In?HT@$Kz&^uj~TmFREXkt!+CO zbfNsXqj^@&?pXD!q6?TyjM>sfLt8ehpUl1~_`>aN8^^9qJEcHD%6IQj6mRLJlpTB0 zUZ^$crJZ`Sp7{mCT_KRo-X+zU6(YL=leaF?JHlAfkk|Tv?_SmmiL+H-%bd-#*8F;W zCx~qUa|t)g{3mPM_%3SyJ|a>Q+!R+Qxq!Juo8|dm?ws})1`!;0SD2pT$~n&jR=PfA zZQGmUFAQ#T++DGekN5Cm2H6GCiWd2fQwloyetx~sE47+yWw96U0_GB7w&!PE?kc_b z$MQn$m12{eOTHGQqU3z-P_$|C9;-43^Rjm3ZQi_#%talJU*VOpF>h|Hn-+h3fwWeSrDr1j+IPIPi{0{9 z_66Go#$8XFJ};WE$n^Q#u6D2{y&nO)KJC6-yPOZ=Bk?)cyM$XNXJ%#|?_d6=HE-gV zx|-V4?r%=tKXz`W@FC_e1r?v83fC^?KPNi>$FtKS<|W-rzMq{f@3o-*MbXY20l5Y3 zhpi`G?V7Fe{>}T>r;DApd)(i3f2w2q2L{`iduv*z7~K1@Kx&sH_m}M1>+Fw-$}VU> zy4zVtQT2?OoNrTJ8`IsLU%ws9wr}~sU@LRa^5&x(_p1Kelo*S=__FFB`@ePE+oqm7 zdgIX^WvM$yg5%pBh%!H#@jvu=_=Vh)dz@MSetiE?UM&2JLB(yhvaa|iyRGkD(EW6; z?E2yUn6kvHvAYXCe3Ong-^=}wvC3jkUEb6zO}!} z2~Iy*AB=gGUi?f-E&7)N7zq__f)xNHJzRmpGv1Q*YkEE}>?QUCa*eLDN{^E>C{GNziS7$$; z+0A)J!OA6l*$?*POQM(G`{HwvDNjIlk#X>yzjdnwE@?&`;9cZglGerjS!MU*igoGF z*b16o2+W=NXsh&+=;qkJv%&Juy1Lh|?YuZ6_$EJCcle$yF1t!sXgrJstKRwO(QTci zH`d{Mxwrx zWoDjd!Vh-&8qxT3&pp$zc^v^{1(`Pwl4j6-TPwqqO`3b%1T`3 ztL^&W662@5!?C2}+=drvzpj7V=LmM{!be-BpWF_WOqp3N1NKT#Nw8XH(XM&b++gmC z9T#E2g+@O5#WnfwFAUuRsFuU{*Bp4Gf{ z?ymU4?tkoYpKf|H$Sq=@dRIzyoynVr;?1=WmW5oL9`kI6fAT%%iVi#Fi~SR#pZ8jV zg!R*k@2r^-`6!>)BF+80O4>Bpi;uMKeJv1?KPMJ`>g1#F|BTzkC-y2^mHlNEm*1zm ztTl=BXbn5B~v zd4&xm6)Cd!K$h%=n`@Fway;~l*MSq|D*biQ*)QW=oA2#-Da8UZ(WfhKQcl&(VqQ?H z2(K;t>-c#>%H`7`KA?Es^{TtRBw&VmWY3;BaI`IY5w`NhV)FyMiy|tu$`btP{xGLVCB|EvRZYwAw9Wd6cnwj!BgL{W#Nke+zm$aEL;v1x0 z*ekwuO`mcpao*jHY3fj^?4XPODor3cHCKC`n>EhdAkW;?-dAlucZVa-S!r-8@w{lQ zvMAdhY=)Yvz0yt{{}^VFLP?!}|BS3eCrel>TDh=Sn7OW>_UG@~ry5|>gT55aO<(;_ z4$Rr`gEe{QhD+6}wr~G_Ar$PpB`>aSk+r#hed^@L$vGaNgd2Nw(RHpxzh{&{f~T_Z zY}+m6lB36$WP^j}R(Jl7F2Ct-vh~!!s^1{_(m6CAfZ-FeI2a_T}%E8)(p>HW-;CwQjr z{gdxvcSOVL#Sgoi9B}@!z~#TCsO?Xd9hyeJ53EnWTcaB-@OMev`s)lfF8pf0m)1*r zJkI_RUeF!0tC9SL$8=F#e!!(arsxGfx-l#|yda-|^&9tjtBlUp_AD z_jBcWovE{RJ6*WBUgyuEH~ibdw$;h%vTv986SANl_%eSlN^6u+4r}`4$Z$(bC2A$uc{NXzUP^gB=q27 zBD28W876xxA1piZhI#*7>9eQpo_9^V{qNzOH>&m@|D3p}nadz2du{s*&Lumia_4oO z5qrPb{rHQW{o;E*E&ll~vsA0v`elS;(1P zx{{%vPkO}*Q0bs`(RSiAwhu2hFm~+gG|7E_z<24Q<<}T(tk&$$d3(zJJNN8QpFeXQ z1c_QO%E?~K|2Dn-(WUNz_UQ<&O6*X%!`-6*#^QW%;{pq1uLGbn$?`CwU zyMGg{YH)9_wVLNp66R8V_3G8HUo7S_^z;1=$(UzzhQ-{>Mf}k&*#{RB_sU-MeQ<2j zi(7&X?(I`uk6pMh;pP7O4kgQ6o~y+$^z&)=u6oq+sIvdx6rMbvPP@N&LBq zy5oqtexr2sqW^Ka&Ci{N2vmRzL0zv7c% z-NjESiaR!{R=xUBDz(nwU2Up;uE|q@*q=wgmwf21>Nq#+#X70w#dG*~tJPG;>+Dkq zoXU7nH2%*A)4GlC7+unRO3H7nZ_j@}`N;A$XPgCMKOQw+_2cMVcJG6|z88;&x8%Rq zRr@(nc9Hguib&)AI~XrUirVf|s;Rc$;F@t=+_~haczo((@x20b123B1J@?`5--wwP z#3F<}Rh&y$#q~FQ>W=0YYm}bt{>5|mJU5GP|7QJg-;uIEe zwd+{q;r7GxWZU+>zw!9%B-usU8+V*Mq|c^U^H(wTdy$oF-xk!6^h1E>At-Oocb)sd; zD_;p-c`aPF`|iUPooil97GC)OO|{gM+nurku{)2-+pk+|`tSAo-3N7DZa%u>5iG3L zVZQ0FZ??>{3%>QarY!U4+BGTiHs>BNyB2dbknN&qc+KI*UjLem)^Xl(ni_jq)4qkP z?R8d|%eM*tX5O`PTx4A`zxL{lFU3Doq;B&F*vcF=UX`q-uE;4Wh<)Euie(HZNL6y z`!wC#@waY8?W&)7F=F=AtuZ}o)nnN#p z_|g!d%I%hpNv{tb@BXLZLz+Soo39O-IkUU-t2v+nEGOM zn`^F5x19FA8;RoQ=E+W8s9|Tiph#zL?XwjeI=4*)BQIJe+Hn0&5x5yF{?s#UbLoMy zv*!gKvQ5_cyTGofHtluL)$V)d9-(XH-@ZDn8aQpk%bn*^d7k_a^INRta{bPee%BO> z)_KoQ7vI0UPqXCHjgJ9ueqFISD)n&tE2cAL*CxD|s`hxdZR#7oviFxC&HueQySVPZ zFCVAw^cc6p{+GmND|Wgt`CgarS&+TI#{27{=U=({CAP0wJn{F*;Bxc5Y5ui)qfRVf zjBS2pvFLw*jC|&=uN&sS@SG|x&v|rR`E{1*YnRVgFje8da7T{sLtN7Ny9^8hTS4an z*vITJ&3--e>DGCZpL@E?_7?gcsIJf2a;d1je_ezBf1|LA8^k7jS#gbH*S4ZUmhRVf zH@uTSKRXMW0&1AG;}gTag9=s07_{0NBBlhd7UEQpd*$e8*W|LL&R0uNb+jCz2kW|&X${T zJU;vHNXq8?|G)43v%mNI{q3`9te3gAh14Z5eqEd3QpdBg%Jjtj_*?rp#eyg}+>_qaCmBV*Be*7BUW_XQj%i)Nhw=W-lQL#25 zwDQ8Nm@2NR-z*L7^z`(!^3vJ@FMil<&$#aY^TeNbb{ZNQ9{urA%e4bAnB8+v`6!)VAHbXpz$1 z1Ao#Ka*tKsVP|vY?bUn`bK;V#y@u@MyVt`Rce2@}=T*L_$d*1CUv7Nu=Fexf|2`a@ zwfm0VN5ReC*D^fbc&9n`Y0rtWEs9$g%Kwb-47Q%QXG>FB>*4i+q3#bQCvBGgyt~=a zn5mAFhnq*os^rCmna1f(U3OpGfBs*P9jof1+*W6BGsDD???u`r{%noyt|v2Rzy)rs6wzg246F#K*WN$@(s^Lesw_43ER_NJ|6;F~>9&LH4RtCW&i znd2gsIsMb8x3|@FM(>ZW_|LX#&7y5A0e|*Ay=Ivu;J?4WtLxXh>zaqfU&tB15%RxY z#ZuLv9KUT|@0RKJDi4d9eV@IaQ;93^`mb-(9A=4n@wK#Vb6mCP+C}z1rjDnl8Ld0? zUcaZaN5^`@fwHx$6Ism6=9z7b^%g%9kZ6#YyxBO4x5#;`!2F9+HO9`Y^NVlYxs#Ka zcKQmlkoU)Xp-N`|mWkYYxz=Xh-@?GCr{p4vTOeQ`C+s3l8oQiru>zg;Ly*s zQocp(y;P|3FVV27<>f`iwY9Z@f#2>IzP?|uysopD+ehgMf4v@ivsdKJw999M%@tV9 z*_vmRPtRvjJ}>g&{!L4>YprfN67SDWtvTzIw&00c%+HJKr=`TKQafw&&c0Ik{*1e3 zEfYDf4I%b zZ-}ijyZ7@`&c0<)^=mjj)t#)8IV8Q|_tJJlbM6VRuD`$WrB_C4=hABrmP5!}5lKA*kBuFp*(a~{L@_widAY_G-n@AN4A@W@f>!Qz{f z>}IF%ottAkIfd^@^^c;jicx9PF3g;=%dG5__wDPcDr#v@pBu#h!m^DLg(f7A&g8t43>2`4GGTG*9_Bw5w zY@uqY+P2RjnI;FO`}A%!6`Pk^C$-i0E<=-SP*&&)v#vJv3*~!W8b1u)TPpr-J&TLG zkoR4~f+rKbmR?y|y=(QwU+y`Kw;K-bZK~VUV!HNY$TAkqljqiJeohDuzMTEZ`rnzJ zmNjdnE^|f+$j%LRyWQd=P_b`;@w&BB8IF8#WwPticrfXwP%p2MlHjZjvi9c7c=);( zIPP5#In!*8xc{9mMRz`ia*OZYvDR+Z?1|#>JMw-X`+Y7uSp96zr};MbGQ`q;?XzC; zRrtSIc>GEJW6$|hjM^^CSnm-&zw705b{4aya@|x`iFYUWf4zI|hm?w*p5B$Se3g%H zvhMoFU(g-m<)T)$9c z27l<(stTss+tqxfpTEfSlrH2965(>J2Kt6}hU-6E*)?xoU*D4_5;tZt>?@h8`*l%c zmH(Q+oguGOTbcKrm@Fq-B64x}45KBYm;6qp+%=MXb6}zG;bOJtv)+ehgfUcJYvcIW z>$#d?V`#TxXlqSbRQPLu$>~hn8S1@iTzJ)lZd+Xu>{iv9pw#qM=S*|vk!>?&wmK*U znpYj-332wjTe|he3&V;90vxuI*9wj74x1loe8g==g;%~JS;XCRXz2Z z>9ee&=i`Qe|1UTiR=?)DqtD{7XU{6@hnh1Y6a>1D@&<^ne`YJnS1~f}eHeqTu7sSqtU2`YxPsu)ygeixI;b zbD<44c)jxMkEpiyr=~v_*ABDVt*Tz0bKZ99#OWN5?e4w(v^JVwE=Tq!%aK{yoYz)} z=&fr%tn{gRBG*q<8TSdZe7So4(X(D&$eRIq4!xhIV6s$1lDT`K8A${#lW^_N4dG4yiSPzRagJ>unBnO9kGZ za&}3r${ERhihN)C-Fp=~R!$Y$rgd+Ru zM*e>P2daC;3Y6k}lroQfYg+Fl^vIluh>9_8e8SnRfJ(~pD$ z{_i~By?EhRC_g`9MxmB?@As326EAMc+myR^?~X#ty%XGDY?c1Q%wj9s!MNt)t*9&Z zJJ?tY`@M+BLyVYXR4itkBSc%miDB##;vzT0G1aG&k-1 zWxic{J40>l2UfkiC!_;7FR+F*F)V(~V8V1M^Wtow`S*6Z{b<~g@2jDw*LIHYo>0P_ zfCm0&-n()q{}enTaFvPCOqz3VZpz{8ms_q&HC;1I;rV{eck%a&i`{c`ZOh-?(PSx< zpC8UKt?E|Ywd-a-7p`5q_RlGX>FY1N{QXS&!}7=XY_BX(f7aI?cqwb&RWUD(y>qo| z)iNb6m$Q8@n|*Lseq_d$+`~qT^vxeAb4_MVIobR4^lgtSjaw2DZ{Dt&tU50xHC0tr zb?dgK=}Xq=#4k@=(08xt!J32JJZb0 zuXk@(&Cf1P?HL>Hq$Y2PDz<5k`*fu9a30%lnYp4FN;fZFES#YCR{Bj<$3tHCrRTT3 zwUO)Rmzc*~-FNk8&E~g%B$ik1n+sr+L%b zdmfqFsupW)Ut_*o@-M%7``XKW0(`->rB|=_{$XMLHP1_({r*LrrN@0{pR%{k-FJ5C zd&}1kexF~qdtJ(j}hVJLy zC$2URnziEd@8|N| zJ3WVgWIxoXdfVJ|F-U_$XLrhlunURrmc{HAxqPvuxw&bfxAlz)K~@z;Q)Ki+ym-Q$ zHs2_iZ98Z7Lhnshrm9lE{)eoOyBQdCbDCVNcl&eWBZnRy{{B_D{_A7kKUP}zyI#e* ztbG+Hwjug%(T~-93*7%-FfTsbT(X<}7MHSMdR}c&QHXP{#r6|=6QX~v2~;po6+hwo zIwSAYvEpE}Ink+4SB9|`UpAX_JL>$j=)j&A4l7QVzwupNb?$=wy2WLgk^0ro*%owP z`VsYo?V(#T$E}MR=e#otFG@`@X4!wVt++eKV*Ad!Uu-SRa}V!4YBgi_wu^VR@IJq0 zz<)T-8Dwtsn)K6aKji)jjn6rCyg6)bR@8d`^nWs2T3<%{3Twu#>Rq~HU$FWPyNuPz zOPj^@?^Y(&f4H4vVQyPuVxC#?mvzqUZ4d8k>3g;;*JOLei-f$-)(fk6&**Nucqhp= zIwFCe0%uWo|=qmhmVt|PW;HR`zl}m+|~CNhOg^x4_{c_d_U{( zPFJ&e(Gith&&6(59_K@7soco?Ywrr#(IxTPJ`JjkPEN;zS{`$?(AzI|L?!A zBz@-XrtI6#r`aAZioTtYmu>5v=c+l^clE{ct=@;DJZxP9Ce2Nj?0flb?w!v+x678D zo;&^Drn7gBzW&zB^EIeiqk4C`WBHA%2Lrq!ow~z$5ATp#boAhzB##o2#>ZEAO*q?~JYElS7l&RRLKth zdcOH{)IPm65!)+URx5CK2 zSrqX3@5CqPlvT7l?dpF|t$GwtyY{G2j+Xn2{Cq*zGLbd-sHBS-hP6$}JM(s{zi$d=|H(Np76L-R=pM3au;=VL_)oG{x zDqegse;J4Qwwfz1?yE`ty{;=d?c~vl{XQnYo@=W1EKqjIKRValBdo4w)5F8|i#IQq z{+Dqi{d{Aba`}{q@L$uzrT6_*c3ly@E}OM((Ghk1cR!Q7TNgk1`Ezr+wB0YgTpv5> zxf7CPES5z?ysF!={9U`hzu48jtEcrm4jZ&3E4N=*E%EihX+t$bNolw06Sb}!e%Tl1 znLUzxw{ybfv<=ml{!UzQ!}#^T6^C|YuRUXvEL3!*^!J<|Wq2f_pA`s~___Uuw9<*3 WxDU$~IxsLWFnGH9xvX [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. -Contains details of GitLab [Database Load Balancing](database_load_balancing.md). +Contains details of GitLab [Database Load Balancing](postgresql/database_load_balancing.md). Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/database_load_balancing.log` diff --git a/doc/administration/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md similarity index 51% rename from doc/administration/database_load_balancing.md rename to doc/administration/postgresql/database_load_balancing.md index 45f27a8a8f2ecf..d04e4de1cf32f4 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -8,150 +8,128 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60894) from GitLab Premium to GitLab Free in 14.0. - -Distribute read-only queries among multiple database servers. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334494) for Sidekiq in GitLab 14.1. ## Overview -Database load balancing improves the distribution of database workloads across -multiple computing resources. Load balancing aims to optimize resource use, -maximize throughput, minimize response time, and avoid overload of any single -resource. Using multiple components with load balancing instead of a single -component may increase reliability and availability through redundancy. -[_Wikipedia article_](https://en.wikipedia.org/wiki/Load_balancing_(computing)) - -When database load balancing is enabled in GitLab, the load is balanced using -a simple round-robin algorithm, without any external dependencies such as Redis. - -In the following image, you can see the load is balanced rather evenly among -all the secondaries (`db4`, `db5`, `db6`). Because `SELECT` queries are not -sent to the primary (unless necessary), the primary (`db3`) hardly has any load. - -![DB load balancing graph](img/db_load_balancing_postgres_stats.png) - -## Requirements +With Database Load Balancing, read-only queries can be distributed across +multiple PostgreSQL nodes to increase performance. -For load balancing to work, you need at least PostgreSQL 11 or newer, -[**MySQL is not supported**](../install/requirements.md#database). You also need to make sure that you have -at least 1 secondary in [hot standby](https://www.postgresql.org/docs/11/hot-standby.html) mode. +This functionality is provided natively in GitLab Rails and Sidekiq where +it can be configured to balance the load round robin, without any external dependencies. -Load balancing also requires that the configured hosts **always** point to the -primary, even after a database failover. Furthermore, the additional hosts to -balance load among must **always** point to secondary databases. This means that -you should put a load balancer in front of every database, and have GitLab connect -to those load balancers. +```plantuml +@startuml +card "**Internal Load Balancer**" as ilb #9370DB +skinparam linetype ortho -For example, say you have a primary (`db1.gitlab.com`) and two secondaries, -`db2.gitlab.com` and `db3.gitlab.com`. For this setup, you need to have 3 -load balancers, one for every host. For example: +together { + collections "**GitLab Rails** x3" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} -- `primary.gitlab.com` forwards to `db1.gitlab.com` -- `secondary1.gitlab.com` forwards to `db2.gitlab.com` -- `secondary2.gitlab.com` forwards to `db3.gitlab.com` +collections "**Consul** x3" as consul #e76a9b -Now let's say that a failover happens and db2 becomes the new primary. This -means forwarding should now happen as follows: +card "Database" as database { + collections "**PGBouncer x3**\n//Consul//" as pgbouncer #4EA7FF + + card "**PostgreSQL** //Primary//\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// **x2**\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_secondary #4EA7FF -- `primary.gitlab.com` forwards to `db2.gitlab.com` -- `secondary1.gitlab.com` forwards to `db1.gitlab.com` -- `secondary2.gitlab.com` forwards to `db3.gitlab.com` + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]r-> postgres_secondary +} -GitLab does not take care of this for you, so you need to do so yourself. +gitlab -[#32CD32]-> ilb +gitlab -[hidden]-> pgbouncer +gitlab .[#32CD32,norank]-> postgres_primary +gitlab .[#32CD32,norank]-> postgres_secondary -Finally, load balancing requires that GitLab can connect to all hosts using the -same credentials and port as configured in the -[Enabling load balancing](#enabling-load-balancing) section. Using -different ports or credentials for different hosts is not supported. +sidekiq -[#ff8dd1]-> ilb +sidekiq -[hidden]-> pgbouncer +sidekiq .[#ff8dd1,norank]-> postgres_primary +sidekiq .[#ff8dd1,norank]-> postgres_secondary -## Use cases +ilb -[#9370DB]-> pgbouncer -- For GitLab instances with thousands of users and high traffic, you can use - database load balancing to reduce the load on the primary database and - increase responsiveness, thus resulting in faster page load inside GitLab. +consul -[#e76a9b]r-> pgbouncer +consul .[#e76a9b,norank]r-> postgres_primary +consul .[#e76a9b,norank]r-> postgres_secondary +@enduml +``` -## Enabling load balancing +### Requirements -For the environment in which you want to use load balancing, you'll need to add -the following. This balances the load between `host1.example.com` and -`host2.example.com`. +Note the following requirements for Database Load Balancing: -**In Omnibus installations:** +- A HA Postgres setup that has 1 or more secondary nodes replicating the primary. +- Each Postgres node can be connected to with the same credentials and on the same port. -1. Edit `/etc/gitlab/gitlab.rb` and add the following line: +When using Omnibus GitLab the additional requirements apply when [configuring a multi-node setup](replication_and_failover.md): - ```ruby - gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } - ``` +- PgBouncer configured on each PostgreSQL node to pooling all load balanced connections. -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +## Configuring Database Load Balancing ---- +Database Load Balancing can be configured in one of two ways: -**In installations from source:** +- Hosts - A list of PostgreSQL hosts. (recommended) +- Service Discovery - A DNS record that returns a list of PostgreSQL hosts. -1. Edit `/home/git/gitlab/config/database.yml` and add or amend the following lines: +Refer below to each section for more details. - ```yaml - production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - hosts: - - host1.example.com - - host2.example.com - ``` +### Hosts -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +Configuring with a list of hosts is straight forward. For the environment in which you want to use +load balancing, you add the `gitlab_rails['db_load_balancing']` setting into the `gitlab.rb` file in +the GitLab Rails / Sidekiq nodes. -### Load balancing for Sidekiq +For example, on an environment that has PostgreSQL running on the hosts `host1.example.com`, +`host2.example.com` and `host3.example.com` and reachable on the same port configured with +`gitlab_rails['db_port']`: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334494) in GitLab 14.1, load balancing for Sidekick is enabled by default. + +1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following line: -Sidekiq jobs mostly write to the primary database, but there are read-only jobs that can benefit -from the use of Sidekiq load balancing. -These jobs can use load balancing and database replicas to read the application state. -This allows to offload the primary database. +```ruby +gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] } +``` -For Sidekiq, we can define -[data consistency](../development/sidekiq_style_guide.md#job-data-consistency-strategies) -requirements for a specific job. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + -## Service Discovery **(PREMIUM SELF)** +### Service Discovery > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in GitLab 11.0. -Service discovery allows GitLab to automatically retrieve a list of secondary -databases to use, instead of having to manually specify these in the -`database.yml` configuration file. Service discovery works by periodically +Service discovery allows GitLab to automatically retrieve a list of PostgreSQL +hosts to use. Service discovery works by periodically checking a DNS A record, using the IPs returned by this record as the addresses for the secondaries. For service discovery to work, all you need is a DNS server and an A record containing the IP addresses of your secondaries. -To use service discovery you need to change your `database.yml` configuration -file so it looks like the following: - -```yaml -production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - discover: - nameserver: localhost - record: secondary.postgresql.service.consul - record_type: A - port: 8600 - interval: 60 - disconnect_timeout: 120 + +When using Omnibus GitLab the provided [Consul](../consul.md) service works as +a DNS server and will return PostgreSQL addresses via the `postgresql-ha.service.consul` +record. An example of how you would configure this along with details for each +setting after is as follows: + +1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following line: + +```ruby +gitlab_rails['db_load_balancing'] = { 'discover' => { + 'nameserver' => 'localhost' + 'record' => 'postgresql-ha.service.consul' + 'record_type' => 'A' + 'port' => '8600' + 'interval' => '60' + 'disconnect_timeout' => '120' + } +} ``` + -Here, the `discover:` section specifies the configuration details to use for -service discovery. - -### Configuration - -The following options can be set: +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. | Option | Description | Default | |----------------------|---------------------------------------------------------------------------------------------------|-----------| @@ -179,45 +157,40 @@ When the list of hosts is updated, it might take a while for the old connections to be terminated. The `disconnect_timeout` setting can be used to enforce an upper limit on the time it takes to terminate all old database connections. -Some nameservers (like [Consul](https://www.consul.io/docs/discovery/dns#udp-based-dns-queries)) can return a truncated list of hosts when -queried over UDP. To overcome this issue, you can use TCP for querying by setting -`use_tcp` to `true`. +### Handling Stale Reads **(PREMIUM SELF)** -## Balancing queries - -Read-only `SELECT` queries balance among all the secondary hosts. -Everything else (including transactions) executes on the primary. -Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. - -## Prepared statements - -Prepared statements don't work well with load balancing and are disabled -automatically when load balancing is enabled. This should have no impact on -response timings. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. -## Primary sticking +To prevent reading from an outdated secondary the load balancer checks if it +is in sync with the primary. If the data is determined to be recent enough the +secondary is used, otherwise it is ignored. To reduce the overhead of +these checks we only perform these checks at certain intervals. -After a write has been performed, GitLab sticks to using the primary for a -certain period of time, scoped to the user that performed the write. GitLab -reverts back to using secondaries when they have either caught up, or after 30 -seconds. +There are three configuration options that influence this behavior: -## Failover handling +| Option | Description | Default | +|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| +| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | +| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | +| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | -In the event of a failover or an unresponsive database, the load balancer -tries to use the next available host. If no secondaries are available the -operation is performed on the primary instead. +The defaults should be sufficient for most users. -If a connection error occurs while writing data, the -operation is retried up to 3 times using an exponential back-off. +An example of configuring these options with a hosts list would be: -When using load balancing, you should be able to safely restart a database server -without it immediately leading to errors being presented to the users. +```ruby +gitlab_rails['db_load_balancing'] = { + 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] + 'max_replication_difference' => 16777216 # 16 MB + 'max_replication_lag_time' => 30 + 'replica_check_interval' => 30 +} +``` ## Logging The load balancer logs various events in -[`database_load_balancing.log`](logs.md#database_load_balancinglog), such as +[`database_load_balancing.log`](../logs.md#database_load_balancinglog), such as - When a host is marked as offline - When a host comes back online @@ -237,36 +210,35 @@ For example: {"severity":"INFO","time":"2019-09-02T12:12:01.728Z","correlation_id":"abcdefg","event":"host_online","message":"Host came back online","db_host":"111.222.333.444","db_port":null,"tag":"rails.database_load_balancing","environment":"production","hostname":"web-example-1","fqdn":"gitlab.example.com","path":null,"params":null} ``` -## Handling Stale Reads **(PREMIUM SELF)** +## Implementation Details -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. +### Balancing queries -To prevent reading from an outdated secondary the load balancer checks if it -is in sync with the primary. If the data is determined to be recent enough the -secondary is used, otherwise it is ignored. To reduce the overhead of -these checks we only perform these checks at certain intervals. +Read-only `SELECT` queries balance among all the given hosts. +Everything else (including transactions) executes on the primary. +Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. -There are three configuration options that influence this behavior: +### Prepared statements -| Option | Description | Default | -|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| -| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | -| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | -| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | +Prepared statements don't work well with load balancing and are disabled +automatically when load balancing is enabled. This should have no impact on +response timings. -The defaults should be sufficient for most users. Should you want to change them -you can specify them in `config/database.yml` like so: - -```yaml -production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - hosts: - - host1.example.com - - host2.example.com - max_replication_difference: 16777216 # 16 MB - max_replication_lag_time: 30 - replica_check_interval: 30 -``` +### Primary sticking + +After a write has been performed, GitLab sticks to using the primary for a +certain period of time, scoped to the user that performed the write. GitLab +reverts back to using secondaries when they have either caught up, or after 30 +seconds. + +### Failover handling + +In the event of a failover or an unresponsive database, the load balancer +tries to use the next available host. If no secondaries are available the +operation is performed on the primary instead. + +If a connection error occurs while writing data, the +operation is retried up to 3 times using an exponential back-off. + +When using load balancing, you should be able to safely restart a database server +without it immediately leading to errors being presented to the users. diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 10e6d717a18144..424173e128a7d7 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -29,7 +29,7 @@ project.feature_available?(:feature_symbol) ## Restricting global features (instance) However, for features such as [Geo](../administration/geo/index.md) and -[Load balancing](../administration/database_load_balancing.md), which cannot be restricted +[Load balancing](../administration/postgresql/database_load_balancing.md), which cannot be restricted to only a subset of projects or namespaces, the check is made directly in the instance license. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index cbf3c09b28b336..9481de931fa10f 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -160,10 +160,10 @@ query. This in turn makes it much harder for this code to overload a database. ## Use read replicas when possible -In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. +In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/postgresql/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. By default, queries use read-only replicas, but due to -[primary sticking](../administration/database_load_balancing.md#primary-sticking), GitLab uses the +[primary sticking](../administration/postgresql/database_load_balancing.md#primary-sticking), GitLab uses the primary for some time and reverts to secondaries after they have either caught up or after 30 seconds. Doing this can lead to a considerable amount of unnecessary load on the primary. To prevent switching to the primary [merge request 56849](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56849) introduced the diff --git a/doc/development/scalability.md b/doc/development/scalability.md index fdae66b7abc800..7a3f3c7097d5a8 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -123,8 +123,7 @@ the read replicas. [Omnibus ships with Patroni](../administration/postgresql/rep #### Load-balancing -GitLab EE has [application support for load balancing using read -replicas](../administration/database_load_balancing.md). This load balancer does +GitLab EE has [application support for load balancing using read replicas](../administration/postgresql/database_load_balancing.md). This load balancer does some actions that aren't traditionally available in standard load balancers. For example, the application considers a replica only if its replication lag is low (for example, WAL data behind by less than 100 MB). diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index e28a328888d1d7..4d1f0daf66bcb4 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -574,7 +574,7 @@ of reading a stale record is non-zero due to replicas potentially lagging behind When the number of jobs that rely on the database increases, ensuring immediate data consistency can put unsustainable load on the primary database server. We therefore added the ability to use -[database load balancing for Sidekiq workers](../administration/database_load_balancing.md#load-balancing-for-sidekiq). +[database load balancing for Sidekiq workers](../administration/postgresql/database_load_balancing.md). By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas under several strategies outlined below. -- GitLab From c9342d4c187943fe7c0f519bcc1da55d98114c63 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 16 Nov 2021 10:56:58 +0000 Subject: [PATCH 2/7] Fix broken link --- doc/development/merge_request_performance_guidelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 9481de931fa10f..74b1d5680b0966 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -187,7 +187,7 @@ Internally, our database load balancer classifies the queries based on their mai - Sidekiq background jobs After the above queries are executed, GitLab -[sticks to the primary](../administration/database_load_balancing.md#primary-sticking). +[sticks to the primary](../administration/postgresql/database_load_balancing.md#primary-sticking). To make the inside queries prefer using the replicas, [merge request 59086](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59086) introduced `fallback_to_replicas_for_ambiguous_queries`. This MR is also an example of how we redirected a -- GitLab From 805b9e8484442a31bd4f188e29507b430a235961 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 16 Nov 2021 13:50:42 +0000 Subject: [PATCH 3/7] Fix broken link --- .../monitoring/performance/performance_bar.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index d798feb71a9f70..3420935e462c29 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -26,11 +26,10 @@ From left to right, the performance bar displays: details for each query: - **In a transaction**: shows up below the query if it was executed in the context of a transaction - - **Role**: shows up when [database load - balancing](../../database_load_balancing.md) is enabled. It shows - which server role was used for the query. "Primary" means that the query - was sent to the read/write primary server. "Replica" means it was sent - to a read-only replica. + - **Role**: shows up when [database load balancing](../../postgresql/database_load_balancing.md) + is enabled. It shows which server role was used for the query. + "Primary" means that the query was sent to the read/write primary server. + "Replica" means it was sent to a read-only replica. - **Config name**: shows up only when the `GITLAB_MULTIPLE_DATABASE_METRICS` environment variable is set. This is used to distinguish between different databases configured for different -- GitLab From be98073ea2296571a503e0fa602a9bd9129ba1a6 Mon Sep 17 00:00:00 2001 From: Marcia Ramos Date: Tue, 16 Nov 2021 16:15:44 +0000 Subject: [PATCH 4/7] Apply 16 suggestion(s) to 5 file(s) --- doc/administration/index.md | 2 +- .../monitoring/performance/performance_bar.md | 2 +- .../postgresql/database_load_balancing.md | 30 ++++++++----------- .../licensed_feature_availability.md | 2 +- doc/development/sidekiq_style_guide.md | 2 +- 5 files changed, 17 insertions(+), 21 deletions(-) diff --git a/doc/administration/index.md b/doc/administration/index.md index 17938d820c8930..ef9c56c1615dbf 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -77,7 +77,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Enabling and disabling features flags](feature_flags.md): how to enable and disable GitLab features deployed behind feature flags. - [Application settings cache expiry interval](application_settings_cache.md) -- [Database load balancing](postgresql/database_load_balancing.md): Distribute database queries among multiple database servers. +- [Database Load Balancing](postgresql/database_load_balancing.md): Distribute database queries among multiple database servers. - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). #### Customizing GitLab appearance diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 3420935e462c29..14a560223f9911 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -26,7 +26,7 @@ From left to right, the performance bar displays: details for each query: - **In a transaction**: shows up below the query if it was executed in the context of a transaction - - **Role**: shows up when [database load balancing](../../postgresql/database_load_balancing.md) + - **Role**: shows up when [Database Load Balancing](../../postgresql/database_load_balancing.md) is enabled. It shows which server role was used for the query. "Primary" means that the query was sent to the read/write primary server. "Replica" means it was sent to a read-only replica. diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index d04e4de1cf32f4..9de4d0754ef1c5 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -15,7 +15,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w With Database Load Balancing, read-only queries can be distributed across multiple PostgreSQL nodes to increase performance. -This functionality is provided natively in GitLab Rails and Sidekiq where +We provide this functionality natively in GitLab Rails and Sidekiq where you can configure it to it can be configured to balance the load round robin, without any external dependencies. ```plantuml @@ -58,31 +58,27 @@ consul .[#e76a9b,norank]r-> postgres_secondary @enduml ``` -### Requirements +### Requirements to enable Database Load Balancing Note the following requirements for Database Load Balancing: -- A HA Postgres setup that has 1 or more secondary nodes replicating the primary. -- Each Postgres node can be connected to with the same credentials and on the same port. +- An HA Postgres setup that has one or more secondary nodes replicating the primary. +- Each Postgres node connected with the same credentials and on the same port. -When using Omnibus GitLab the additional requirements apply when [configuring a multi-node setup](replication_and_failover.md): - -- PgBouncer configured on each PostgreSQL node to pooling all load balanced connections. +For Omnibus GitLab, you also need PgBouncer configured on each PostgreSQL node to pool +all load-balanced connections when [configuring a multi-node setup](replication_and_failover.md). ## Configuring Database Load Balancing Database Load Balancing can be configured in one of two ways: -- Hosts - A list of PostgreSQL hosts. (recommended) -- Service Discovery - A DNS record that returns a list of PostgreSQL hosts. - -Refer below to each section for more details. +- (Recommended) [Hosts](#hosts): a list of PostgreSQL hosts. +- [Service Discovery](#service-discovery): a DNS record that returns a list of PostgreSQL hosts. ### Hosts -Configuring with a list of hosts is straight forward. For the environment in which you want to use -load balancing, you add the `gitlab_rails['db_load_balancing']` setting into the `gitlab.rb` file in -the GitLab Rails / Sidekiq nodes. +To configure a list of hosts, add the `gitlab_rails['db_load_balancing']` setting into the +`gitlab.rb` file in the GitLab Rails / Sidekiq nodes for each environment you want to balance. For example, on an environment that has PostgreSQL running on the hosts `host1.example.com`, `host2.example.com` and `host3.example.com` and reachable on the same port configured with @@ -103,8 +99,8 @@ gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.ex > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in GitLab 11.0. Service discovery allows GitLab to automatically retrieve a list of PostgreSQL -hosts to use. Service discovery works by periodically -checking a DNS A record, using the IPs returned by this record as the addresses +hosts to use. It periodically +checks a DNS A record, using the IPs returned by this record as the addresses for the secondaries. For service discovery to work, all you need is a DNS server and an A record containing the IP addresses of your secondaries. @@ -114,7 +110,7 @@ a DNS server and will return PostgreSQL addresses via the `postgresql-ha.service record. An example of how you would configure this along with details for each setting after is as follows: -1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following line: +1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby gitlab_rails['db_load_balancing'] = { 'discover' => { diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 424173e128a7d7..629d0027ffe2d0 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -29,7 +29,7 @@ project.feature_available?(:feature_symbol) ## Restricting global features (instance) However, for features such as [Geo](../administration/geo/index.md) and -[Load balancing](../administration/postgresql/database_load_balancing.md), which cannot be restricted +[Database Load Balancing](../administration/postgresql/database_load_balancing.md), which cannot be restricted to only a subset of projects or namespaces, the check is made directly in the instance license. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 4d1f0daf66bcb4..e6d865d56e9ef1 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -574,7 +574,7 @@ of reading a stale record is non-zero due to replicas potentially lagging behind When the number of jobs that rely on the database increases, ensuring immediate data consistency can put unsustainable load on the primary database server. We therefore added the ability to use -[database load balancing for Sidekiq workers](../administration/postgresql/database_load_balancing.md). +[Database Load Balancing for Sidekiq workers](../administration/postgresql/database_load_balancing.md). By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas under several strategies outlined below. -- GitLab From fdd5700f47d84df5eabe7ecf8a08e3729cbcca1a Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 16 Nov 2021 16:20:15 +0000 Subject: [PATCH 5/7] Further review updates --- .../postgresql/database_load_balancing.md | 41 ++++++++----------- 1 file changed, 18 insertions(+), 23 deletions(-) diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index 9de4d0754ef1c5..69462aa1cb51f6 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -10,13 +10,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60894) from GitLab Premium to GitLab Free in 14.0. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334494) for Sidekiq in GitLab 14.1. -## Overview - With Database Load Balancing, read-only queries can be distributed across multiple PostgreSQL nodes to increase performance. -We provide this functionality natively in GitLab Rails and Sidekiq where you can configure it to -it can be configured to balance the load round robin, without any external dependencies. +This functionality is provided natively in GitLab Rails and Sidekiq where +they can be configured to balance their database read queries in a round-robin fashion, +without any external dependencies. ```plantuml @startuml @@ -58,7 +57,7 @@ consul .[#e76a9b,norank]r-> postgres_secondary @enduml ``` -### Requirements to enable Database Load Balancing +## Requirements to enable Database Load Balancing Note the following requirements for Database Load Balancing: @@ -84,15 +83,13 @@ For example, on an environment that has PostgreSQL running on the hosts `host1.e `host2.example.com` and `host3.example.com` and reachable on the same port configured with `gitlab_rails['db_port']`: - 1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following line: -```ruby -gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] } -``` + ```ruby + gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] } + ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - ### Service Discovery @@ -104,7 +101,6 @@ checks a DNS A record, using the IPs returned by this record as the addresses for the secondaries. For service discovery to work, all you need is a DNS server and an A record containing the IP addresses of your secondaries. - When using Omnibus GitLab the provided [Consul](../consul.md) service works as a DNS server and will return PostgreSQL addresses via the `postgresql-ha.service.consul` record. An example of how you would configure this along with details for each @@ -112,18 +108,17 @@ setting after is as follows: 1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following: -```ruby -gitlab_rails['db_load_balancing'] = { 'discover' => { - 'nameserver' => 'localhost' - 'record' => 'postgresql-ha.service.consul' - 'record_type' => 'A' - 'port' => '8600' - 'interval' => '60' - 'disconnect_timeout' => '120' + ```ruby + gitlab_rails['db_load_balancing'] = { 'discover' => { + 'nameserver' => 'localhost' + 'record' => 'postgresql-ha.service.consul' + 'record_type' => 'A' + 'port' => '8600' + 'interval' => '60' + 'disconnect_timeout' => '120' + } } -} -``` - + ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -234,7 +229,7 @@ tries to use the next available host. If no secondaries are available the operation is performed on the primary instead. If a connection error occurs while writing data, the -operation is retried up to 3 times using an exponential back-off. +operation retries up to 3 times using an exponential back-off. When using load balancing, you should be able to safely restart a database server without it immediately leading to errors being presented to the users. -- GitLab From 4c446b4635c0e4bc018e270d6a09c619d07b6fd5 Mon Sep 17 00:00:00 2001 From: Marcia Ramos Date: Fri, 19 Nov 2021 11:12:26 +0000 Subject: [PATCH 6/7] Apply 12 suggestion(s) to 1 file(s) --- .../postgresql/database_load_balancing.md | 25 +++++++++---------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index 69462aa1cb51f6..8ddb8beab6a456 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -14,8 +14,8 @@ With Database Load Balancing, read-only queries can be distributed across multiple PostgreSQL nodes to increase performance. This functionality is provided natively in GitLab Rails and Sidekiq where -they can be configured to balance their database read queries in a round-robin fashion, -without any external dependencies. +they can be configured to balance their database read queries in a round-robin approach, +without any external dependencies: ```plantuml @startuml @@ -59,10 +59,10 @@ consul .[#e76a9b,norank]r-> postgres_secondary ## Requirements to enable Database Load Balancing -Note the following requirements for Database Load Balancing: +To enable Database Load Balancing, make sure that: -- An HA Postgres setup that has one or more secondary nodes replicating the primary. -- Each Postgres node connected with the same credentials and on the same port. +- The HA Postgres setup has one or more secondary nodes replicating the primary. +- Each Postgres node is connected with the same credentials and on the same port. For Omnibus GitLab, you also need PgBouncer configured on each PostgreSQL node to pool all load-balanced connections when [configuring a multi-node setup](replication_and_failover.md). @@ -89,7 +89,7 @@ For example, on an environment that has PostgreSQL running on the hosts `host1.e gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). ### Service Discovery @@ -102,9 +102,8 @@ for the secondaries. For service discovery to work, all you need is a DNS server and an A record containing the IP addresses of your secondaries. When using Omnibus GitLab the provided [Consul](../consul.md) service works as -a DNS server and will return PostgreSQL addresses via the `postgresql-ha.service.consul` -record. An example of how you would configure this along with details for each -setting after is as follows: +a DNS server and returns PostgreSQL addresses via the `postgresql-ha.service.consul` +record. For example: 1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following: @@ -153,9 +152,9 @@ upper limit on the time it takes to terminate all old database connections. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. To prevent reading from an outdated secondary the load balancer checks if it -is in sync with the primary. If the data is determined to be recent enough the +is in sync with the primary. If the data is recent enough, the secondary is used, otherwise it is ignored. To reduce the overhead of -these checks we only perform these checks at certain intervals. +these checks we only perform them at certain intervals. There are three configuration options that influence this behavior: @@ -167,7 +166,7 @@ There are three configuration options that influence this behavior: The defaults should be sufficient for most users. -An example of configuring these options with a hosts list would be: +To configure these options with a hosts list, use the following example: ```ruby gitlab_rails['db_load_balancing'] = { @@ -212,7 +211,7 @@ Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. ### Prepared statements Prepared statements don't work well with load balancing and are disabled -automatically when load balancing is enabled. This should have no impact on +automatically when load balancing is enabled. This shouldn't impact response timings. ### Primary sticking -- GitLab From d411abacdced1652de42166fba35c8191bd7bdd7 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 19 Nov 2021 12:20:28 +0000 Subject: [PATCH 7/7] Add back in redirect --- doc/administration/database_load_balancing.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/administration/database_load_balancing.md diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md new file mode 100644 index 00000000000000..92b8342f2518cf --- /dev/null +++ b/doc/administration/database_load_balancing.md @@ -0,0 +1,9 @@ +--- +redirect_to: 'postgresql/database_load_balancing.md' +remove_date: '2022-02-19' +--- + +This file was moved to [another location](postgresql/database_load_balancing.md). + + + -- GitLab