Ug- ddlZddlmZddlZddlZddlmZddlm Z m Z ddl m Z ddl mZmZmZmZddlmZddlmZmZmZmZmZmZdd lmZmZmZmZm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&dd l'm(Z(m)Z)m*Z*m+Z+m,Z,dd l-m.Z.dd l/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5m6Z6ddl7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@ddlAmBZBmCZCmDZDmEZEmFZFmGZGgdZHdZIdZJe jKZLeLjMeLjNcZOZPejQeOjRdZSejTUdZVeVWeSeSddZSeOeSePeScZOZPe jXZYeYjMeYjNcZZZ[dZ\dZ]ej^ddgdZ_ej`adeHej`adddgd Zbej`adeHej`adddgd!Zcej`adeHej`adddgd"Zdej`adeHej`adddgd#Zeej`adeHej`adddgd$Zfej`adeHej`adddgd%Zgej`adeHej`adddgej`ad&dgeEzej`ad'd(d)gd*Zhd+Zid,Zjd-Zkd.Zld/Zmd0Znej`ad1gd2ej`ad3eEd4Zoej`ad5gd6ej`ad7ddgej`ad3eEd8Zpej`ad5gd6ej`ad7ddgej`ad3eEd9Zq dd@Zrej`adAdBegdCdgeEzDej`adDgdEej`adFejQdGdHZsej`adIdJdKgej`adLejtgeEzej`adMdNdOgej`adddgej`adPgdQdRZudSZvej`adIdJdKgej`adLejtgeEzej`adTdUdVgej`adWgdXdYZwej`ad&dgeEzej`adZgd[d\Zxd]Zyd^Zzej`ad_ed`efed`efgdaZ{ej`ad_eefeefgej`adbddGgdcZ|ddZ}deZ~dfZdgZej`adhddie\gej`adbde*djgej`ad&dgeEzdkZej`adbde*djgej`ad&dgeEzdlZdmZdnZej`adoe5ej`adpe@ege?qej`adredJsge?qdtZej`adue6dvdwZej`aduee2dxZej`adyeyeze~eeefej`ad3eEdzZd{Zej`ad|eefd}Zd~Zej`adhdde]gdZej`adhddie\gdZej`adeegdZej`adbddGgej`adeegdZdZdZej`adddGgdGdggej`ad&eCeDzeEzeFzeGzdZdZej`adeegej`adddiedfddiedfddiedfgdZej`adeegdZdZe n_features. If "wide", then n_features > n_samples. For "wide", we return the minimum norm solution w = X' (XX')^-1 y: min ||w||_2 subject to X w = y Returns ------- X : ndarray Last column of 1, i.e. intercept. y : ndarray coef_ols : ndarray of shape Minimum norm OLS solutions, i.e. min ||X w - y||_2_2 (with minimum ||w||_2 in case of ambiguity) Last coefficient is intercept. coef_ridge : ndarray of shape (5,) Ridge solution with alpha=1, i.e. min ||X w - y||_2_2 + ||w||_2^2. Last coefficient is intercept. rL) )rQrP) n_samples n_featureseffective_rank random_stateNMbP? lowhighsizer^rJr)rWrW)paramminr@random RandomStater rr7alluniformnormalTdiagidentitysolvenorm)global_random_seedrequestrRrSkrngXUsVtU1U2Vt1_coef_olsyalphad coef_ridgeR_OLSR_Ridges rFols_ridge_datasetr`s,>} % :: % : Iz""A )   2 3 3C 1SV   AAaaaeHz!}}HAq" 6!d(   qqq"1"uXqABBxB AAAY122qqq5 C};;3Rj;AA L R#**)j"8*99Q> >> KKCbyK 9 95271q5>>)BD014 E  J'''AAfIacAgk13733J H E!j. G 9>>% 29>>'#:#: : : : : a: %%rHsolver fit_interceptTFc|\}}}}d}t|d||dvrdnd|} |tj|z } |||zz } dtj| dztj| dzz z } t di| } |d d d d f}|r |d }n2||d z }||z }d }| |||d d }| jtj|ksJt| j || ||tj| ksJt di| ||tj |j d  } | jtj|ksJt| j || ||tj| ksJ| j|ksJd S)zTest that Ridge converges for all solvers to correct solution. We work with a simple constructed data set with known solution. ?Tr;r<V瞯<绽|=rzrrtolrUrVrJNrWraxis sample_weight)dictr@rAsumrfit intercept_pytestapproxr)coef_scoreonesshapesolver_)rrrrlrpryrwcoefrzrNres_null res_RidgeR2_Ridgemodel intercepts rFtest_ridge_regressionrs&MAq!T E ..EEE' F271::~HAH I26)Q,''"&1*=*===H OOFOOE !!!SbS& AH A  L  IIaOOO 9D  v}Y77 7 7 7 7EK&&& ;;q!   h 7 7 7 7 7 7 OOFOO  1BGAGAJ4G4G  H HE  v}Y77 7 7 7 7EK&&& ;;q!   h 7 7 7 7 7 7 =F " " " " " "rHcn|\}}}}|j\}} d} t| dz |||dvrdnd|} |ddddf}d tj||fd z}tj|t || d z ksJ|r |d} n2||d z }||z }d } | |||dd}| j tj | ksJt| j tj||fd dS)a Test that Ridge converges for all solvers to correct solution on hstacked data. We work with a simple constructed data set with known solution. Fit on [X] with alpha is the same as fit on [X, X]/2 with alpha/2. For long X, [X, X] is a singular matrix. rrJrrrrNrW?rVrr:0yE>atol)rrr@ concatenater matrix_rankrarArrrrr)rr_ rrrrlrpryrwrrRrSrzrrs rF test_ridge_regression_hstacked_Xrs_&MAq!TGIz E ai#..EEE'    E !!!SbS& A bnaV!,,,,A 9  # #s9j1n'E'E E E E EH A  L  IIaOOO 9D  v}Y77 7 7 7 7EKtTz!2>>>>>>rHcf|\}}}}|j\}} d} td| z|||dvrdnd|} |ddddf}tj||fd }tj|t || ksJtj||f}|r |d} n2||d z }||z }d } | |||dd}| j tj | ksJt| j|d dS) aJTest that Ridge converges for all solvers to correct solution on vstacked data. We work with a simple constructed data set with known solution. Fit on [X] with alpha is the same as fit on [X], [y] [X], [y] with 2 * alpha. For wide X, [X', X'] is a singular matrix. rrJrrrrNrWrrrr)rrr@rrrrarrArrrrr)rrs rF test_ridge_regression_vstacked_XrsZ&MAq!TGIz E %i#..EEE'    E !!!SbS& A 1vA&&&A 9  # #s9j'A'A A A A A ad AH A  L  IIaOOO 9D  v}Y77 7 7 7 7EKD111111rHcN|\}}}}|j\}} d} t| |||dvrdnd|} td i| } |r!|ddddf}|d} |dd}nd} | |||| ks|s6| jt j| ksJt| j|dSt| ||t||z| z|tj tj | j| jftj tj | |fksJt jd | jt j| ksJt| j|dS) aTest that unpenalized Ridge = OLS converges for all solvers to correct solution. We work with a simple constructed data set with known solution. Note: This checks the minimum norm solution for wide X, i.e. n_samples < n_features: min ||w||_2 subject to X w = y rrrrrNrW1Ridge does not provide the minimum norm solution.reasonr)rrrrrrrr)rpredictr@rrkrxfail)rrrrlrpryrrwrRrSrzrNrrs rF!test_ridge_regression_unpenalizedr's&MAq$GIz E #..EEE' F OOFOOE aaa"fIH CRCy  IIaOOO :]6=#;#;;;;; T*****  a((!,,,D9,a000y~~beE$4ek$ABCCbinn E)T/ "G G       OPPPP6=#;#;;;;; T*****rHc"|\}}}}|j\}} d} t| |||dvrdnd|} |r!|ddddf}|d} |dd}nd} dtj||fd z}tj|t || ksJ| |||| ks|sa| jtj | ksJ|d krtj t| j tj||fdSt| ||tjtj| j| j ftjtj| ||fksJtjd | jtj | ksJt| j tj||fdS)a^Test that unpenalized Ridge = OLS converges for all solvers to correct solution. We work with a simple constructed data set with known solution. OLS fit on [X] is the same as fit on [X, X]/2. For long X, [X, X] is a singular matrix and we check against the minimum norm solution: min ||w||_2 subject to min ||X w - y||_2 rrrrrNrWrrVrr9rr)rrr@rrrrarrrrskipr)rrrrkr rrrrlrpryrrwrRrSrzrrs rF,test_ridge_regression_unpenalized_hstacked_Xr^s&MAq$GIz E #..EEE'    E aaa"fIH CRCy  bnaV!,,,,A 9  # #s9j'A'A A A A A IIaOOO:]6=#;#;;;;; Z   KMMM RU4:%677777  a((!,,,y~~beE$4ek$ABCCbinn E)T4' (G G       OPPPP6=#;#;;;;; RU4:%677777rHc|\}}}}|j\}} d} t| |||dvrdnd|} |r!|ddddf}|d} |dd}nd} tj||fd}tj|t || ksJtj||f}| |||| ks|s6| j tj | ksJt| j |dSt| ||tjtj| j | j ftjtj| |fksJtjd | j tj | ksJt| j |dS) aTest that unpenalized Ridge = OLS converges for all solvers to correct solution. We work with a simple constructed data set with known solution. OLS fit on [X] is the same as fit on [X], [y] [X], [y]. For wide X, [X', X'] is a singular matrix and we check against the minimum norm solution: min ||w||_2 subject to X w = y rrrrrNrWrrr)rrr@rrrrarrrrrr)rrrkrrs rF,test_ridge_regression_unpenalized_vstacked_Xrs&MAq$GIz E #..EEE'    E aaa"fIH CRCy  1vA&&&A 9  # #s9j'A'A A A A A ad A IIaOOO:]6=#;#;;;;; T*****  a((!,,,y~~beE$4ek$ABCCbinn E)T/ "G G       OPPPP6=#;#;;;;; T*****rHsparse_containerrzr{Gz?c|=|r|tvrtjn|s|tvrtj|\}}}} |j\} } t dd| } t||||dvrdndd| } |dddd f}tj ||fd }tj ||f}tj | d| z f|z} |r | d }n2|| d z }|| z }d}| ||}| ||| | dd } | j tj|ksJt| j| dS) zTest that Ridge with sample weights gives correct results. We use the following trick: ||y - Xw||_2 = (z - Aw)' W (z - Aw) for z=[y, y], A' = [X', X'] (vstacked), and W[:n/2] + W[n/2:] = 1, W=diag(W) NrrVr[rrr順)rzrrrmax_iterrUrWrr)SPARSE_SOLVERS_WITH_INTERCEPTrr SPARSE_SOLVERS_WITHOUT_INTERCEPTrrorerr@rrrArrrr)r)rrrrzrrlrpryrwrrRrSswrrs rF$test_ridge_regression_sample_weightsrs$#  V+HHH KMMMM 61Q#Q#Q KMMM%MAq!TGIz  3 3B #..EEE'    E !!!SbS& A 1vA&&&A ad A r1r6z U "BH A  L #  Q   IIa"I%%% 9D  v}Y77 7 7 7 7EK&&&&&rHcBtdd}tt|dg}t jttj}t||dg}t jtj|j}t||dS)NrWrVrrz) y_diabetesreshaper X_diabetesr@dotrgrr+)ryrK dual_coefcoef2s rFtest_primal_dual_relationshiprs2q!!A :q 7 7 7D z:<((A&q!D6:::I F:< + + -EdE*****rHc 2tjd}|d}|dd}d}t jt |5t||ddddd ddddS#1swxYwYdS) NrrZz3sparse_cg did not converge after [0-9]+ iterations.matchrr8rV)rzrrrverbose)r@rbrcrandnrwarnsr r)roryrpwarning_messages rF&test_ridge_regression_convergence_failr s )   " "C ! A !RAPO ( @ @ @   qKS4QR                      s(B  BBctjd}d\}}|||}||}|ddtjf}tj|d|zf}t }||||jj |fksJ|j j dksJt|jtj sJt|j tsJ||||jj d|fksJ|j j dksJt|jtj sJt|j tj sJ||||jj d|fksJ|j j dksJt|jtj sJt|j tj sJdS)NrrrZrVrrVrJ)rJ)r@rbrcrnewaxisc_rrrrr isinstancendarrayfloat)rorRrSrpryY1Yridges rFtest_ridge_shapes_typers )   " "C!Iz )Z((A )A 111bj= B aQhA GGE IIaOOO ;  - - - -   !R ' ' ' ' ek2: . .... e& . .... IIa ; J / / / /   !T ) ) ) ) ek2: . .... e& 3 3333 IIaOOO ; J / / / /   !T ) ) ) ) ek2: . .... e& 3 333333rHctjd}d\}}|||}||}tj|d|zf}t }||||j}|||t|jd|t|jd|dzdS)NrrrrV) r@rbrcrrrrrr*)rorRrSrpryrrrs rFtest_ridge_interceptr4s )   " "C!Iz )Z((A )A aqjA GGE IIaOOO I IIaOOO(+Y777(+Y_=====rHctjd}d\}}||}|||}t dd}t d}||||||t|j|j||||||t|j|jdS)Nr)rrQrFrzrr) r@rbrcrrr rr*r)rorRrSryrprolss rFtest_ridge_vs_lstsqrFs )   " "C Iz )A )Z((A 5 1 1 1E  / / /C IIaOOOGGAqMMM SY/// IIaOOOGGAqMMM SY/////rHcN tjd}d\}}}||| ||| tj| tj fdt jD} fddD}|D]}t||t dd}d}tj t| 5| ddddS#1swxYwYdS) N*)rZrcjg|]/\}}t|d|j0S)r9rzrrrr).0rztargetrps rF z3test_ridge_individual_penalties..gsM   v j 1 1 1 5 5a @ @ F   rHcfg|]-}t|dj.S)-q=)rzrrr)rrrp penaltiesrys rFrz3test_ridge_individual_penalties..msK  If%888<$>4@P#QRR&**1--x||A???&(,,Q//1B1BCCCCCrHr))rZrV) r)r)rJrJ)rruniform_weightscDtjd}|j|}|r tj|jd}n|d|d}tj|}tj|d|}||z |dddfz}| |j } |||dddfz} td} | | |\} } t|| t| | dSNrrV)rweightsTr)r@rbrcrrr chisquaresqrtaveragerrgr _compute_gramr))rrrrorprr X_mean X_centered true_gramX_sparsegcv computed_gram computed_means rFtest_compute_gramr s )   " "C 5A( WQWQZ  ]]1eAh ' 'gbkkG Z2 . . .Ff*4 00Jz|,,I}QD!1122H $ ' ' 'C##4Xw#G#G M=FM***I}-----rHcDtjd}|j|}|r tj|jd}n|d|d}tj|}tj|d|}||z |dddfz}|j |} |||dddfz} td} | | |\} } t|| t| | dSr)r@rbrcrrrrrrrgrr_compute_covariancer))rrrrorprr rrtrue_covariancerr computed_covrs rFtest_compute_covariancer%s! )   " "C 5A( WQWQZ  ]]1eAh ' 'gbkkG Z2 . . .Ff*4 00J l&&z22O}QD!1122H $ ' ' 'C"%"9"9(G"L"LL-FM***O\22222rHdrrZrV*@>@c &t|||||||d|  \} } }|dkrtj|g}| |z } tj| d|| jdk}| }d| |<d||<| ||z} | rG| | tj |dz|z z } tj |dz}|dkr|d}| r| | |fS| | fS)NT) rRrS n_informativerbiasnoiseshufflerrUrVrr) r r@asarrayrbrcbinomialrcopyrabs)rRrSproportion_nonzeror*rr+X_offsetr,r-rpositiverUrprycmask removed_Xs rF_make_sparse_offset_regressionr8s6#  !   GAq!Q JsOOMA l++44Q8JAGTTWXX IAteHIdOq  A QUU26!99q=1$ % %% F1IIMQ aD !Qw a4KrHzsolver, sparse_containerc#0K|]\}}||dv ||fVdS)N)r8ridgecvr)rrrs rF r;sO &V%  #v1I'I'I !" (J'I'I'I rH)r9r;r8r:r<r:z"n_samples,dtype,proportion_nonzero))rfloat32皙?)(r<r)rfloat64皙?seedrc d}|dkrdnd}tdd||||\}} t|}td| || } ||d }| |d } | ||}|d krt |g } nt|d|} | || t | j| jddt | j| jdddS)Nrg?gI@g@@rZ)r+rSr2r,rUrRr7)rrzF)r0r:alphasr)rrrzrXrrtol) r8r"rrastyperr)rr) rr2rRdtyperrArzr,rpry svd_ridgers rFtest_solver_consistencyrKs7& E&,,DD%E ) -    DAq QAU%00044Q::I U##A U##A#  Q   w'''Ve<<< IIaOOOEKt$GGGGE$i&:DQQQQQQrHgcv_moder7eigen X_containerX_shape)rr)rrzy_shape, noise))rr)rrVr()rrb@c |\}}t|dkr|dnd}t|||dd|d\} } | |} gd} t||| d } t||| } | | | || }| || | jt j| jksJt| j | j d t| j | j d dS)NrJrWrVrFrrRrSrrUr-r,r*rXr=r$@g@@neg_mean_squared_errorcvrrEscoring)rLrrErXrG) rr8rrralpha_rrr)rr)rLrNrOy_shaperr,rRrSrrpryrE loo_ridge gcv_ridgeX_gcvs rFtest_ridge_gcv_vs_ridge_loo_cvrbsG$Iz"7||q00 aI )   DAq 'A ( ( (F #( I #I MM!Q KNNE MM%  v}Y-=>> > > > >IOY_4@@@@I()*>TJJJJJJrHc d}d\}}d}t|||dddd\}}gd}t|d || }td || }|||||||jt j|jksJt |j|jd t |j|jd dS)Nexplained_variance)rZrrVrFrrUrVTrY)rrEr[rXr\) r8rrr]rrr)rr) r[rRrSrrpryrEr_r`s rFtest_ridge_loo_cv_asym_scoringreEs"G!IzI )   DAq) ( (F DId67KKKI MM!Q MM!Q  v}Y-=>> > > > >IOY_4@@@@I()*>TJJJJJJrHrSrrzy_shape, fit_intercept, noise))rPTr)rQTg4@)rRTrS)rRFr(c.gd}tjd}t|dkr|dnd}t d||dd|\} } | |} d |t| z} | | z dzt} tj tj | j d| | t} | | } } t| j d }|| |  }t!||d | }|| | t%|j|}|| |  }t)|| | |}| |z dzfdtj | j dDtj|| }t!|d||}||| | t|dkr-|jdddd||jf}n)|jdd||jf}|jt1j|jksJt5|dt5|j|jdt5|j|jddS)NrVrrJrWrVrF)rRrSrrUr-r,r)n_splits)groupsrX)rErZr[rrrZcPg|]"}tj|kd#S)rr)r@r)riindices kfold_errorss rFrz1test_ridge_gcv_sample_weights..s>78|GqL)222rHT)rEstore_cv_resultsrLrrrXr\)r@rbrcrr8rrrarHintrepeatrrrrsplitrrrr]r!r. cv_results_indexrrr)rr)rLrNrrSr^r,rErorrpryrX_tiledy_tiledrZsplitskfold ridge_reg predictionsrar` gcv_errorsrlrms @@rFtest_ridge_gcv_sample_weightsr{cs") ( (F )   " "C"7||q00 aI )    DAq 'A #a&&)))M"]%6%6%8%881<DDSIIMi !'!*--}==G!((//Mz1W:WG QWQZ ( ( (B XXgwwX 7 7F  (#    E  IIgwEL FFFI XXgwwX 7 7F#IwFKKKKk)a/L<>Iagaj > > > > > >rHc<tjd}g}| td}}n|td}}t|}||t|j}||t}ttd}td|} || j|t| jtj |ksJd} t| }td|} || j|t| jtj |ksJtd} td| } | |t| jtj |ksJ|N||ttj| |jtj |ksJtjttfj}|||||}||t||}t'tj||fj|d |S) NrTFr)greater_is_better)rr[c$t|| Sr?)r)xrys rFfuncz_test_ridge_loo..funcs"1a((((rHrXrh㈵>r\)rrrrrr]appendr-rrrrrrr@rvstackrgrr))rrRretrpr ridge_gcvr]fr[ ridge_gcv2r ridge_gcv3scorer ridge_gcv4rY_predrDs rF_test_ridge_loors| #I C%t=++J77= 666IMM!Z    FJJv A,FFFGug>>>JAAjna$$$   f 5 5 5 5 5 5)))$Gug>>>JAAjna$$$   f 5 5 5 5 5 50 1 1Fuf===JNN1j!!!   f 5 5 5 5 5 5 a2793E3E FFF6=#8#88888 :z*++-A MM!Q   q ! !F MM!Z     q ! !FBIvv.//16EEEE JrHcb|tn|t}t}||t||t |jjdksJt|j tj usJtd}| |||t||t |jjdksJt|j tj usJdS)NrVrri)rrrrrrrrtyperr@r?r set_params)rrpridge_cvrZs rF_test_ridge_cvrs&. 4D4DZ4P4PAyyH LLJ Q x~# $ $ ) ) ) ) # $ $ 2 2 2 2 qB 2 LLJ Q x~# $ $ ) ) ) ) # $ $ 2 2 2 2 2 2rHzridge, make_datasetrncv|dd\}}|||t|drJdS)NrrRrUrr)rhasattr)r make_datasetrprys rF$test_ridge_gcv_cv_results_not_storedr sI .9s7UUUvgV,,,00F;;BUUUrHT)rEalpha_per_targetr)rErrnr2)rErr[)rErZrz3cv!=None and alpha_per_target=True are incompatiblerr)r@rbrcrrrrgrrr,r]r+rrrrrrrisscalarr rrr) rorRrSrryoptimal_alphasrmsgrprEs @@rF"test_ridge_cv_individual_penaltiesr&s )   # #C(0$Iz9 )Y''A qQCy"'1j/2233 &111qc6D27Az?#;#;; < < = &111qc6EBGQ O$<$<< = = > ))Iz * * +FVUUUUQRQTUUUNft<<<@@AFFH~x777 HO$$$((A..4hn ftdSSSWW 1H ? YL 0 0 0 0   %) 5 5 5 5   %)S[[))L L L L La$NNNRRSTVWXXH ? YL 0 0 0 0   %) 5 5 5 5   %)Y)B B B B BftdSSSWW 1QQQT7H ;x ' '''' ;x+ , ,,,,   %)S[[)A A A A AftTJJJNNqRSTTH~x777 HO$$$((A..4hn fNNNH ?C z - - - QfTBBBH z - - - Qs$N::N>N>2PPPc|tn|t}td}||tt j||tdS)NFrr)rrrrr@roundr)rrprs rF_test_ridge_diabetesrms`&. 4D4DZ4P4PA  & & &E IIa 8EKK:.. 2 22rHc|tn|t}tjttfj}tjd}t d}||||jjd|fksJ| |}||t| |}ttj||fj|ddS)NrVFrrJrdecimal) rr@rrrgrrrrrr+)rrprrSrrrDs rF_test_multi_ridge_diabetesrts&. 4D4DZ4P4PA :z*++-A!!$J  & & &E IIaOOO ; J / / / / ]]1  F IIa ]]1  Fbi(899;VQOOOOOOrHctjtjd}tjd}|tn|t}t t fD]i}||t|jj||fksJ| |}tj t|kdksJjtd}t |}||t| |}tj t|kdksJdS)NrrVgHzG?rrig?) r@uniquey_irisrX_irisrrrrrrAr)r n_classesrSrpregrDrZs rF_test_ridge_classifiersrs# &!!'*IaJ"*0@0@0H0HA!!#4#6#6700 6y9j"99999Qwv'((4///// qB r " " "CGGAv [[^^F 76V# $ $ + + + + + +rHr[accuracyrc|tn|t}t|rt|n|}t||}||t |dS)N)r[rZ)rcallablerrrrr)rr[rZrpscoring_clfs rF"test_ridge_classifier_with_scoringrsw#*0@0@0H0HA'/'8'8E{7###gH H 4 4 4CGGAvq!!!!!rHchd}|tn|t}tjddd}t|t ||}||t |jtj dksJ|j tj |dksJdS) NcdS)NzG?rrBs rF _dummy_scorez:test_ridge_regression_custom_scoring.._dummy_scorestrHrJr)num)rEr[rZrr) rr@logspacerrrrrrrr])rrZrrprErs rF$test_ridge_regression_custom_scoringrs #*0@0@0H0HA [QA & & &F 6;|3L3LQS T T TCGGAv ?fmD11 1 1 1 1 :vay11 1 1 1 1 1 1rHcd|tn|t}tdd}||t||t}tdd}||t||t}||ksJdS)NrF)rrrX)rrrrr)rrprrridge2score2s rF_test_tolerancers&. 4D4DZ4P4PA d% 0 0 0E IIa KK: & &E t5 1 1 1F JJq* \\!Z ( (F F??????rHct||}t|}t|}|||}|||} ||||j} |j} td5t||| } | j} | j dksJ| j |j ksJtt| || t|| j}|j dksJ|j |j ksJtt||| t|ddddS#1swxYwYdS)NdeviceTarray_api_dispatch)rQ)xprr)r.rrHrr.rrrrrrrIr)r&r%)name estimatorarray_namespacer dtype_namer X_iris_np y_iris_np X_iris_xp y_iris_xpcoef_np intercept_np estimator_xpcoef_xp intercept_xps rFcheck_array_api_attributesrs ov 6 6B j))I j))I 9V 44I 9V 44I MM)Y'''oG'L 4 0 0 0  Y''++IyAA $}$$$$} //// g" - - -  ++    $. !R''''!Y_4444 lr 2 2 2  ++                      s'C FFFz#array_namespace, device, dtype_namecheck)idsrrc>|jj}||||||dS)N)rr) __class____name__)rrrrrrs rFtest_ridge_array_api_compliancers0   'D E$ ?6jQQQQQQrHr)include_numpy_namespacescVt|d}|tdd}|tdd}tjddj}|ddhz D]}t ||dk}d |jd |d }tj t| 5td 5| ||dddn #1swxYwYdddn #1swxYwYt dd }d|jd}tj t| 5td 5| ||dddn #1swxYwYdddn #1swxYwYt }d|jd}tj t| 5td 5| ||dddn #1swxYwYddddS#1swxYwYdS)Nrrrrr|r7lbfgsrr4z Array API dispatch to namespace z" only supports solver 'svd'. Got 'z'.rTrzYThe solvers that support positive fitting do not support Array API dispatch to namespace zc. Please either disable Array API dispatch, or use a numpy-like namespace, or set `positive=False`.z&Using Array API dispatch to namespace z with `solver='auto'` will result in using the solver 'svd'. The results may differ from those when using a Numpy array, because in that case the preferred solver would be cholesky. Set `solver='svd'` to suppress this warning.)r.r.rrr_parameter_constraintsoptionsrrrrrrr UserWarning)rrrravailable_solversrr expected_msgs rF6test_array_api_error_and_warnings_for_solver_parameterrs od ; ; ;B 6"1":&&I 6"1":&&I4X>qAI#vuo5 0 0Vf.?@@@ -r{ - -"( - - -  ]:\ : : : 0 04888 0 0 )Y/// 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 $ / / /E .+-; . . . z 6 6 6,, t 4 4 4 , , IIi + + + , , , , , , , , , , , , , , ,,,,,,,,,,,,,,,, GGE D D D D k 6 6 6,, t 4 4 4 , , IIi + + + , , , , , , , , , , , , , , ,,,,,,,,,,,,,,,,,,,s:D C." D.C2 2D5C2 6DD D FE<0 F<F FF FFFH#H: HH H H HH"%H"cvt|d}|tdd}|tdd}t }d}t j5t jd|ttd5| ||dddn #1swxYwYdddn #1swxYwYtd5t dd  ||ddddS#1swxYwYdS) NrrzkResults might be different than when Array API dispatch is disabled, or when a numpy-like namespace is usederror)messagecategoryTrr|r) r.r.rrrwarningscatch_warningsfilterwarningsrrr)rrrrrrs rF)test_array_api_numpy_namespace_no_warningr's od ; ; ;B 6"1":&&I 6"1":&&I GGE ;  " ",, TTTT t 4 4 4 , , IIi + + + , , , , , , , , , , , , , , ,,,,,,,,,,,,,,,, 4 0 0 0FF Vd+++// 9EEEFFFFFFFFFFFFFFFFFFsH9-C &C = C  C C C C  C$'C$;&D..D25D2 test_funccf|d}||}||t||ddSdSdS)Nrr)r+)rr ret_dense ret_sparses rFtest_dense_sparser?sT $I=))J!7!)ZCCCCCC!7!7rHctjddgddgddgddgddgg}gd}td}|||t |ddggtjd gtd d i}|||t |ddggtjd gtd }|||t |ddggtjd gtjddgddgddgddgg}gd }td}|||td }|||t |jdksJt|j |j t|j |j dS)Nr皙rrrVrVrVrWrW class_weightr@rVrXrWbalanced)rVrVrWrWrJ) r@rrrr,rrclasses_r+rr)rpryrregas rFtest_class_weightsrUs 4,q D$<#sc3ZPQQAA t , , ,CGGAqMMMs{{S$K=1128QC==AAA 5z 2 2 2CGGAqMMMs{{S$K=1128RD>>BBB z 2 2 2CGGAqMMMs{{S$K=1128QC==AAA 4,q D$<#sDEEAA t , , ,CGGAqMMM  3 3 3DHHQNNN t}   " " " "ci444cndo>>>>>rHrc|}|tjtj|d}|tjtjt |j|jt jtjj}|tjdkxxdzcc<dddd}|}|tjtj|||}|tjtjt |j|j|}|tjtj|dz||}|tjtj|t |j|jd S) z5Check class_weights resemble sample_weights behavior.rrrVr&rgY@)rrVrJrJN) ririsdatarr*rr@rr)rreg1reg2rrs rF"test_class_weight_vs_sample_weightr xs 355DHHTY $$$ 3J ' ' 'DHHTY $$$ DJ///GDK-..M$+"###s*###u--L 355DHHTY ]333 3L ) ) )DHHTY $$$ DJ/// 355DHHTY ]A%5666 3L ) ) )DHHTY ]333 DJ/////rHcntjddgddgddgddgddgg}gd}tdgd}|||td d igd }|||t |d d ggtjdgdS)Nrrrrrr)rr=rV)rrErVrX)rr=rVrZgɿrJrW)r@rrrr,r)rpryrs rFtest_class_weights_cvrs 4,q D$<#sc3ZPQQAA nnn E E ECGGAqMMM !U 0.0.r) r@rbrcrrrrrrr) rZrrorErRrSryrprs rFtest_ridgecv_alphas_zeror s1 )   " "C F IzG IIi  KK1i ( ( )Z((A B///I z ]:-R S S S  MM!Q                       asCCCc.tjd}d}dD]\}}||}|||}d||z}t d}t ||}||||d|i} tt| | } | ||||j | j j ksJt|j| j jdS) Nrr)rrrrrrrrzri)r@rbrcrrandrrrrrr]best_estimator_rzr+r) rorErRrSryrprrZr: parametersgss rFtest_ridgecv_sample_weightr' s )   " "C F"3KK : IIi  IIi , ,chhy111 1XXB/// Aq 666v& %'':" 5 5 5 q!=111~!3!99999!'-1C1IJJJJKKrHc@ ddg}ddg}tjd}t||D]_\}}||| ||||dzdz}d}d}|ddtjf |tjddft d  |  |  | fd} fd } d } tj t| 5|dddn #1swxYwYd } tj t| 5| dddn #1swxYwYadS) NrJrrrVrg@rc6dSr?r)rprsample_weights_not_OKrysrFfit_ridge_not_okzStest_raises_value_error_if_sample_weights_greater_than_1d..fit_ridge_not_ok<s IIa1 2 2 2 2 2rHc6dSr?r*)rprsample_weights_not_OK_2rysrFfit_ridge_not_ok_2zUtest_raises_value_error_if_sample_weights_greater_than_1d..fit_ridge_not_ok_2?s IIa3 4 4 4 4 4rHz)Sample weights must be 1D array or scalarr) r@rbrcrrrrrrrr) n_sampless n_featuressrorRrSsample_weights_OKsample_weights_OK_1sample_weights_OK_2r,r/rrprr+r.rys @@@@@rF9test_raises_value_error_if_sample_weights_greater_than_1dr5$ssQJa&K )   # #C!$Z!=!=!! : IIi , , IIi IIi00A59!! 1!!!RZ- @"3BJM"BA  !Q)*** !Q+,,, !Q+,,, 3 3 3 3 3 3 3 3 5 5 5 5 5 5 5 5> ]:W 5 5 5                      > ]:W 5 5 5 ! !    ! ! ! ! ! ! ! ! ! ! ! ! ! ! !7!!s$; EE E : FF F zn_samples,n_featuresrJctjd}tdd}tdd}|||}||}||dzdz}||} || ||||||t |j|jd dS) NrrFrrJrVrrr)r@rbrcrrrr+r) rRrSrro sparse_ridge dense_ridgerprysample_weightsrs rF&test_sparse_design_with_sample_weightsr:Ks )   # #Cs%888Lc777K )Z((A )AYYy))Q.2N""HXq???OOAqO777l0+2CQOOOOOOrHctjddgddgddgddgddgg}gd}td}|||dS) Nrrrrrr)rVrZr&rD)r@rrr)rpryrs rFtest_ridgecv_int_alphasr<asb 4,q D$<#sc3ZPQQAA < ( ( (E IIaOOOOOrHzparams, err_type, err_msgrE)rVrWiz alphas\[1\] == -1, must be > 0.0)grg$z"alphas\[0\] == -0.1, must be > 0.0)rVr1z1alphas\[2\] must be an instance of float, not strcd\}}t||}tdd|}tj||5|di|||ddddS#1swxYwYdS)z?Check the `alphas` validation in RidgeCV and RidgeClassifierCV.rrrJrNr)rorrrrr)rrNerr_typerrRrSrprys rFtest_ridgecv_alphas_validationr@js(!Iz )Z((A Aq)$$A xw / / /&& F1%%%&&&&&&&&&&&&&&&&&&sA==BBcd\}}t||}|turt|}ntdd|}|d||dS)zCheck the case when `alphas` is a scalar. This case was supported in the past when `alphas` where converted into array in `__init__`. We add this test to ensure backward compatibility. rrrJrVrDN)rorrrr)rrRrSrprys rFtest_ridgecv_alphas_scalarrBs}!Iz )Z((AG IIi  KK1i ( ( IQAq!!!!!rHctdd}|tt|jjdtjdksJdS)Nr8rV)rrr)rrrrrr)rs rFtest_sparse_cg_max_iterrDsP {Q / / /CGGJ ### 9?1 !1!!4 4 4 4 4 4 4rHc d}tt}}tj||dfj}t ddD]W}dD]R}t ||d}|||t|j tj||SXdD]3}t |dd}||||j J4dS) NrJrVrQ)r;r<r:r)rrr)r8r7r9r=) rrr@tilergrangerrr,n_iter_)rrpryy_nrrrs rF test_n_iterrJsI zqA '!i^ $ $ &C!QKKJJ- J JFveDDDC GGAsOOO s{BGHi,H,H I I I I J 3##6A4888 3{""""##rH)r:r8rr|with_sample_weightc|dk}td||\}}d}|rCtj|}d||jdz}|dkrd n|} t | d | } t |d | } | ||| | |||| t| j | j t| j | j d dS)aCheck that ridge finds the same coefs and intercept on dense and sparse input in the presence of sample weights. For now only sparse_cg and lbfgs can correctly fit an intercept with sparse X with default tol and max_iter. 'sag' is tested separately in test_ridge_fit_intercept_sparse_sag because it requires more iterations and should raise a warning if default max_iter is used. Other solvers raise an exception, as checked in test_ridge_fit_intercept_sparse_error rr)rSrUr4Nrrr_r|r8r)rrr4rgƠ>r\) r8r@rbrcrerrrr)rr) rrKrlrr4rpryrro dense_solverr8r7s rFtest_ridge_fit_intercept_sparserNs"  H )$6   DAqM;i##$677ckkqwqzk::: #)F"2"2;;L|JJJKEHEEELOOAq O666]]1%%q FFFK*L,CDDDK%|'9EEEEEErH)r<r7r9c"tdd\}}||}t|}d|}tjt |5|||ddddS#1swxYwYdS)Nrr)rSrUrzsolver='{}' does not supportr)r8rformatrrrr)rrrpryX_csrr7rs rF%test_ridge_fit_intercept_sparse_errorrRs *Ra H H HDAq M!  E'''L,33F;;G z 1 1 1##"""##################s BB Bc@tdd|d\}}|rDtj|}d||jdz}nd}||}t ddd d d }tdi|} tdi|} | ||| tj 5tj dt| ||| dddn #1swxYwYt| j| jdt| j| jdt!jtd5tdd dd||ddddS#1swxYwYdS)Nrrg@)rSrRrUr3rrr_r;Trr)rzrrrrrr-C6?r\z"sag" solver requires.*rrX)rrrrr)r8r@rbrcrerrrrrr simplefilterrr)rrrr) rKrlrrpryrorrQrNr8r7s rF#test_ridge_fit_intercept_sparse_sagrVsb *1Cc   DAqi##$677ckkqwqzk:::  M!  E %tF//&//K??6??LOOAq O666  " "@@g{333???@@@@@@@@@@@@@@@K*L,C$OOOOK%|'9EEEE k)B C C CWW U$D4HHHLLUTUVVVWWWWWWWWWWWWWWWWWWs$3DDD(FFFreturn_interceptrr container)r|r8r9r:r;r<rc 0td}|dd}gd}tj||}d}|rd}||z }||} d\} } trdnd } |d k} |d vrN|rLt jtd 5t| || |||| | dddn #1swxYwYdSt| || ||| || }|r-|\}}t||d| t||d| dSt||d| dS)z=check if all combinations of arguments give valid estimationsrrr)rVrJr=rg@)rXư>rXrTr)r;r|zIn Ridge, only 'sag' solverr)rzrrrWr4rN)rzrrr4rWrrrGr) r#r#r@rr1rrrrr))rWrrXrrorp true_coefsrytrue_intercept X_testingrzrrr4outrrs rF.test_ridge_regression_check_arguments_validityr`s R C qAJ q*AN! A ! IJE3 &44$D H _$$)9$ ]:-J K K K   +!1!                      #)    C<ijqt<<<< >EEEEEEZad;;;;;;s B00B47B4)r7r8r9r:r;r<rcVtjd}d}|dk}d\}}|||}||}|tj}|tj} dtjtjjz} t||d| |} | || | j } t||d| |} | ||| j }| j |j ksJ|j |j ksJ| |j |j ksJ| |j |j ksJt| j | j dd dS) Nrrrr"rJ)rzrrrr4rTgMb@?r[)r@rbrcrrHr<finfo resolutionrrrrIrr))rrorzr4rRrSX_64y_64X_32y_32rridge_32coef_32ridge_64coef_64s rFtest_dtype_matchrm@s )   " "C E H Iz 99Y + +D 99Y  D ;;rz " "D ;;rz " "D bhrz""- -CFScHH LLtnGFScHH LLtnG =DJ & & & & =DJ & & & &   D ! ! '4: 5 5 5 5   D ! ! '4: 5 5 5 5HNHNDIIIIIIrHctjd}tjddg}d\}}}|||}|||}|tj}|tj}t|d} | ||| j } t|d} | ||| j } | j |j ksJ| j |j ksJ| |j |j ksJ| |j |j ksJt| j | j ddS) Nrrr)rrrJr9rrr) r@rbrcrrrHr<rrrrIrr*) rorzrRrSn_targetrerfrgrhrirjrkrls rFtest_dtype_match_choleskyrpesu )   " "C Hc3Z E&-#Iz8 99Y + +D 99Y ) )D ;;rz " "D ;;rz " "D5444H LLtnG5444H LLtnG =DJ & & & & =DJ & & & &   D ! ! '4: 5 5 5 5   D ! ! '4: 5 5 5 5BBBBBBrH)r7r9r:r8r;r<rctj|}d\}}|||}||}tj||d||zz}d}|dk} t } |dkrdnd} tjtjfD]E} t| | | | |||d| dd d d | | <F| tjj tjksJ| tjj tjksJt| tj| tj| dS) Nr"rrrr8rXrrbrF) rzrrUrr4rr return_n_iterrWr) r@rbrcrrrr<r?rrHrIr)) rrArUrRrSrprryrzr4resultsr current_dtypes rF%test_ridge_regression_dtype_stabilityrusq 9((..L Iz9j11A   j ) )D q$$!3!3I!>!>>>A E HffG[((44dD*bj1   !1 HH] # # HH] # #%" " " "   2:  $ 2 2 2 2 2:  $ 2 2 2 2GBJ')<4HHHHHHrHctd\}}tj|}|dddddf}|ddd}td||dS)NrrUrJr;r)r r@asfortranarrayrr)rprys rFtest_ridge_sag_with_X_fortranryss  + + +DAq !A ##A#qqq& A ##A#A Aq!!!!!rHzClassifier, paramsctdd\}}|dd}tj||gd}|d i|||}||}|j|jksJt|dddf|dddftd||dS) zRCheck that multilabel classification is supported and give meaningful results.rVr)rrUrWrNr;rr) r rr@rrrrr,r) ClassifierrNrpryrrrs rFtest_ridgeclassifier_multilabelr|s *AA F F FDAq "aA 1vA&&&A *  v   " "1a ( (C [[^^F <17 " " " "vaaad|VAAAqD\222 Aq!!!!!rHr|r)rXrr=rcftjddgddgddgddgg}tjdd g}|rd }|||z}n||}t|d || }|||tj|jd ksJdS)z:Test that positive Ridge finds true positive coefficients.rVrJrrQrrrrrYrTrzr4rrrN)r@rrrrrdr)rrrzrprrryrs rF#test_ridge_positive_regression_testrs 1a&1a&1a&1a&122A 8QH  D EE$KK) # EE$KK d6   E IIaOOO 6%+" # ######rHctjd}|dd}|dd|jd}|r d}||z|z}n||z}|||jdd zz }g}d D]C}t|||d } || ||j Dt|d dddS)zTest that Ridge w/wo positive converges to the same solution. Ridge with positive=True and positive=False must give the same when the ground truth coefs are all positive. r,r&r=rrVr_rr)TFr)rzr4rrrZrFN) r@rbrcrrerrfrrrrr)) rrzrorprrryrsr4rs rF%test_ridge_ground_truth_positive_testrs )   # #C #sA ;;sCagaj; 1 1D Hy  H $ $t ++AG!..(-U    uyyA,----W4a000000rH)r7r9r:r8r;r<c d}tjddgddgg}tjddg}||z}t|d|d }tjt d 5|||d d d n #1swxYwYtjt d 5t|||d|d\}}d d d d S#1swxYwYd S)z5Test input validation for positive argument in Ridge.r=rVrJrrQrWTFr~zdoes not support positiverNzonly 'lbfgs' solver can be used)r4rrW)r@rrrrrrr)rrzrprryrrws rFtest_ridge_positive_error_testrst E 1a&1a&!""A 8QG  D DA V5 Q Q QE z)D E E E !Q z)J K K K   q%$v   1                  s$&B  B B /CCCcj tddd\ dd}d fd }t  }td  }||}||}||ksJt|D]}||| }||ksJdS)z?Check ridge loss consistency when positive argument is enabled.rrrRrSrUr=r&Nrc:|j}|Jtj|}|j|d||jjz}n|j}dtj|zz |z dzzdztj|dzzzS)Nrr_rrJ)rr@rbrcrrerr) rrU noise_scalerrorrprzrys rF ridge_lossz,test_positive_ridge_loss..ridge_losss$  #)'' 55C;Q %+BS!T!TTDD;DRVQT\I5!;<<>  D#N LOOMGGAq G...CIt$////3 222 KKDqqwqzKBBMM"##bccFFFdNFFFGGAq G... 9>>  D#N GGAcrc111fIq"v]3B3-?G@@@CIt$////3 222 ??6?? % %BEF7O,C % D DDHHQ!6H777     TvTTTUUUDJ400004333# IIKK A. Q./0q 9 9 9B A. Q./0 1 1B#((**O$i1n$%%%*%%%n  &6 Q&678qO#  Q    b ! ! ??6??  q!?  C CD ??6??  r2_  E EDDJ +++:99999::rHctdd\}}td}d}tjt|5|||dddn #1swxYwYtdd }d }tjt|5|||ddddS#1swxYwYdS) z-Check `store_cv_values` parameter deprecated.rrrT)store_cv_valuesz'store_cv_values' is deprecatedrN)rnrz2Both 'store_cv_values' and 'store_cv_results' were)r rrr FutureWarningrrrrpryrrs rF%test_ridge_store_cv_values_deprecatedrsK QR 8 8 8DAq D ) ) )E +C m3 / / / !Q T4 @ @ @E >C z - - - !Qs$A%%A),A)CC Cctdd\}}td}d}tjt|5||||jddddS#1swxYwYdS) zCheck `cv_values_` deprecated.rrrTrz$Attribute `cv_values_` is deprecatedrN)r rrrrr cv_values_rs rFtest_ridge_cv_values_deprecatedrs QR 8 8 8DAq T * * *E 0C m3 / / / !Q sA--A14A1enable_slep006 metaestimatorc>|dS)zTest that `RidgeCV` or `RidgeClassifierCV` with default `scoring` argument (`None`), don't enter into `RecursionError` when metadata is routed. N)get_metadata_routing)rs rF*test_metadata_routing_with_default_scoringrs" MOO((*****rH) r&r&rrZrVr'r(r(TFFN)r itertoolsrnumpyr@rscipyrsklearnrr sklearn.basersklearn.datasetsrr r r sklearn.exceptionsr sklearn.linear_modelr rrrrrsklearn.linear_model._ridgerrrrrrrsklearn.metricsrrrsklearn.model_selectionrrrr r!sklearn.preprocessingr" sklearn.utilsr#sklearn.utils._array_apir$r%r&r'r(sklearn.utils._testingr)r*r+r,r-sklearn.utils.estimator_checksr.r/r0sklearn.utils.fixesr1r2r3r4r5r6SOLVERSrr load_diabetesdiabetesr rrrrrindrbrcror- load_irisrrrrGrKfixturermark parametrizerrrrrrrrrrrrrrr r%r8rKr.rbrer{rrrrrrrrrrrrrrrsortedrrrr rrrrr r'r5r:r<r TypeErrorr@rBrDrJrNrRrVrrr`rmrprGruryr|rrrrrrrrr usefixturesrrrHrFrs ,,,,,,,, 211111HGGGGGGGGG/.....,,,,,,  B A A 4#S !8 ! # #! Jbi  #$$iA C $3$i#C*S/ JxDK%%%+++'(((C&C&)(C&L7++4-88)#)#98,+)#X7++4-88#?#?98,+#?L7++4-88%2%298,+%2P7++4-882+2+98,+2+j7++4-88282898,+28j7++4-882+2+98,+2+j7++4-88+dVn-DEE3+../'/'/.FE98,+/'d+++   444<>>>$000*@"2"2"233.99 D D:943 D"N"N"NOO*T5M::.99..:9;:PO.$"N"N"NOO*T5M::.9933:9;:PO3&     ****Z*1' G G G F^ #+ +   (FFF1..RR/.    RBeW%566 ~(EFFWh$7884-88$K$K9898GF76$KNKKK<eW%566 ~(EFF2w//#9G9G0/GF769Gx+dVn-DEE8??FE?666r333$ % ( ( (/:  E 2 2 24GH--- gii!$5$5$7$79L#MNay))00*)  0DDDN333 P P P,,,$T:7I$JKKeeAhh/00+dVn-DEE""FE10LK"eeAhh/00+dVn-DEE 2 2FE10 2       F)+T+T+V+V %'AB   U%  RR  R''GGG',',',T*FF3I,J,JKKFFLKF."   .99DD:9  D ? ? ?F2C DEE00FE08 A A A .0LM:T:7I$JKKCCLKC8w0A&BCC==DC=(ay))w0A&BCCDC*)(KKK0$!$!$!N/1a&1a&1ABB^#n4~EVPP CB P"w0A&BCC M "J0ST * +  1 } %  @   &&!DC"&w0A&BCC""DC""555 ###$#I#I#IJJ-e}==.99#F#F:9>=KJ#FL#>#>#>??.99##:9@?#-e}==.99WW:9>=W8+eT];;4*?@@rxj>&ABB OOO4<4<CBA@<; 4