Also available as PDF (figures and tables appear in the PDF).
Monthly rainfall in north-west India is strongly seasonal, right-skewed and, in the dry season, zero-heavy, and it is usually evaluated by aggregate error measures that may hide failures in the months that matter for agriculture. This study proposes a decision-focused evaluation protocol and applies it to a 124-year series of monthly rainfall for the Haryana-Chandigarh-Delhi region (1,488 months, 1901-2024). A two-stage forecaster, which multiplies the probability of a wet month (rainfall above 5 mm) from a gradient-boosted classifier by the expected amount in wet months from a gradient-boosted regressor, is compared with nine benchmarks (persistence, seasonal naive, monthly climatology, ridge regression, a Tweedie generalised linear model, a random forest, single-stage boosting with squared and Poisson loss, and a two-stage linear model) in an expanding-window, one-month-ahead evaluation over 1926-2024 (1,188 forecasts) with annual re-estimation. Accuracy is assessed by MAE and RMSE on all, monsoon and wet months, by the Brier skill score and calibration for occurrence, by drought-month recall, flood-month precision and onset detection, and by stability across 25-year blocks, with Diebold-Mariano tests and a block bootstrap. The main finding is negative and informative: no model beats the climatological benchmark. The two-stage boosted model has an MAE 7.3% above climatology (bootstrap interval 1.031–1.110) and a Jun-Sept MSE skill of -0.152, and is statistically indistinguishable from single-stage boosting (Diebold-Mariano p = 0.83 for absolute error). The best-calibrated probabilistic forecast is climatology itself (Brier skill scores of all other forecasters are negative). Decision-focused metrics rank models very similarly to aggregate metrics, drought recall (0.32-0.46 at a 20% alarm rate) and flood precision (0.15-0.25 at a 15% alarm rate) differ among models by amounts well inside bootstrap uncertainty, and onset predictions do no better than always predicting July. Monthly anomalies have almost no autocorrelation (0.035 at lag 1), and the calendar month alone explains 63% of the variance; the remainder is not predictable from past rainfall alone. Skill relative to climatology does not change systematically across 25-year blocks, and there is no significant trend in monsoon rainfall (Kendall τ = -0.02, p = 0.74). The results show that the benefit of two-stage structure depends on the presence of predictable occurrence and amount signals, which are absent in rainfall-history-only forecasting at monthly resolution, and they provide a reproducible benchmark and protocol for evaluating richer predictors such as sea-surface temperature or other states’ rainfall.
Keywords: zero-inflated time series; monsoon rainfall; two-stage forecasting; decision-focused evaluation; climatology benchmark; Brier skill score; gradient boosting; zero-inflatedtimeseries; monsoonrainfall; two-stageforecasting; decision-focusedevaluation; Brierskillscore
benchmark;Brierskillscore;gradientboosting methodology including the hyperparameters of all 1.Introduction models (Section 4), reports results (Section 5), dis- Agriculture in Haryana and the National Capital Re- cussesthem(Section6)andconcludes(Section7). giondependsonthesummermonsoon.Whetherthe rainsarriveontime,whethertheyareadequate,and whetherlate-seasonrainfallissufficientforcropcom- 2.ReviewofLiterature pletiondrivesowingdates,irrigationschedulingand Occurrence-and-amountmodelshavelongbeenused risk management for the kharif season (Gadgil and forprecipitation,usuallyatdailyresolution,wherea Gadgil,2006).OperationalforecastsoftheIndianmon- Markovchaingovernswetanddrydaysandaskewed soonaremostlyseasonalandprobabilistic(Gadgilet distribution, often gamma, governs amounts (Stern al.,2005;Rajeevanetal.,2007),butdecisionsaretaken and Coe, 1984; Woolhiser and Roldán, 1982). The monthbymonth,andaforecasterwhocansaysome- samelogiccarriesovertomachine-learningforecast- thingaboutthenextmonth’srainfallfromtherecord ers,whichcanmodeltheoccurrenceprobabilitywith ofpastmonthswouldbevaluableifsuchskillexisted. aclassifierandtheamountwitharegressortrainedon Monthlyrainfallisawkwardforstandardforecasting wetmonthsonly.Arelatedfamilyofmodelstreatsthe methods.Itisnon-negativeandright-skewed,ithasa zero-heavyresponsedirectly:theTweediecompound strongseasonalcycle,andindrymonthsalargeshare Poisson–gammadistributionallowsexactzeroswitha ofobservationsareclosetozero.Amodeltrainedby continuouspositivepart(Jørgensen,1987;Dunnand minimisingsquarederroronallmonthsestimatesa Smyth,2005),andPoisson-lossgradientboostinghan- conditionalmean,whichforaskewed,seasonalseries dlesnon-negative,zero-heavytargetswithoutasepa- isdominatedbytheseasonalcycleandcanunderstate rateoccurrencemodel.Theseapproachesarenatural peaksandoverstatedrymonths. Anaturalresponse benchmarksforatwo-stagedesign,andthepresent inhydrologyistheoccurrence-and-amountmodel,in studyincludesboth.
whichafirststagepredictswhetherawetperiodoccurs Forecastverificationhasamaturemethodology.Prob- andasecondstagepredictsitsamount(SternandCoe, abilisticoccurrenceforecastsarescoredwiththeBrier 1984;WoolhiserandRoldán,1982).Evaluation,how- score(Brier,1950),whichcanbedecomposedintoreli- ever,usuallyreliesonMAEandRMSEoverallmonths, abilityandresolution(Murphy, 1973), andwithskill whicharedominatedbytheseasonalcycleandsay scores that measure improvement over a reference littleaboutthequestionsafarmerasks:willthemon- forecastsuchasclimatology(Murphy,1992;Murphy soonbelate,willadroughtmonthoccur,andwilla andWinkler,1987;Wilks,2011). Properscoringrules flood-scalemonthoccur?
andprobabilisticforecastingarereviewedbyGneiting Thispapermakesthreecontributions. Itspecifiesa andRaftery(2007)andGneitingandKatzfuss(2014), decision-focusedevaluationprotocolforzero-inflated andpracticalguidanceforweatherandclimateveri- monthlyrainfallthatcombineslevelaccuracy,proba- ficationisgivenbyJolliffeandStephenson(2012). A bilisticoccurrenceskillagainstclimatology(theBrier keylessonofthatliteratureisthatclimatologyisthe skill score and calibration), event-based metrics for minimumstandardofreference: aforecastthatcan- droughts, floods and monsoon onset, and stability notbeatthelong-termaverageforthecalendarmonth across 25-year blocks. It applies the protocol to addsnoinformation.Asecondlessonisthatskillmea- a 124-year series with a transparent, reproducible, suresshouldbetiedtouserdecisions,sothatevent- expanding-windowdesign,andcomparesatwo-stage basedmetricssuchashitratesandfalse-alarmratios boostedmodelwithninebenchmarkswhosehyperpa- complementaggregateerrors.
rametersarelistedinfull.Anditreportstheresulthon- FortheIndianmonsoon,forecastingisamaturebut estly,whichisthatonthisseriesnothingbeatsmonthly difficultfield.Seasonalmonsoonrainfallispartlypre- climatology,forreasonsthatthedataexplain.Theneg- dictablefromlarge-scalepredictorssuchassea-surface ativeresultisinformativebecauseitidentifieswhata temperature,butoperationalmodelshavehadnotable rainfallforecastermustaddtoahistory-onlymodel, failures(Gadgiletal.,2005),andstatisticallong-range andbecausetheprotocolandbenchmarksprovidea forecastmodelsrelyonexternalpredictors(Rajeevan standardagainstwhichricherpredictorscanbeeval- etal.,2007).Predictabilityatmonthlyleadtimesfrom uated.ThestudyusestheHaryana-Chandigarh-Delhi therainfallrecordaloneislimitedbythechaoticnature series;dataforotherstatesareavailabletotheauthor oftheatmosphere(Lorenz,1963).Studiesoflong-term andarethenaturalextension.
monsoon change report mixed and sometimes con- Theremainderofthepaperreviewstheliterature(Sec- tradictoryresults,withsomeattributingaweakening tion 2), describes the data (Section 3), sets out the ofthemonsoontoanthropogenicaerosols(Bollasina etal.,2011)andothersemphasisinginternalvariabil- mostvariabletransitionmonth,withameanof49.7 ityandthedifficultyofdetectingtrends(Turnerand mm,astandarddeviationof34.5mmandrainfallbelow Annamalai,2012).Trendtestsforhydrologicalseries 5mmin1%ofyears.
commonlyusetheMann–KendalltestandSen’sslope estimator(Mann,1945;Sen,1968).
Forstatisticalcomparisonofforecasts, theDiebold– Marianotestisstandard(DieboldandMariano,1995; Harvey et al., 1997) with heteroskedasticity and autocorrelation-consistentvarianceestimates(Newey andWest,1987),anduncertaintyinverificationmea- suresiscommonlyassessedbyblockbootstrap(Kün- sch, 1989; Politis and Romano, 1992). Gradient- boosted trees (Friedman, 2001), random forests
calendarmonth.Right:June–Septembertotalrain- andKennard,1970)arestandardlearnersfortabular fallbyyear(black)andmeansfor25-yearblocks features, implemented in widely used software (Pe- (red).
dregosaetal.,2011).Thegapaddressedhereisatrans- Twofeaturesoftheseriesshapeeverythingthatfol- parent evaluation of a two-stage rainfall forecaster lows.First,calendarmonthexplains63.3%ofthevari- againstafullsetofbenchmarks,withdecision-focused anceofmonthlyrainfall, soaforecasterthatknows andstabilityanalyses,overalongrecord. onlythemonthalreadyaccountsformostofthevaria- tion.Second,theanomaliesthatremainafterremov- 3.DataDescription ingthemonthlymeanarealmostuncorrelatedthrough time:theautocorrelationis0.035atlag1,0.007atlag The data are average monthly rainfall (mm) for the 2,0.005atlag3and-0.001atlag12.Thereisthuslittle Haryana-Chandigarh-Delhiregion,suppliedasasin- persistenceonwhichahistory-onlyforecastercould glecolumnof1,488valueswithmonthlabelsofthe build.Thebriefforthisstudyanticipatedthatroughly form“Jan-01”.Thetwo-digityearlabelsrepeat(thela- halfofallmonthswouldhavenear-zerorainfalland bel“Jan-01”appearsatthefirstrowandatrow1,201), thatJunetoSeptemberwouldaccountfor70-80%of sothedateswerereconstructedfromtheroworder, annualrainfall;thefirstisnotborneoutbythedata beginninginJanuary1901andendinginDecember (about18%ofmonthsarebelow1mmand33%below 2024,andthereconstructionwascheckedagainstthe 5mm),andthesecondisslightlyunderstated(82%for labels: themonthandthetwo-digityearofall1,488 JunetoSeptember).
rowsagreewiththereconstructeddate(labelsrun01, 02, ..., 99, 00, 01, ..., 24). Therearenomissingval- uesandnonegativevalues.Thedatasourceandthe 4.Methodology methodbywhichtheregionalaveragewasconstructed Thetaskistoforecastrainfallonemonthahead. At arenotdocumentedinthefileandshouldbespecified eachforecastorigint(theendofmontht), amodel beforesubmission.
forecastsy(t+1).Evaluationusesanexpandingwindow. Theinitialtrainingsampleis1901–1925(25years),and Table1summarisestheseries.Meanmonthlyrainfall forecastsaremadeforeverymonthfromJanuary1926 is44.6mmandmeanannualrainfall534.8mm. The toDecember2024(1,188forecastsover99years).Mod- distributionisstronglyright-skewed(skewness2.10), els are re-estimated once a year, at the end of each withamedianof14.1mmandamaximumof404.7 December,usingonlyrowswhosetargetmonthhad mm(August1995),followedby398.2mm(August1908). beenobservedbythatdate,andtheresultingmodelis Theextentofzeroinflationismoremoderatethanoften usedforthenext12forecasts;thismimicsaforecaster assumed: 7.0%ofmonthshaveexactlyzerorainfall, whoupdatesthemodelannually.The25-yearburn-in 17.8%havelessthan1mm,and32.5%havelessthan5 meansthatthefirst25-yearblock,1901-1925,isused mm,thewet–drythresholdusedbelow.Themonsoon fortrainingonlyandcannotbeevaluatedoutofsam- dominates:JunetoSeptemberaccountfor82.3%of ple;stabilityisthereforeassessedoverthefourlater annualrainfallandJulytoSeptemberfor73.0%.Table blocks(1926-1950,1951-1975,1976-2000,2001-2024). 2givesthemonthlyclimatology.JulyandAugustare thewettestmonths(meansof153.5and148.7mm) Theproposedtwo-stageforecasterdecomposesthe andalmostneverdry,Novemberisthedriest(mean forecast into occurrence and amount. Stage 1 esti- 3.3mm,below5mmin81%ofyears),andJuneisthe matestheprobabilityp(t+1)=P(y(t+1)>5mm)with Statistic Value Monthlyobservations 1,488 Meanmonthlyrainfall(mm) 44.6 Standarddeviation(mm) 66.3 Median(mm) 14.1 Maximum(mm) 404.7 Skewness 2.10 Monthswithrainfallexactly0mm 7.0% Monthswithrainfall<1mm 17.8% Monthswithrainfall<5mm(drybythe5mmrule) 32.5% Meanannualrainfall(mm) 534.8 ShareofannualrainfallinJun–Sep 82.3% ShareofannualrainfallinJul–Sep 73.0% Month-of-yearshareofvariance(R²) 0.634 Autocorrelationofmonthlyanomalies, 0.035/0.007/0.005 lag1/lag2/lag3 Note.Anomaliesaredeviationsfromthecalendar-monthmean.R2 =shareofthevarianceofmonthlyrainfallexplainedby calendarmonth.
Month Mean(mm) SD(mm) Share<5mm Share<1mm Maximum(mm) Jan 17.4 16.1 0.30 0.13 70.3 Feb 17.1 19.1 0.32 0.16 91.2 Mar 13.5 16.1 0.39 0.19 71.6 Apr 7.6 12.6 0.60 0.27 82.3 May 15.6 16.4 0.30 0.06 73.6 Jun 49.7 34.5 0.01 0.00 200.0 Jul 153.5 70.0 0.00 0.00 369.4 Aug 148.7 77.5 0.00 0.00 404.7 Sep 88.4 73.2 0.02 0.01 361.8 Oct 12.7 23.6 0.56 0.33 132.1 Nov 3.3 7.2 0.81 0.60 43.4 Dec 7.2 10.7 0.58 0.40 68.9 Note.Computedover1901–2024(124valuespermonth a gradient-boosted classifier. Stage 2 estimates the environment.
conditionalmeanamountm(t+1)=E[y(t+1)|y(t+1)>5 Table 3 lists the benchmark models and, for every mm]withagradient-boostedregressortrainedonlyon model,thehyperparameters.Thebenchmarksareof monthswithrainfallabove5mm.Thepointforecastis fourkinds:naiverules(persistence,seasonalnaiveand ˆy(t+1)=p(t+1) × m(t+1).Bothstagesusethesame16 monthlyclimatology,thelastwithanexpandingwin- predictors: themonthofthetarget(anintegerindex dow);single-stagelearnerstrainedonallmonths(ridge treatedascategoricalbythetreemodels,andsineand regression,aTweediegeneralisedlinearmodel,aran- cosineterms),rainfallofthecurrentmonthandofthe domforest,andboostingwithsquared-errorandwith twoprecedingmonths,rainfallofthesamecalendar Poissonloss);atwo-stagelinearmodel(logisticregres- monthayearearlier,rollingmeansandstandardde- sionforoccurrenceandagammageneralisedlinear viationsoverthelatest3,6and12months,thetotal modelforwet-monthamounts);andtheproposedtwo- rainfallofthemostrecentcompletedJune-September stageboostedmodel.Hyperparameterswerefixedin season,theexpanding-windowclimatologicalmean advanceandwerenottunedonthetestdata,sothat ofthetargetmonthcomputedfromdataavailableat thecomparisoncannotbeaffectedbyselectiononthe theorigin,andayearindexscaledto0-1forthecen- evaluationsample; theconsequencesofthischoice tury. Theimplementationusesthehistogram-based arediscussedinSection6. Randomseedsarefixed gradientboostingestimatorsinscikit-learn(Pedregosa at 0. For the occurrence forecasts, the probabilistic etal.,2011),whichbelongtothesamefamilyasLight competitorsaretheclimatologicaloccurrenceratefor GBM(Keetal.,2017)butarenotLightGBMitself,asthe the month, the seasonal-naive indicator, logistic re- LightGBMlibrarywasnotavailableinthecomputing gression,arandom-forestclassifierandtheboosted classifier.
yearswithblocksofthreeyearsand2,000resamples (Kunsch,1989;PolitisandRomano,1992),recomput- Accuracy is evaluated at four levels. Level accuracy ingeachmetricfromyear-levelsums;theintervalsare isMAEandRMSEonallmonths,onmonsoonmonths 95%percentileintervals.Nomultiplicitycorrectionis (JunetoSeptember)andonwetmonths(rainfallabove applied,andtestsaredescriptive.
5mm),togetherwithMAErelativetoclimatologyand amean-squared-errorskillscore,1-MSE(model)/MSE (climatology),computedonJunetoSeptember;posi- 5.Results tivevaluesmeanthemodelbeatsclimatology.Occur- Table4andFigure2givethelevel-accuracycompari- renceaccuracyistheBrierscoreforP(y>5mm),the son.Monthlyclimatologyisthebestforecasteronev- Brierskillscoreagainsttheclimatologicaloccurrence erymeasure.OverallmonthsitsMAEis23.65mmand rate(BSS,positiveisbetter)forallmonths,forOctober itsRMSE40.05mm;forthemonsoonmonthsitsRMSE toMayandforJunetoSeptember,andtheexpected is65.78mm.Noothermodelimprovesonthesevalues. calibrationerror(ECE)overtenequal-widthprobability Thelearnersarewithin1%to7%ofclimatologyinMAE bins,withreliabilitydiagrams.
andbetween1%and15%worseinmonsoon-month meansquarederror:ridgeregression(MAEratio1.029), Decision metrics are defined on the 396 June to theTweedieGLM(1.028), therandomforest(1.056), Septembertargetmonths.Adroughtmonthisamon- Poissonboosting(1.051),single-stageboostedregres- soon month with rainfall below 20 mm (37 events), sion(1.070), thetwo-stagelinearmodel(1.010)and a flood month one with rainfall above 200 mm (52 theproposedtwo-stageboostedmodel(1.073). Per- events).Aforecastoftheamountisturnedintoawarn- sistenceandseasonalnaiveforecastsaremuchworse ingbyafixed-alarm-raterule,becauseforecaststhat (MAEratiosof1.78and1.34),becausetheycarrylast areconditionalmeansrarelyfallbelow20mmorex- monthsorlastyear’snoiseratherthananaverage.The ceed200mminamonsoonmonth:adroughtwarning two-stageboostedmodelhasaJune-SeptemberMSE isissuedforthe20%ofmonsoonmonthswiththelow- skillof-0.152,theworstamongthelearners,andthe estforecasts,andafloodwarningforthe15%withthe bestofthem,ridgeregression,has-0.025.Thesecon- highest.Droughtrecallistheshareofdroughtmonths clusionsarethesameforwet-monthaccuracy(Table4, thatwerewarnedabout,floodprecisiontheshareof columnsforwetmonths),sothezero-inflatedstructure floodwarningsthatwerefollowedbyafloodmonth, ofthedataisnotdrivingthem.
andtheareaundertheROCcurve(AUC)oftheforecast foreacheventisreportedasathreshold-freesummary. Theliteralthresholds(forecastabove200mm, fore- castbelow20mm)arealsoreportedasthenumberof monthsinwhichtheyaretriggered.Onsetisdefined foreachofthe99testyearsasthefirstmonthfrom JunetoSeptemberwithrainfallabove50mm;thepre- dictedonsetisthefirstmonthinwhichtheone-month- aheadforecastexceeds50mm.Onsetisevaluatedby theexactmonth,bybeingwithinonemonth,andby classificationofalateonset(Junerainfallatorbelow
erroroverJune–September,relativetoclimatology Stabilityisassessedbycomputingthemainmetrics (valuesabove1areworsethanclimatology). separatelyforeach25-yearblockandbyMann-Kendall Theblock-bootstrapintervalsinTable5showhowde- trendtestswithSenslopesformonsoonandmonthly cisivetheorderingis.TheMAEratioofthetwo-stage rainfall,togetherwithLeveneandKruskal-Wallistests boostedmodelhasa95%intervalof1.031to1.110, acrossblocks. StatisticalinferenceusestheDiebold- whichexcludesone;sodotheintervalsofthesingle- Mariano test of equal predictive accuracy with the stageboostedmodel(1.034to1.106)andtherandom small-samplecorrectionofHarveyetal. (1997)and forest(1.030to1.081).TheonlylearnerwhoseMAEin- Bartlett-kernelvariances(lag6forallmonthsandlag3 tervalincludesoneisthetwo-stagelinearmodel(0.989 forthemonsoonsubset;NeweyandWest,1987),ap- to1.031),whichisthereforenotdistinguishablefrom pliedtoabsoluteandsquarederrors,withapositive climatologyinabsoluteerror,althoughitsmonsoon statisticmeaningthatthefirstforecastinapairhas MSEskillissignificantlynegative(-0.076,interval-0.130 thelargerloss. Uncertaintyinthemainmetricsisas- to-0.022).TheDiebold–MarianotestsinTable6agree: sessedbyacircularblockbootstrapoverthe99test thetwo-stageboostedmodelhasasignificantlylarger Model Role Specificationandhyperparameters Forecast=rainfallofthecurrentmonth, Persistence Benchmark ˆy(t+1)=y(t).Noparameters.
Forecast=rainfallofthesame Seasonalnaive Benchmark calendarmonthoneyearearlier, y(t+1−12).Noparameters.
Expanding-windowmeanofthe targetcalendarmonthusingonly Climatology Benchmark datauptotheforecastorigin; occurrenceprobability=pastshareofmonths above5mm.
L2penalty α =10;standardisedfeatures Benchmark Ridgeregression with12monthindicatorsinplaceofthe (single-stage,linear) monthindex;predictionstruncatedat0.
Power1.5(compoundPoisson–gamma), Benchmark loglink,L2penalty α =0.1,upto TweedieGLM (single-stage, 2,000iterations,standardisedfeatures zero-inflatedaware) withmonthindicators.
200trees,minimumleafsize5, Benchmark Randomforest 50%offeaturespersplit,bootstrapsampling, (single-stage,trees) seed0;trainedonallmonthswithsquarederror. Histogramgradient-boostedregressiontrees (scikit-learn):200iterations,learningrate0.05, Benchmark maximumdepth3,minimumleaf15, HGBsingle-stage (single-stage,boosting) L2penalty1.0,255bins,monthindexas categoricalfeature,squared-errorloss, seed0;trainedonallmonths.
AsHGBsingle-stagebutwith Benchmark HGBPoisson Poissonloss(non-negative,suitedto (single-stage,boosting) zero-heavydata).
Occurrence:logisticregression, L2penaltyC=1.0,upto1,000iterations.
Benchmark Two-stagelinear Amount:gammaGLMwithloglink, (two-stage) α =0.1,trainedonwetmonthsonly.
Stage1occurrence: HGBclassifier,log-loss, 200iterations,learningrate0.05, depth4,minimumleaf20,L21.0, monthcategorical.Stage2amount: Two-stageHGB Proposed HGBregressortrainedonlyon monthswithrainfallabove5mm, 200iterations,learningrate0.05,depth3, minimumleaf15,L21.0.
Climatologicalrate;seasonal-naiveindicator; Occurrenceclassifiers Probabilistic logisticregression(asabove);random-forestclassifier (Brier) benchmarks (200trees,minimumleaf5,50%offeatures); HGBclassifier(asStage1).
Note.Alllearnersusethe16predictorsdescribedinthetext.Linearmodelsusestandardisedpredictorsand12months indicatorsinsteadofthemonthindex.HGB=histogramgradientboosting.Alllearnersarere-estimatedeveryDecemberon datarealisedbythen.
lossthanclimatologyforallmonthsandforthemon- decomposition neither helps nor hurts relative to a soonmonthsunderbothabsoluteandsquarederror single-stagemodelofthesametype. Itdoessignifi- (statisticsof4.25,3.50,3.27and3.28,allp<.001),and cantlyworsethanthetwo-stagelinearmodel(statistic asignificantlysmallerlossthanseasonalnaive(statis- 3.73,p<0.001,inabsoluteerror)thantherandomfor- ticsfrom-5.77to-6.65). Importantly, thetwo-stage estinsquarederror(2.89,p=0.004).
boostedmodelisnotdifferentfromsingle-stageboost- ing(statistics0.22forabsoluteand1.00forsquared Occurrenceforecaststellthesamestory(Table7,Fig- erroroverallmonths,p=0.83and.32;0.32and0.92 ure3).TheBrierscoreoftheclimatologicaloccurrence forthemonsoon,p=0.75and0.36),sothetwo-stage rateis0.152overallmonthsand0.224forOctoberto May, andnootherforecasterbeatsit: theBrierskill MAE RMSE MAE RMSE MAE RMSE MAErel. Jun–Sep Model all all Jun–Sep Jun–Sep wet wet toclim. MSEskill Persistence 42.14 70.57 82.82 107.56 52.17 79.79 1.782 -1.673 Seasonal 31.63 55.24 68.77 91.20 42.47 66.17 1.337 -0.922 naive Climatology 23.65 40.05 50.12 65.78 30.54 48.00 1.000 0.000 Ridge 24.34 40.69 50.97 66.60 31.27 48.60 1.029 -0.025 Tweedie 24.31 41.50 52.11 68.36 31.61 49.75 1.028 -0.080 GLM Random 24.98 41.15 52.55 67.79 31.80 49.10 1.056 -0.062 forest HGB 25.31 42.50 53.66 69.94 32.72 50.90 1.070 -0.130 single-stage HGB 24.84 42.43 53.20 70.06 32.37 50.89 1.051 -0.134 Poisson Two-stage 23.88 41.44 51.21 68.24 31.29 49.77 1.010 -0.076 linear Two-stage 25.37 42.93 53.85 70.62 33.11 51.50 1.073 -0.152 HGB Note.Allmonthsn=1,188;Jun–Sepn=396;wetmonths=monthswithrainfallabove5mm.MAErel.toclim.=MAEoverall monthsdividedbythatofclimatology.MSEskill=1-MSE/MSEofclimatologyoverJun–Sep(positiveisbetter).
MAEall MAErelativetoclimatology Jun–SepMSEskill Model (95%CI) (95%CI) (95%CI) Climatology 23.65[22.51,24.76] 1.000[1.000,1.000] 0.000[0.000,0.000] Seasonalnaive 31.63[29.34,33.93] 1.337[1.266,1.414] -0.922[-1.189,-0.677] Randomforest 24.98[23.61,26.20] 1.056[1.030,1.081] -0.062[-0.116,-0.010] HGBsingle-stage 25.31[23.77,26.74] 1.070[1.034,1.106] -0.130[-0.214,-0.037] Two-stagelinear 23.88[22.56,25.13] 1.010[0.989,1.031] -0.076[-0.130,-0.022] Two-stageHGB 25.37[23.86,26.73] 1.073[1.031,1.110] -0.152[-0.243,-0.055] Note.Circularblockbootstrapover99testyears(3-yearblocks,2,000resamples).Valuesinbracketsare95%percentile intervals.
scoresforOctobertoMayare-0.024forlogisticregres- sion(interval-0.038to-0.009),-0.056fortherandom forest(-0.085to-0.027)and-0.138fortheboostedclas- sifier (-0.188 to -0.089), and -0.877 for the seasonal- naiveindicator.Thereliabilitydiagramshowswhy.Cli- matologyiswellcalibratedbyconstruction(ECE0.028), logisticregressionandtherandomforestareslightly over-dispersed(ECE0.046and0.034),andtheboosted classifier,whichfitsthetrainingdatamoreclosely,is theleastwellcalibratedofthelearnedmodels(ECE
anddistributionofforecastprobabilities(right,log than the observed frequencies justify. For June to scale).
September,rainabove5mmoccursinabout99%of months,sotheBrierscoresaretiny(0.008)andskill Decision-focusedmetrics(Table8)donotreversethe scorestherearenotinformative.
ranking. Among the 396 monsoon months, 37 are droughtmonths(rainfallbelow20mm),ofwhich19 fallinJune,16inSeptemberandonlytwoinJuly;the droughteventsarethereforelateonsetsandearlywith- drawalsofthemonsoonratherthanmid-seasonfail- ures.Witha20%alarmrate,droughtrecallis0.405for thetwo-stageboostedmodel,0.405forthesingle-stage boostedmodel,0.459fortherandomforest,0.378for Allmonths, Allmonths, Jun–Sep, Jun–Sep, Comparison(firstvssecond) abs.error sq.error abs.error sq.error Two-stageHGBvsClimatology 4.25(<.001) 3.50(<.001) 3.27(.001) 3.28(.001) Two-stageHGBvsHGBsingle-stage 0.22(.827) 1.00(.320) 0.32(.750) 0.92(.356) Two-stageHGBvsRandomforest 1.19(.234) 2.89(.004) 1.40(.161) 2.56(.011) Two-stageHGBvsSeasonalnaive -6.65(<.001) -5.94(<.001) -5.81(<.001) -5.77(<.001) Two-stageHGBvsTwo-stagelinear 3.73(<.001) 1.90(.058) 2.26(.024) 1.62(.106) HGBsingle-stagevsClimatology 4.18(<.001) 3.27(.001) 3.22(.001) 3.10(.002) RandomforestvsClimatology 4.81(<.001) 2.30(.021) 3.29(.001) 2.36(.019) Two-stagelinearvsClimatology 0.97(.332) 3.05(.002) 1.56(.119) 2.96(.003) RidgevsClimatology 4.72(<.001) 3.27(.001) 2.69(.007) 2.37(.018) Note.Positivestatisticsindicatethatthefirstforecasthasthelargerloss.Lag6forallmonths,lag3forJune–September.
Brier, BSS, BSS, BSS, Calibrationerror Occurrenceforecaster allmonths allmonths Oct–May(95%CI) Jun–Sep (ECE) Climatologicalrate 0.152 0.000 0(reference) 0.000 0.028 -0.877 Seasonalnaive 0.285 -0.879 -0.994 0.285 [-1.007,-0.755] -0.024 Logistic 0.156 -0.024 -0.022 0.046 [-0.038,-0.009] -0.056 Randomforest 0.160 -0.056 -0.078 0.034 [-0.085,-0.027] -0.138 HGBclassifier 0.173 -0.137 -0.062 0.072 [-0.188,-0.089] Note.BSS=Brierskillscorerelativetotheclimatologicaloccurrencerate;valuesbelowzeromeanworsethanclimatology. ECE=expectedcalibrationerrorover10probabilitybins.Interval=block-bootstrap95%interval. thetwo-stagelinearmodeland0.324forclimatology. Onsetresults(Table9)showthatthetesthaslittledis- Thesedifferencesaresmallrelativetosamplinguncer- criminatingpowerinthisseries. In42ofthe99test tainty:thebootstrapintervalforthetwo-stageboosted yearsthemonsoonstartedinJune(Junerainfallabove modelis0.237to0.576,andthatofclimatologyis0.179 50mm),in54inJulyandinthreeinAugust;thereareno to0.484,sothedatacannotdistinguishthem.TheAUC yearswithoutonset.Beingwithinonemonthisalmost fordroughtstellsamorecoherentstory,withthehigh- guaranteed,witheverymodelat96-100%,becausethe estvaluesforthetwo-stagelinearmodel(0.735),ridge onsetisalwaysJune,JulyorAugust.Theexactonset regression(0.727)andclimatology(0.718),andthetwo- monthispredictedcorrectlyin47.5%ofyearsbythe stageboostedmodelat0.709.Forfloods(52events:28 two-stageboostedmodel,51.5%byclimatologyand inJuly,20inAugust,fourinSeptember),floodpreci- 53.5%bythetwo-stagelinearmodel,butalwayspre- sionata15%alarmrateliesbetween0.150and0.250 dictingJulywouldbecorrectin54.5%ofyears. For forallmodels(0.217forthetwo-stageboostedmodel theclassificationofalateonset(Junerainfallatorbe- and0.217forclimatology,againstabaserateof13.1%), low50mm,57.6%ofyears),accuracyis48.5%forthe sofloodwarningsfromanyofthesemodelsareonly two-stageboostedmodel,53.5%forclimatologyand slightlybetterthanrandomalarms;theAUCsare0.757 55.6%forthetwo-stagelinearmodel,against57.6% (randomforest),0.747(climatology)and0.710(two- foralwayspredictingalateonset.Persistence,which stage boosted). The literal thresholds are informa- predictsalateonsetinmostyears, attainsthehigh- tiveontheirown: climatologyandthetwo-stagelin- est late-onsetF1 (0.720)only by predicting lateness earmodelneverforecastmorethan200mm,andthe almostalways.Nomodelcananticipatethetimingof boostedmodelsdosoin5to16of396months,while theonsetfromtheprecedingmonths’rainfall. forecastsbelow20mmoccurinonlytwomonthsfor thetwo-stageboostedmodel. Adecisionrulebased As it is seen the models fail to beat climatology, Ta- onabsolutethresholdsappliedtoconditional-mean ble10givesdiagnostics.Theanomalycorrelationbe- forecastswouldthereforeissuealmostnowarnings, tweenforecastandobserveddeparturesfromclimatol- whichsupportstheuseofrank-basedorprobabilistic ogyisbetween-0.08and0.05forallmodelsoverJune warnings.
toSeptember,withthetwo-stageboostedmodelat -0.006,soforecastscontainessentiallynoinformation Drought Drought Drought Flood Flood Flood Monthsforecast> Monthsforecast< Model recall precision AUC precision recall AUC 200mm 20mm Persistence 0.405 0.188 0.614 0.150 0.173 0.538 48 89 Seasonalnaive 0.378 0.175 0.648 0.250 0.288 0.686 53 38 Climatology 0.324 0.150 0.718 0.217 0.250 0.747 0 0 Ridge 0.324 0.150 0.727 0.250 0.288 0.739 0 0 Randomforest 0.459 0.212 0.722 0.233 0.269 0.757 2 0 HGBsingle-stage 0.405 0.188 0.705 0.217 0.250 0.722 5 2 HGBPoisson 0.351 0.163 0.708 0.250 0.288 0.736 16 0 Two-stagelinear 0.378 0.175 0.735 0.200 0.231 0.730 0 0 Two-stageHGB 0.405 0.188 0.709 0.217 0.250 0.710 8 2 Note.Drought=Jun–Seprainfallbelow20mm(37events);flood=above200mm(52events).Warnings:lowest20%of forecasts(drought)andhighest15%(flood)among396Jun–Sepmonths.AUC=areaundertheROCcurvefortheforecastasa score.Thelasttwocolumnscountmonthsinwhichtheliteralthresholdsaretriggered.
Model Droughtrecall(95%CI) Floodprecision(95%CI) Climatology 0.324[0.179,0.484] 0.217[0.125,0.299] Seasonalnaive 0.378[0.222,0.544] 0.250[0.148,0.348] Randomforest 0.459[0.289,0.628] 0.233[0.123,0.339] HGBsingle-stage 0.405[0.243,0.571] 0.217[0.122,0.302] Two-stagelinear 0.378[0.226,0.526] 0.200[0.097,0.292] Two-stageHGB 0.405[0.237,0.576] 0.217[0.120,0.316] Note.Block-bootstrap95%intervals(3-yearblocks,2,000resamples). beyondthemonthlymean. Theregressionslopeof >0.80),orJunerainfall(τ =0.09,p=0.16).Themean observedonforecastanomaliesisnearzeroornega- June-Septembertotalwas440mmin1901-1925,423 tiveformostmodels,andinfloodmonths(observed mmin1926-1950,479mmin1951-1975,464mmin mean261.3mm)theforecastsaverage134-157mm 1976-2000and394mmin2001-2024;thelastblockis forthelearners,sothepeaksarenotanticipatedby thelowestandhasthesmallestdispersion,butthevari- anymodel. Theunderlyingreasonisthatthewithin- ationamongblocksisnotsignificant(Kruskal-Wallisp monthvariationisnotpredictablefrompastrainfall: =0.12;Levenetestforequalvariancesp=0.26).Model thelaggedautocorrelationsofanomaliesarecloseto skilldoesnotchangesystematicallyacrossblocks.The zero(Table1).Aflexiblelearnertrainedonsuchdata two-stageboostedmodelhasaJune-SeptemberMSE fitsnoise,andtheresultingforecastsareworsethan skill of -0.139, -0.131, -0.266 and -0.062 in the four thestableclimatologicalmeanbytheamountofthe blocks,andsingle-stageboosting-0.037,-0.153,-0.246 estimationerror.
and-0.079;thetwomodelsdonotdiffersignificantlyin anyblock(Diebold-Marianopfrom0.28to0.57).Both aresignificantlyworsethanclimatologyinabsolute errorinthemiddleblocks(p ≤ 0.05for1926-1950and p<0.001for1951-1975and1976-2000),andindistin- guishablefromitin2001-2024(p=0.70and0.65),the blockthathastheleasttrainingdatarelativetovari- ance. Brierskillscoresfortheboostedclassifierare negativeineveryblock(-0.251,-0.098,-0.101,-0.103). Droughtandfloodmetricsbyblockrestonveryfew events(fourto15droughtmonthsandfiveto17flood
June–September for two-stage and single-stage ratherthanevidenceofchange.
boosting. Thecloudisflat,soforecastanomalies carrynoinformationaboutobservedanomalies. StabilityacrosstherecordisshowninTables11-14and Figure5.Themonsoon(JunetoSeptember)totalhas nosignificanttrend(Kendall τ =-0.02,p=0.74; Sen slope-0.12mmperyear),nordoesannualrainfall(τ = -0.01,p=0.83),July,AugustorSeptemberrainfall(allp Model Exactonsetmonth Withinonemonth Late-onsetaccuracy Late-onsetF1 Late-onsetrecall Persistence 0.030 0.980 0.576 0.720 0.947 Seasonalnaive 0.444 0.960 0.465 0.531 0.526 Climatology 0.515 0.990 0.535 0.685 0.877 Ridge 0.434 1.000 0.465 0.576 0.632 Randomforest 0.465 0.980 0.475 0.500 0.456 HGB 0.485 0.970 0.485 0.541 0.526 single-stage HGBPoisson 0.525 0.970 0.525 0.584 0.579 Two-stagelinear 0.535 0.990 0.556 0.699 0.895 Two-stageHGB 0.475 0.980 0.485 0.532 0.509 Note.Onset=firstJun–Sepmonthwithrainfallabove50mm;predictedonset=firstmonthwithforecastabove50mm.Late onset=Junerainfallatorbelow50mm(57.6%of99testyears).AlwayspredictingJulygivesanexact-monthaccuracyof 54.5%;alwayspredictingalateonsetgivesanaccuracyof57.6%.
Meanforecastin Regressionslopeof Anomalycorrelation, Anomalycorrelation, floodmonths Model observedonforecast allmonths Jun–Sep (mm;observed anomaly,Jun–Sep mean261.3) Persistence -0.025 -0.032 -0.024 95.8 Seasonalnaive 0.040 0.031 0.031 156.6 Ridge -0.053 -0.073 -0.729 144.7 TweedieGLM -0.051 -0.055 -0.230 141.3 Randomforest 0.006 -0.016 -0.069 148.4 HGBsingle-stage 0.012 -0.003 -0.006 141.7 HGBPoisson 0.047 0.042 0.103 140.5 Two-stagelinear -0.066 -0.079 -0.311 134.5 Two-stageHGB 0.004 -0.006 -0.014 140.1 Note.Anomaly=deviationfromtheexpanding-windowclimatologicalmeanofthetargetmonth.Slope=coefficientfrom regressingobservedanomaliesonforecastanomaliesinJun–Sep(1wouldindicatecalibratedanomalyforecasts). becausetheycollapsetowardzero.Inthepresentdata thesingle-stagemodelsdonotcollapse:theytrackthe seasonalcyclewell,becausethecalendarmonthex- plains63%ofthevarianceandtheclimatologicalmean ofthetargetmonthisincludedamongthepredictors. Whattheycannotdoispredictdeparturesfromthe seasonalcycle,sincethedeparturesarealmostserially uncorrelated. Thefailureisthereforenotoftheloss functionbutoftheinformationset.
Figure5.Stabilityacross25-yearblocks.Left:Jun– Thischangeshowtheevaluationprotocolshouldbe SepMSEskillagainstclimatology.Right:Brierskill read.Decision-focusedmetricswereexpectedtorank scoreforoccurrenceinOctober–May.Allvaluesare models differently from aggregate metrics, and the atorbelowzero,withnosystematictrendacross questionwaswhetherthemodelwiththebestRMSEis blocks.
themodelwiththebestdroughtrecall.Inthedata,the rankingisnearlyidentical,becauseallmodelshavelit- 6.Discussion tleskill,anddifferencesinrecallorprecisionarewithin Thecentralfindingisthatonthisseriesnothingbeats samplingnoise.Thisisnotevidenceagainstdecision- climatologyataone-monthhorizon,neitherintheac- focusedevaluation.Itisareminderthatsuchmetrics curacyofamountsnorintheprobabilityofoccurrence, restonfewevents(37droughtand52floodmonths andthatthetwo-stagestructuredoesnotchangethis. in99years),havewidebootstrapintervals,andcan- Thisresultcontradictstheworkingexpectationthata notrescueaforecasterwithoutsignal. Theprotocol two-stagemodelwouldcaptureoccurrenceinforma- didshowsomethinguseful:itidentifiedthatdrought tionthatasingle-stagemodeldiscards,anditalsocon- monthsinthisregionarelateonsets(June)andearly tradictstheassumptionthatsingle-stagemodelsfail withdrawals(September)ratherthanmid-seasonfail- Block Years Jun–Sepmean(mm) SD(mm) CV 1901-1925 25 440.1 138.3 0.31 1926-1950 25 422.8 134.0 0.32 1951-1975 25 479.5 112.7 0.24 1976-2000 25 463.9 158.0 0.34 2001-2024 25 393.6 107.0 0.27 Note.Kruskal-Wallistestacrossthefiveblocks,p=.12;Levenetestforequalvariances,p=0.26.CV=coefficientofvariation.
Senslope Series Kendall τ p-value (mmperyear) Annualtotal -0.013 .829 -0.087 JJAStotal -0.020 .736 -0.117 June 0.086 .157 0.107 July -0.011 .859 -0.030 August -0.015 .809 -0.043 September 0.004 .948 0.006 Occurrence(>5mm)share,Oct–May 0.020 .761 0.000 Note.AnnualandJun–Septotalsareyearlyseries(124values);monthlyseriesusethe124valuesforthatmonth;thelastrow istheyearlyshareofOct–Maymonthsabove5mm. ures, that conditional-mean forecasts almost never thantuned,andatunedmodel,particularlyonewith crossabsoluteeventthresholds,sothatrank-based strongerregularisation,mightapproachclimatology orprobabilisticwarningsareneeded,andthatonset moreclosely,thoughtheevidencesuggestsitwouldbe timinghasabaseratethatnomodelbeats.
unlikelytoexceedit.Thetestexcludes1901-1925,and thenumberofdroughtandfloodeventsperblockis Thebenchmarkcomparisonoffersfurtherlessons.The small.Dataqualityintheearlyrecordisuncertain,and best learners are the most heavily regularised ones thewet-drythreshold(5mm)andeventthresholds(20 (ridge regression and the two-stage linear model), mm,200mm,50mm)followthestudydesignrather andtheproposedboostedmodels,whichhavemore thanbeingestimated.Finally,becausethestudyused flexibility,performworse,inlinewiththestatistical- onlythisregion,theresultscannotbegeneralisedto learningviewthatflexiblemodelsneedsignaltojus- otherstates,whoserainfallregimesmaybemoreper- tifytheirvariance(andconsistentwiththesample-size sistentormorepredictable;amulti-statepanelwould analysis in the companion study). The Poisson-loss beanaturalextension,anditwouldalsoprovidemore and Tweedie models, designed for zero-heavy data, eventsfordroughtandfloodevaluation.
performsimilarlytosquared-errormodels,indicating thatthelossfunctionmatterslittlewhenthereisno Thepracticalimplicationforagriculturalplanningis signal.Thetwo-stageboostedclassifieristheworstcal- acautionaryone. Afarmerorplannershouldnotex- ibratedoftheoccurrenceforecasters,suggestingthat pectamodelthatusesonlypastmonthlyrainfallfor probabilitycalibration(forexamplebyPlattscalingor theregiontosayanythingusefulaboutthenextmonth isotonicregression)wouldbeanecessaryadditionto beyondtheclimatologicalexpectation,andanyfore- apracticalsystem,althoughcalibrationcannotcreate castthatclaimsskillfromsuchinputsaloneshouldbe skillthatisnotthere.
evaluatedagainstclimatologywiththemetricsused here.Improvementswillhavetocomefromadditional Severallimitationsapply. Theseriesisaregionalag- information,suchasocean–atmosphereindices,soil gregateforHaryana,ChandigarhandDelhi,notstation- moisture,dailydataorupstreamstations,evaluated leveldata,andthedatasourceandconstructionare withthesameprotocol.
notdocumentedinthefile.Monthlyresolutionhides thedailystructure,suchasactiveandbreakspells,that mattersforcrops.Thehorizonisonemonth,andsea- 7.Conclusion sonalortwo-monthhorizonsmaybehavedifferently, Thispaperspecifiedadecision-focusedevaluationpro- particularlywithpredictorssuchassea-surfacetem- tocolforzero-inflatedmonthlyrainfallandappliedit perature,whichareknowntocarrymonsooninforma- toa124-yearseriesfortheHaryana-Chandigarh-Delhi tion(Gadgiletal.,2005;Rajeevanetal.,2007)butwere region,comparingatwo-stageboostedforecasterwith notavailablehere.Hyperparameterswerefixedrather ninebenchmarksinaone-month-aheadexpanding- Jun–Sep Jun–Sep Droughtrecall Floodprecision Block Model MAEall RMSE skill (events) (alarms) Climatology 67.47 0.000 23.24 0.33(15) 0.00(0) HGBsingle-stage 68.69 -0.037 24.43 0.47(15) 0.13(8) 1926-1950 Two-stagelinear 71.54 -0.124 23.25 0.53(15) 0.50(2) Two-stageHGB 71.99 -0.139 25.24 0.53(15) 0.09(11) Climatology 69.23 0.000 23.91 0.45(11) 0.00(2) HGBsingle-stage 74.35 -0.153 26.61 0.36(11) 0.21(19) 1951-1975 Two-stagelinear 73.46 -0.126 24.77 0.45(11) 0.00(7) Two-stageHGB 73.62 -0.131 26.05 0.27(11) 0.25(16) Climatology 65.25 0.000 23.80 0.25(4) 0.28(40) HGBsingle-stage 72.85 -0.246 26.16 0.25(4) 0.29(24) 1976-2000 Two-stagelinear 66.42 -0.036 23.88 0.25(4) 0.29(35) Two-stageHGB 73.43 -0.266 26.19 0.25(4) 0.29(24) Climatology 60.68 0.000 23.63 0.14(7) 0.11(18) HGBsingle-stage 63.04 -0.079 24.00 0.43(7) 0.11(9) 2001-2024 Two-stagelinear 60.50 0.006 23.62 0.00(7) 0.06(16) Two-stageHGB 62.53 -0.062 23.93 0.43(7) 0.11(9) Note.Skill=1-MSE/MSEofclimatologyoverJun–Sep.Droughtrecall:shareofdroughtmonthswarnedaboutata20%alarm rate(numberofdroughtmonthsinbrackets).Floodprecision:shareoffloodwarningsfollowedbyafloodmonth(numberof warningsinbrackets);warningthresholdsarefixedoverthefulltestperiod.
BSS: BSS: DM:two-stagevs DM:singlevs BSS: DM:two-stagevs Block random HGB climatology climatology logistic single(Jun–Sep) forest classifier (allmonths) (allmonths) 1926-1950 -0.050 -0.106 -0.251 1.08(.281) 1.94(.053) 1.36(.176) 1951-1975 -0.015 -0.024 -0.098 −0.85(.395) 3.09(.002) 3.75(<.001) 1976-2000 -0.025 -0.057 -0.101 0.80(.427) 3.59(<.001) 3.21(.001) 2001-2024 -0.006 -0.037 -0.103 −0.58(.566) 0.39(.696) 0.46(.649) Note.BSSforOct–Mayrelativetoclimatologicaloccurrencerate.DMstatisticsforabsoluteerror(p-valuesinparentheses); positivemeansthefirstmodelisworse.
windowevaluationover1926-2024. Nomodelbeats shouldbedirected.
monthly climatology on level accuracy, occurrence skill, drought recall, flood precision or onset detec-