Easy Sample Weights

[Beerstabr]

Often in forecasting it makes sense to use sample weights that make your model focus more on the recent history. And with most Sklearn models you can introduce this through the fit method. It would be great if Darts could make it easy to implement sensible weighting schemes for forecasting such as an exponentially decaying weighting function.

Many thanks for the library!

[hrzn]

good idea, adding to the backlog :) (and contributions are welcome!)

[Beerstabr]

I would definitely like to contribute!

[Beerstabr]

I was thinking of solving it much like the _create_lagged_data function from RegressionModel class.

Starting out with three options:

• equal weights
• linearly decaying weights
• exponentially decaying weights like
  

[hrzn]

Hi @Beerstabr, after checking, I think it should already be possible to do something like this:

my_model = RegressionModel(..., lags=n_lags)
my_model.fit(..., sample_weight=[1. / (in_len - i) for i in range(n_lags)])

because all the kwargs received by fit() are passed to the underlying estimator's fit() method.

[Beerstabr]

Hi @hrzn, yes that's true. That's how I am currently doing it.

However, it gets slightly more complicated when you start using lags, an output_chunk_length >1 or start training on multiple series (and there's probably other things to consider as well).

For example, when you use 8 lags your series gets cut short by 8 data points. In that case I think it should be:

my_model = RegressionModel(..., lags=n_lags, input_chunk_length=in_len)
my_model.fit(..., sample_weight=[1. / (in_len - i) for i in range(in_len - n_lags)])

And, if you want both lags and output_chunk_length > 1, then I believe it should be:

my_model = RegressionModel(..., lags=n_lags, output_chunk_length=out_len, input_chunk_length=in_len)
my_model.fit(..., sample_weight=[1. / (in_len - i) for i in range(in_len - np.max([n_lags, out_len])])

And finally, when you're training on multiple series and these series differ in length, it gets a bit more complicated. It that case you'll need to take into account the order and the difference in length of the series. For example, in the case of exponentially decaying weights it could be like this:

# function for calculating exponential weights
def exponential_sample_weights(ts, n_lags=8, multiple_series=False, max_series_length=np.nan):
  
    if not multiple_series:
        T = len(ts) - n_lags
        sample_weights = [-np.log(1-t/T)/(T-1) for t in range(1,T+1) if t<T] + [np.log(T)/(T-1)]
    else:
        T = max_series_length - n_lags
        T_self = len(ts) - n_lags
        sample_weights = [-np.log(1-t/T)/(T-1) for t in range(1 +(T-T_self),T+1) if t<T] + [np.log(T)/(T-1)]
    
    return sample_weights

# create a list with the weights in the same order as the series to which they belong
seq_sample_weights = []
max_len = np.max([len(series) for series in seq_series])
for series in seq_series:
        seq_sample_weights_sample_weights += exponential_sample_weights(ts=series, 
                                                                        multiple_series=True, 
                                                                        max_series_length=max_len)

# fit the model (without a specific input_chunk_length)
my_model = RegressionModel(..., lags=n_lags)
my_model.fit(..., sample_weight=seq_sample_weights)

In the latter case you have to be very mindful of the fact that if series differ in length and you train on multiple series, then when you calculate exponentially decaying weights T should be the same for all series if you want to put equal weight on the series.

So, if you want to apply sample weights and use Darts, currently it requires of you that you know what happens behind the scenes. Otherwise it's hard to get it going in the non-trivial cases and it's easy to make mistakes. Therefore I think it would be nice to have an easier way of doing it like:

my_model = RegressionModel(..., lags=n_lags)
my_model.fit(..., sample_weight_type='exponential')

Later on you could also add functionality to let the model focus more on specific series. But I would say that's of lesser importance.

[hrzn]

Hi @Beerstabr , first off, I'm sorry because I realised I made a mistake in my previous message - the sample_weight are (obviously) per-sample weights and not per-dimension weights, as I was too quick to assume. Indeed the actual number of samples is a relatively non-trivial function of the input chunk length (or nr. of lags used on the target), the number of targets, and potentially the parameter max_samples_per_ts. Then once all samples are built (this is done in the function RegressionModel._create_lagged_data(), the weights should be assigned to them as a function of how far in the past the lag of the y column corresponds to.

I think it can be done and it could be a pretty nice feature indeed. However it would also add a bit of complexity, because it would be strongly coupled to the tabularization logic. Nevertheless, if you feel like tackling it, we would be very happy to receive a PR in this direction. However, I would recommend that you wait a little before you start, as we have another couple of initiatives ongoing that are touching the tabularization itself, so it'd be better to do it afterwards to avoid conflicts.

[Beerstabr]

Hi @hrzn,

Seems to me like a fun challenge to tackle. I’ll wait for the right moment though. How will I know the ongoing initiatives will be done? Are their specific backlog items I can follow?

[madtoinou]

Hi @Beerstabr,

The PR refactoring the taburalization has been merged. If you're still interested in implementing this feature, it's more than welcome!

[Beerstabr]

Definitely! It’s really just scratching my own itch, because I would love to use the feature myself. Op do 23 mrt. 2023 om 15:36 schreef madtoinou ***@***.***>

…

Hi @Beerstabr <https://github.com/Beerstabr>, The PR refactoring the taburalization has been merged. If you're still interested in implementing this feature, it's more than welcome! — Reply to this email directly, view it on GitHub <#1175 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ26ZSN62CQJPORUZFHE24DW5RNVJANCNFSM6AAAAAAQADQQGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>