service gets stuck and doesn't update anymore after some time of uninterrupted operation

[paolobarbolini]

After a few weeks of deps.rs running without new deploys or restarts it looks like the updater for popular crates and popular repos gets stuck and doesn't update anymore. This has happened multiple times now.

This is easily noticeable when serde releases a new version, since the homepage keeps pointing to the older version, causing the badge to report 1 outdated dependency (serde imports an exact version of serde-derive).

deps.rs

local instance

[Feliix42]

Any ideas on how this can happen? Maybe a caching issue?

[robjtede]

Anything useful logged? Does restarting the deployment help?

[paolobarbolini]

Does restarting the deployment help?

It does for a few weeks and then it happens again.

[Feliix42]

Currently, everything is up to date (presumably due to the merge of #107). Let's see, how long it lasts.

[paolobarbolini]

It happened again

[Feliix42]

So I guess it makes sense to assume that both #156 and #145 are somehow related to this?

As @kid pointed out, the logs show only a few error messages regarding timeouts, but nothing beyond that. Maybe it makes sense to try and locate the problem methodically:

• How does the deployment machine look after deps has been running a while?
  • Is there a slow but steady increase in memory usage for instance?
  • Are we accidentally keeping any connections open?
• Can we get a more detailed error logging set up? Or can we get a full log that spans the time before and after the error start occuring? Sadly the instance restart also means we lost the logs, I think.
• Can we try to reproduce the problem locally on our machines by spamming it for a while with requests?
• @cecton pointed out that we could be rate-limited. Can we check that? Since the log snippet posted by @kid shows both Gitlab and Github timeout-ing, I would guess it's not on the receiving end of our requests at least.

[kid]

definitely looks like a memory leak

[cecton]

If we were hitting the memory limit the process would have been killed with signal 9 and I would expect the thing to restart automatically and to be working again.

[robjtede]

Couldn't it just be the cache getting bigger over time, as we'd expect?