deal with excessive attempts on failing snapshots - #13743
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13743 +/- ##
============================================
+ Coverage 19.64% 19.66% +0.01%
- Complexity 19796 19808 +12
============================================
Files 6368 6368
Lines 575119 575017 -102
Branches 70382 70371 -11
============================================
+ Hits 112994 113059 +65
+ Misses 449839 449675 -164
+ Partials 12286 12283 -3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
This PR adds logic to reduce excessive retry attempts for failing recurring snapshots by tracking outcomes via event history, applying configurable retry/backoff behavior for dispatch failures, and optionally skipping snapshots when a VM hasn’t been running since the last snapshot.
Changes:
- Add configurable max-failure and retry-interval handling for recurring snapshot dispatch failures (and log outcomes as events for consecutive-failure counting).
- Introduce optional “skip snapshot if VM not running since last snapshot” behavior plus a new
SNAPSHOT.SKIPPEDevent type. - Add DAO support to query recent events per resource, and add unit tests covering the new behaviors.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| server/src/main/java/com/cloud/storage/snapshot/SnapshotSchedulerImpl.java | Adds failure counting, scoped config resolution, dispatch failure handling with backoff/give-up, and unchanged-volume skip logic. |
| server/src/test/java/com/cloud/storage/snapshot/SnapshotSchedulerImplTest.java | Adds unit tests for consecutive failure counting, scoped config fallback, dispatch failure behavior, and skip logic. |
| server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java | Registers new snapshot-related config keys. |
| server/src/main/java/com/cloud/storage/snapshot/SnapshotManager.java | Defines new scoped config keys for recurring snapshot failure handling and skip behavior. |
| engine/schema/src/main/java/com/cloud/event/dao/EventDaoImpl.java | Implements querying latest events per resource for failure counting. |
| engine/schema/src/main/java/com/cloud/event/dao/EventDao.java | Adds DAO API for latest-events-by-resource lookup. |
| api/src/main/java/com/cloud/event/EventTypes.java | Adds EVENT_SNAPSHOT_SKIPPED constant and registers it for entity-event details. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (maxFailures > 0 && totalFailures >= maxFailures) { | ||
| logger.warn("Snapshot schedule [{}] for volume [{}] has failed [{}] consecutive times.", snapshotSchedule, volume, totalFailures); | ||
| ActionEventUtils.onCreatedActionEvent(User.UID_SYSTEM, volume.getAccountId(), EventVO.LEVEL_WARN, EventTypes.EVENT_SNAPSHOT_CREATE, true, | ||
| String.format("Recurring snapshot for volume [%s] has failed %d consecutive times.", volume, totalFailures), | ||
| volume.getId(), ApiCommandResourceType.Volume.toString()); | ||
| } |
| @Override | ||
| public List<EventVO> listLatestEventsByResource(long resourceId, String resourceType, String type, int limit) { | ||
| SearchCriteria<EventVO> sc = LatestEventsByResourceSearch.create(); | ||
| sc.setParameters("resourceId", resourceId); | ||
| sc.setParameters("resourceType", resourceType); | ||
| sc.setParameters("type", type); | ||
| Filter filter = new Filter(EventVO.class, "createDate", false, 0L, (long) limit); | ||
| return listBy(sc, filter); | ||
| } |
|
🔴 Test Coverage Grade:
|
| Metric | Value |
|---|---|
| Line coverage | 24.52% |
| Branch coverage | 18.68% |
Grade Scale
| Grade | Line Coverage | Meaning |
|---|---|---|
| 🟢 A | ≥ 80% | Excellent - this code sleeps well at night 😴 |
| 🟡 B | 60-79% | Good - almost there, don't stop now 😉 |
| 🟠 C | 40-59% | Acceptable - your code is wearing a seatbelt, but no airbags 😬 |
| 🔴 D | 20-39% | Marginal - boldly shipping where no test has gone before 🖖 |
| ⛔ F | < 20% | Failing - tests? what tests? 🔥 |
Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run
Description
This PR...
Fixes: #13454
Fixes: #6827
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?