Skip to content

[RAM] 142183 create bulk delete on rules #142466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
acbedf5
create base for bulk delete route
guskovaue Sep 29, 2022
ff542b1
add bulk delete method
guskovaue Oct 3, 2022
8211403
add bulkDelete method for task manager
guskovaue Oct 4, 2022
25a2471
first version of retry if conflicts for bulk delete
guskovaue Oct 5, 2022
e6ded33
move waitBeforeNextRetry to separate file
guskovaue Oct 5, 2022
6c5d4e4
add happy path for unit and integration tests
guskovaue Oct 6, 2022
09e4559
rewrite retry_if_bulk_delete and add tests
guskovaue Oct 7, 2022
fa6aea6
fix happy path integration test
guskovaue Oct 7, 2022
577a0f3
add some tests and fix bugs
guskovaue Oct 12, 2022
8e29403
make bulk_delete endpoint external
guskovaue Oct 12, 2022
bcd74a2
give up on types for endpoint arguments
guskovaue Oct 12, 2022
a5a51db
add unit tests for new bulk delete route
guskovaue Oct 13, 2022
daeceec
add unit tests for bulk delete rules clients method
guskovaue Oct 13, 2022
660c3b9
add integrational tests
guskovaue Oct 13, 2022
64c0621
api integration test running
XavierM Oct 14, 2022
561f63b
Revert "api integration test running"
guskovaue Oct 14, 2022
b8a0868
add integrational tests
guskovaue Oct 14, 2022
eed0b98
use bulk edit constant in log audit
guskovaue Oct 14, 2022
9d18083
unskip skiped test
guskovaue Oct 14, 2022
1099901
fix conditional statement for taskIds
guskovaue Oct 14, 2022
957d99d
api integration for bulkDelete is done
XavierM Oct 14, 2022
c84d0f4
Merge branch 'main' of github.com:elastic/kibana into RAM-142183-crea…
XavierM Oct 14, 2022
bc5971a
small code style changes
guskovaue Oct 16, 2022
593133d
Merge branch 'main' into RAM-142183-create-bulk-delete-on-rules
kibanamachine Oct 17, 2022
d7a326f
delete comments and rename types
guskovaue Oct 17, 2022
2db12bf
Merge branch 'RAM-142183-create-bulk-delete-on-rules' of github.com:g…
guskovaue Oct 17, 2022
597b652
get rid of pmap without async
guskovaue Oct 17, 2022
61aff32
delete not used part of return
guskovaue Oct 17, 2022
487cfdc
add audit logs for all deleted rules
guskovaue Oct 17, 2022
e9d946a
add unit tests for audit logs
guskovaue Oct 18, 2022
1b07f41
delete extra comments and rename constant
guskovaue Oct 18, 2022
a365269
delete extra tests
guskovaue Oct 18, 2022
fad270a
fix audit logs
guskovaue Oct 19, 2022
be09680
restrict amount of passed ids to 1000
guskovaue Oct 19, 2022
f85cd12
fix audit logs again
guskovaue Oct 19, 2022
635384e
fix alerting security tests
guskovaue Oct 19, 2022
5cebdc6
test case when user pass more that 1000 ids
guskovaue Oct 20, 2022
40e7fd1
fix writing and case when you send no args
guskovaue Oct 22, 2022
076b3fa
fix line in the text
guskovaue Oct 22, 2022
867d99f
delete extra test
guskovaue Oct 22, 2022
17f0eba
fix type for rules we passing to bulk delete
guskovaue Oct 22, 2022
0e7beab
add catch RuleTypeDisabledError
guskovaue Oct 22, 2022
d08f2f6
wait before next retry func tests
guskovaue Oct 23, 2022
f84fdde
fix tests for retry function
guskovaue Oct 23, 2022
4652756
fix bulk delete rule client tests
guskovaue Oct 23, 2022
7ac339f
y
guskovaue Oct 24, 2022
e340ee2
Merge branch 'main' into RAM-142183-create-bulk-delete-on-rules
kibanamachine Oct 24, 2022
6cf45fa
add to api return task ids failed to be deleted and wrapp task manage…
guskovaue Oct 25, 2022
0c7110e
Merge branch 'RAM-142183-create-bulk-delete-on-rules' of github.com:g…
guskovaue Oct 25, 2022
8668810
fix type fpr task manager
guskovaue Oct 25, 2022
c482fc3
Revert "y"
guskovaue Oct 25, 2022
f558ab2
fix more types
guskovaue Oct 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ export enum WriteOperations {
UnmuteAlert = 'unmuteAlert',
Snooze = 'snooze',
BulkEdit = 'bulkEdit',
BulkDelete = 'bulkDelete',
Unsnooze = 'unsnooze',
}

Expand Down
126 changes: 126 additions & 0 deletions x-pack/plugins/alerting/server/routes/bulk_delete_rules.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { httpServiceMock } from '@kbn/core/server/mocks';

import { bulkDeleteRulesRoute } from './bulk_delete_rules';
import { licenseStateMock } from '../lib/license_state.mock';
import { mockHandlerArguments } from './_mock_handler_arguments';
import { rulesClientMock } from '../rules_client.mock';
import { RuleTypeDisabledError } from '../lib/errors/rule_type_disabled';
import { verifyApiAccess } from '../lib/license_api_access';

const rulesClient = rulesClientMock.create();

jest.mock('../lib/license_api_access', () => ({
verifyApiAccess: jest.fn(),
}));

beforeEach(() => {
jest.resetAllMocks();
});

describe('bulkDeleteRulesRoute', () => {
const bulkDeleteRequest = { filter: '' };
const bulkDeleteResult = { errors: [], total: 1, taskIdsFailedToBeDeleted: [] };

it('should delete rules with proper parameters', async () => {
const licenseState = licenseStateMock.create();
const router = httpServiceMock.createRouter();

bulkDeleteRulesRoute({ router, licenseState });

const [config, handler] = router.patch.mock.calls[0];

expect(config.path).toBe('/internal/alerting/rules/_bulk_delete');

rulesClient.bulkDeleteRules.mockResolvedValueOnce(bulkDeleteResult);

const [context, req, res] = mockHandlerArguments(
{ rulesClient },
{
body: bulkDeleteRequest,
},
['ok']
);

expect(await handler(context, req, res)).toEqual({
body: bulkDeleteResult,
});

expect(rulesClient.bulkDeleteRules).toHaveBeenCalledTimes(1);
expect(rulesClient.bulkDeleteRules.mock.calls[0]).toEqual([bulkDeleteRequest]);

expect(res.ok).toHaveBeenCalled();
});

it('ensures the license allows bulk deleting rules', async () => {
const licenseState = licenseStateMock.create();
const router = httpServiceMock.createRouter();

rulesClient.bulkDeleteRules.mockResolvedValueOnce(bulkDeleteResult);

bulkDeleteRulesRoute({ router, licenseState });

const [, handler] = router.patch.mock.calls[0];

const [context, req, res] = mockHandlerArguments(
{ rulesClient },
{
body: bulkDeleteRequest,
}
);

await handler(context, req, res);

expect(verifyApiAccess).toHaveBeenCalledWith(licenseState);
});

it('ensures the license check prevents bulk deleting rules', async () => {
const licenseState = licenseStateMock.create();
const router = httpServiceMock.createRouter();

(verifyApiAccess as jest.Mock).mockImplementation(() => {
throw new Error('Failure');
});

bulkDeleteRulesRoute({ router, licenseState });

const [, handler] = router.patch.mock.calls[0];

const [context, req, res] = mockHandlerArguments(
{ rulesClient },
{
body: bulkDeleteRequest,
}
);

expect(handler(context, req, res)).rejects.toMatchInlineSnapshot(`[Error: Failure]`);
});

it('ensures the rule type gets validated for the license', async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to test for special handling for RuleTypeDisabledErrors but there is no special handling in the route. I see there is special handling in the bulk edit route for this type of error. Should we add special handling for that error or remove this test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote the new endpoint like this because the old delete endpoint does not have it. But now I see that most of endpoints catch RuleTypeDisabledError, so I added it: 0e7beab

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the old delete endpoint doesn't have it because the rules client does not check for this.ruleTypeRegistry.ensureRuleTypeEnabled during deletion, which I actually think is ok. Most likely it's in the bulk delete function because it was copied over from bulk edit? I would probably remove the check from the rules client bulk delete and remove that error handling from the route and remove this test. If for some reason, the rule type becomes disabled for license reasons, we shouldn't prevent users from deleting those rules.

const licenseState = licenseStateMock.create();
const router = httpServiceMock.createRouter();

bulkDeleteRulesRoute({ router, licenseState });

const [, handler] = router.patch.mock.calls[0];

rulesClient.bulkDeleteRules.mockRejectedValue(
new RuleTypeDisabledError('Fail', 'license_invalid')
);

const [context, req, res] = mockHandlerArguments({ rulesClient }, { params: {}, body: {} }, [
'ok',
'forbidden',
]);

await handler(context, req, res);

expect(res.forbidden).toHaveBeenCalledWith({ body: { message: 'Fail' } });
});
});
50 changes: 50 additions & 0 deletions x-pack/plugins/alerting/server/routes/bulk_delete_rules.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { schema } from '@kbn/config-schema';
import { IRouter } from '@kbn/core/server';
import { verifyAccessAndContext, handleDisabledApiKeysError } from './lib';
import { ILicenseState, RuleTypeDisabledError } from '../lib';
import { AlertingRequestHandlerContext, INTERNAL_BASE_ALERTING_API_PATH } from '../types';

export const bulkDeleteRulesRoute = ({
router,
licenseState,
}: {
router: IRouter<AlertingRequestHandlerContext>;
licenseState: ILicenseState;
}) => {
router.patch(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

patch seems unusual here - I would have guessed we'd used post instead. Why did we pick patch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @pmuellr!
It was hard choice. We basically use delete, because user can pass up to 1000 ids in payload.
So, when we chose, we follow this:
POST is always for creating a resource
and
PATCH is always for updating a resource.
So basically that. Updating is not completely like deleting :-), but we need to choose something.

{
path: `${INTERNAL_BASE_ALERTING_API_PATH}/rules/_bulk_delete`,
validate: {
body: schema.object({
filter: schema.maybe(schema.string()),
ids: schema.maybe(schema.arrayOf(schema.string(), { minSize: 1, maxSize: 1000 })),
}),
},
},
handleDisabledApiKeysError(
router.handleLegacyErrors(
verifyAccessAndContext(licenseState, async (context, req, res) => {
const rulesClient = (await context.alerting).getRulesClient();
const { filter, ids } = req.body;

try {
const result = await rulesClient.bulkDeleteRules({ filter, ids });
return res.ok({ body: result });
} catch (e) {
if (e instanceof RuleTypeDisabledError) {
return e.sendResponse(res);
}
throw e;
}
})
)
)
);
};
2 changes: 2 additions & 0 deletions x-pack/plugins/alerting/server/routes/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import { bulkEditInternalRulesRoute } from './bulk_edit_rules';
import { snoozeRuleRoute } from './snooze_rule';
import { unsnoozeRuleRoute } from './unsnooze_rule';
import { runSoonRoute } from './run_soon';
import { bulkDeleteRulesRoute } from './bulk_delete_rules';

export interface RouteOptions {
router: IRouter<AlertingRequestHandlerContext>;
Expand Down Expand Up @@ -76,6 +77,7 @@ export function defineRoutes(opts: RouteOptions) {
unmuteAlertRoute(router, licenseState);
updateRuleApiKeyRoute(router, licenseState);
bulkEditInternalRulesRoute(router, licenseState);
bulkDeleteRulesRoute({ router, licenseState });
snoozeRuleRoute(router, licenseState);
unsnoozeRuleRoute(router, licenseState);
runSoonRoute(router, licenseState);
Expand Down
1 change: 1 addition & 0 deletions x-pack/plugins/alerting/server/rules_client.mock.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ const createRulesClientMock = () => {
getActionErrorLog: jest.fn(),
getSpaceId: jest.fn(),
bulkEdit: jest.fn(),
bulkDeleteRules: jest.fn(),
snooze: jest.fn(),
unsnooze: jest.fn(),
calculateIsSnoozedUntil: jest.fn(),
Expand Down
1 change: 1 addition & 0 deletions x-pack/plugins/alerting/server/rules_client/lib/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,6 @@
export { mapSortField } from './map_sort_field';
export { validateOperationOnAttributes } from './validate_attributes';
export { retryIfBulkEditConflicts } from './retry_if_bulk_edit_conflicts';
export { retryIfBulkDeleteConflicts } from './retry_if_bulk_delete_conflicts';
export { applyBulkEditOperation } from './apply_bulk_edit_operation';
export { buildKueryNodeFilter } from './build_kuery_node_filter';
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { KueryNode } from '@kbn/es-query';
import { loggingSystemMock } from '@kbn/core/server/mocks';

import { retryIfBulkDeleteConflicts } from './retry_if_bulk_delete_conflicts';
import { RETRY_IF_CONFLICTS_ATTEMPTS } from './wait_before_next_retry';

const mockFilter: KueryNode = {
type: 'function',
value: 'mock',
};

const mockLogger = loggingSystemMock.create().get();

const mockSuccessfulResult = {
apiKeysToInvalidate: ['apiKey1'],
errors: [],
taskIdsToDelete: ['taskId1'],
};

const error409 = {
message: 'some fake message',
status: 409,
rule: {
id: 'fake_rule_id',
name: 'fake rule name',
},
};

const getOperationConflictsTimes = (times: number) => {
return async () => {
conflictOperationMock();
times--;
if (times >= 0) {
return {
apiKeysToInvalidate: [],
taskIdsToDelete: [],
errors: [error409],
};
}
return mockSuccessfulResult;
};
};

const OperationSuccessful = async () => mockSuccessfulResult;
const conflictOperationMock = jest.fn();

describe('retryIfBulkDeleteConflicts', () => {
beforeEach(() => {
jest.resetAllMocks();
});

test('should work when operation is successful', async () => {
const result = await retryIfBulkDeleteConflicts(mockLogger, OperationSuccessful, mockFilter);

expect(result).toEqual(mockSuccessfulResult);
});

test('should throw error when operation fails', async () => {
await expect(
retryIfBulkDeleteConflicts(
mockLogger,
async () => {
throw Error('Test failure');
},
mockFilter
)
).rejects.toThrowError('Test failure');
});

test(`should return conflict errors when number of retries exceeds ${RETRY_IF_CONFLICTS_ATTEMPTS}`, async () => {
const result = await retryIfBulkDeleteConflicts(
mockLogger,
getOperationConflictsTimes(RETRY_IF_CONFLICTS_ATTEMPTS + 1),
mockFilter
);

expect(result.errors).toEqual([error409]);
expect(mockLogger.warn).toBeCalledWith('Bulk delete rules conflicts, exceeded retries');
});

for (let i = 1; i <= RETRY_IF_CONFLICTS_ATTEMPTS; i++) {
test(`should work when operation conflicts ${i} times`, async () => {
const result = await retryIfBulkDeleteConflicts(
mockLogger,
getOperationConflictsTimes(i),
mockFilter
);

expect(conflictOperationMock.mock.calls.length).toBe(i + 1);
expect(result).toStrictEqual(mockSuccessfulResult);
});
}
});
Loading