Use join instead of views

[vishesh92]

Description

To be merged after #8410 to ensure this doesn't break anything.

This PR contains changes similar to #8012 which is to directly query the tables and make required joins only when there is a requirement to filter on some other field. This will help reduce the query times and thus reduce load on the database as well.

Resource	Count	Before	After	Difference
volumes	748	50	47	3
networks	191	140	131	9
events	7515	61	31	30
accounts	52	100	94	6
domains	51	35	32	3
hosts	101	306	163	143
storage pools	100	41	40	1
service offerings	603	53	41	12
disk offerings	705	48	37	11
templates	603	155	109	46

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

Attention: Patch coverage is 68.29522% with 305 lines in your changes are missing coverage. Please review.

Project coverage is 30.85%. Comparing base (0208e09) to head (61f1382).
Report is 5 commits behind head on 4.19.

Files	Patch %	Lines
...ain/java/com/cloud/api/query/QueryManagerImpl.java	70.45%	138 Missing and 26 partials ⚠️
...c/main/java/com/cloud/utils/db/GenericDaoBase.java	70.66%	16 Missing and 6 partials ⚠️
...b/src/main/java/com/cloud/utils/db/SearchBase.java	47.36%	9 Missing and 11 partials ⚠️
...va/com/cloud/api/query/dao/AccountJoinDaoImpl.java	53.12%	11 Missing and 4 partials ⚠️
...m/cloud/api/query/dao/DiskOfferingJoinDaoImpl.java	53.12%	11 Missing and 4 partials ⚠️
...ava/com/cloud/api/query/dao/DomainJoinDaoImpl.java	53.12%	11 Missing and 4 partials ⚠️
...loud/api/query/dao/ServiceOfferingJoinDaoImpl.java	53.12%	11 Missing and 4 partials ⚠️
.../src/main/java/com/cloud/utils/db/JoinBuilder.java	51.72%	14 Missing ⚠️
...ava/org/apache/cloudstack/acl/RoleManagerImpl.java	56.25%	4 Missing and 3 partials ⚠️
.../storage/datastore/db/PrimaryDataStoreDaoImpl.java	92.98%	2 Missing and 2 partials ⚠️
... and 8 more

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19    #8321      +/-   ##
============================================
- Coverage     30.91%   30.85%   -0.07%     
+ Complexity    34249    34183      -66     
============================================
  Files          5354     5355       +1     
  Lines        376094   376619     +525     
  Branches      54696    54809     +113     
============================================
- Hits         116258   116194      -64     
- Misses       244542   245131     +589     
  Partials      15294    15294              

Flag	Coverage Δ
simulator-marvin-tests	24.64% <68.19%> (-0.09%)	⬇️
uitests	4.39% <ø> (ø)
unit-tests	16.56% <5.19%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[vishesh92]

@blueorangutan package

[blueorangutan]

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[vishesh92]

@blueorangutan package

[blueorangutan]

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8196

[harikrishna-patnala]

@blueorangutan test matrix

[blueorangutan]

@harikrishna-patnala a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-8765)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44377 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t8765-xenserver-71.zip
Smoke tests completed. 119 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_04_change_offering_small	Error	100.44	test_service_offerings.py
test_06_disk_offering_strictness_false	Error	148.42	test_service_offerings.py
test_change_service_offering_for_vm_with_snapshots	Error	98.27	test_vm_snapshots.py
test_01_create_vm_snapshots	Error	90.24	test_vm_snapshots.py
test_02_revert_vm_snapshots	Failure	90.12	test_vm_snapshots.py
test_03_delete_vm_snapshots	Failure	0.01	test_vm_snapshots.py

[blueorangutan]

[SF] Trillian test result (tid-8766)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 49277 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t8766-vmware-67u3.zip
Smoke tests completed. 115 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_root_disk	Error	0.07	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:setup	Error	213.28	test_snapshots.py
test_01_snapshot_usage	Error	0.07	test_usage.py
test_02_balanced_drs_algorithm	Failure	130.02	test_cluster_drs.py
ContextSuite context=TestListIdsParams>:setup	Error	0.00	test_list_ids_parameter.py
test_04_change_offering_small	Error	97.46	test_service_offerings.py
test_06_disk_offering_strictness_false	Error	150.06	test_service_offerings.py
test_change_service_offering_for_vm_with_snapshots	Error	144.69	test_vm_snapshots.py
test_01_create_vm_snapshots	Error	123.73	test_vm_snapshots.py
test_02_revert_vm_snapshots	Failure	90.22	test_vm_snapshots.py
test_03_delete_vm_snapshots	Failure	0.02	test_vm_snapshots.py

[vishesh92]

@blueorangutan package

[blueorangutan]

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[vishesh92]

@blueorangutan test matrix

[blueorangutan]

@vishesh92 a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-8788)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44397 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t8788-xenserver-71.zip
Smoke tests completed. 118 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	Failure	341.81	test_safe_shutdown.py
test_04_change_offering_small	Error	100.42	test_service_offerings.py
test_06_disk_offering_strictness_false	Error	158.00	test_service_offerings.py
test_change_service_offering_for_vm_with_snapshots	Error	211.21	test_vm_snapshots.py

[blueorangutan]

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[weizhouapache]

@blueorangutan package

[weizhouapache]

code lgtm

[yadvr]

@blueorangutan package

[blueorangutan]

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8932

[vishesh92]

@blueorangutan package

[blueorangutan]

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8935

[yadvr]

@blueorangutan test matrix

[blueorangutan]

@rohityadavcloud a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-9482)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40129 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t9482-xenserver-71.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[blueorangutan]

[SF] Trillian test result (tid-9483)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 45959 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t9483-vmware-67u3.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[blueorangutan]

[SF] Trillian test result (tid-9484)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47737 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t9484-kvm-centos7.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	Failure	336.69	test_safe_shutdown.py

[vladimirpetrov]

Tested on a simulator hypervisor, with a separate MS (4 CPUs, 8 GB RAM) and DB VM (8 CPUs, 16 GB RAM), using Apache Benchmark tool (ab), measuring mean response time from 100 executions per list operation.

Number of resources:
Hosts: 1001
Storage pools: 100
Volumes: 25110
Events: 155118
Instances: 8370

Results, mean time in milliseconds

list hosts

	Page=20	Page=50	Page=100
Unpatched	304	676	1052
Patched	259	647	1246
Difference	-15%	-4%	18%

list storage pools

	Page=20	Page=50	Page=100
Unpatched	204	487	972
Patched	193	381	843
Difference	-5%	-22%	-13%

list volumes

	Page=20	Page=50	Page=100
Unpatched	663	878	1205
Patched	389	558	961
Difference	-41%	-36%	-20%

list volume metrics

	Page=20	Page=50	Page=100
Unpatched	699	1057	1329
Patched	374	732	1299
Difference	-46%	-31%	-2%

list events

	Page=20	Page=50	Page=100
Unpatched	2609	2750	2748
Patched	396	522	457
Difference	-85%	-81%	-83%

list instance metrics

	Page=20	Page=50	Page=100
Unpatched	914	1804	3273
Patched	898	1650	3255
Difference	-2%	-9%	-1%

[vladimirpetrov]

LGTM based on performance measurements with and without the patch

[weizhouapache]

Merging based on approvals and CI/manual test results
Thanks @vishesh92 @vladimirpetrov