Skip to content

Rewrite section on acceptance tests in AGENTS.md#4476

Merged
denik merged 1 commit intomainfrom
denik/acc-tests-agents
Feb 9, 2026
Merged

Rewrite section on acceptance tests in AGENTS.md#4476
denik merged 1 commit intomainfrom
denik/acc-tests-agents

Conversation

@denik
Copy link
Contributor

@denik denik commented Feb 9, 2026 •
edited
Loading

Changes

Update AGENTS.md with regard to acceptance tests.

Why

Things I want to correct:

  • ClaudeCode updates out.test.toml instead of test.toml
  • ClaudeCode tries to run subtest by setting env var DATABRICKS_BUNDLE_ENGINE=direct go test ...
  • ClaudeCode does not use scripts from acceptance/bin
  • ClaudeCode is confused when updating tests if different variants generate different output.txt, overwriting each other.

@denik @claude
Add acceptance test tips to AGENTS.md
d5eac83 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@denik denik temporarily deployed to test-trigger-is February 9, 2026 14:25 — with GitHub Actions Inactive
@denik denik changed the title Add acceptance test tips to AGENTS.md Rewrite section on acceptance tests in AGENTS.md Feb 9, 2026
@denik denik enabled auto-merge February 9, 2026 14:42
@denik denik disabled auto-merge February 9, 2026 14:43
@denik denik merged commit 848b359 into main Feb 9, 2026
17 of 18 checks passed
@denik denik deleted the denik/acc-tests-agents branch February 9, 2026 14:44
@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Feb 9, 2026 •
edited
Loading

Commit: d5eac83

Run: 21828897225

Env 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 7 1 7 451 731 56:43
🟨​ aws windows 7 1 7 417 741 43:47
💚​ azure linux 2 9 451 730 59:41
💚​ azure windows 2 9 417 740 51:10
🔄​ gcp linux 3 1 9 438 736 66:58
🔄​ gcp windows 4 2 9 402 746 60:00
21 interesting tests: 7 KNOWN, 7 SKIP, 6 flaky, 1 RECOVERED
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 💚​R 💚​R 🔄​f 💚​R
🙈​ TestAccept/bundle/deployment/bind/alert 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/bundle/deployment/bind/pipelines/recreate ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/deployment/bind/pipelines/recreate/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
🙈​ TestAccept/bundle/generate/alert 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/alerts/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/alerts/with_file 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/bundle/resources/dashboards/detect-change ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/dashboards/detect-change/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/bundle/templates/default-python/combinations/classic ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=no/READPLAN= ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
💚​ TestAccept/ssh/connection 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
7:05 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:57 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:56 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:43 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:07 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:43 aws windows TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9
4:17 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:07 azure linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:02 azure linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.12
4:01 aws linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.12
3:56 gcp windows TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9
3:48 gcp windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no/READPLAN=
3:46 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:44 aws linux TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=direct
3:42 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:42 gcp linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.12
3:39 gcp windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes/READPLAN=
3:39 azure linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.10
3:38 gcp windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=yes/PY=no/READPLAN=1
3:36 gcp windows TestAccept/bundle/deployment/bind/pipelines/recreate/DATABRICKS_BUNDLE_ENGINE=terraform
3:35 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:34 gcp linux TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=yes/READPLAN=
3:34 azure linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.13
3:33 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:32 gcp linux TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=direct
3:31 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:25 gcp linux TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=terraform
3:22 gcp linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.10
3:22 gcp windows TestAccept/bundle/deployment/unbind/permissions/DATABRICKS_BUNDLE_ENGINE=direct
3:20 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:17 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:17 gcp windows TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
3:15 azure linux TestAccept/bundle/resources/secret_scopes/permissions/DATABRICKS_BUNDLE_ENGINE=terraform
3:13 gcp linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.9
3:13 gcp windows TestAccept/bundle/resources/dashboards/simple_outside_bundle_root/DATABRICKS_BUNDLE_ENGINE=terraform
3:12 aws linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.9
3:10 azure linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.11
3:07 gcp linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.11
3:07 gcp linux TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=yes/READPLAN=1
3:06 aws linux TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=terraform
3:06 azure linux TestAccept/bundle/resources/dashboards/destroy/DATABRICKS_BUNDLE_ENGINE=terraform
3:05 gcp windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=no/READPLAN=1
3:05 gcp linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.13
3:04 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:03 azure linux TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=terraform
3:03 azure linux TestAccept/bundle/resources/dashboards/change-name/DATABRICKS_BUNDLE_ENGINE=direct
3:01 azure windows TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=terraform
3:00 gcp linux TestAccept/bundle/deployment/bind/job/job-spark-python-task/DATABRICKS_BUNDLE_ENGINE=terraform
3:00 gcp linux TestAccept/bundle/resources/dashboards/change-embed-credentials/DATABRICKS_BUNDLE_ENGINE=direct
3:00 gcp linux TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pietern pietern pietern approved these changes

@andrewnester andrewnester Awaiting requested review from andrewnester andrewnester is a code owner

@anton-107 anton-107 Awaiting requested review from anton-107 anton-107 is a code owner

@shreyas-goenka shreyas-goenka Awaiting requested review from shreyas-goenka shreyas-goenka is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@denik @eng-dev-ecosystem-bot @pietern