Apache DolphinScheduler 云上部署实战Packer 定制 AMI Terraform 一键拉起 AWS 全栈集群【免费下载链接】dolphinschedulerApache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code项目地址: https://gitcode.com/GitHub_Trending/dol/dolphinscheduler本文基于仓库内置的 AWS 部署文档deploy/terraform/aws/README.md系统讲解如何用 Packer 构建含 DolphinScheduler 发行版的定制 AMI再用 Terraform 在 AWS 上声明式创建 EC2、RDS、S3、ZooKeeper 等全套资源最终得到可访问的 DolphinScheduler 集群模式环境。读完后你将掌握完整的 Packer 变量文件编写、Terraform 变量声明、cloud-init 自动化启动机制以及全部输入/输出参数的含义与默认值。部署架构与整体流程这套 Terraform 脚本的目标是在几分钟内搭建一个 DolphinScheduler 环境独立模式或集群模式其核心思路是“镜像 声明式基础设施 开机自启”三段式Packer 阶段以 Amazon Linux 2022 为基础镜像安装 Java 与 DolphinScheduler 发行版从官方 tar 或本地构建 tar 二选一产出名为ds_ami_name的自定义 AMITerraform 阶段按组件master / worker / api / alert / standalone_server分别创建 EC2 实例同时创建 RDSPostgreSQL、S3 桶、可选的ZooKeeper 节点、密钥对、VPC 与子网cloud-init 阶段每台 DS 实例开机后自动创建ds用户、初始化数据库 Schema、把 S3 配置写入common.properties并注册 systemd 服务自启对应组件。各组件资源定义分散在独立文件中便于按组件维护dolphinscheduler-master.tf、dolphinscheduler-worker.tf、dolphinscheduler-api.tf、dolphinscheduler-alert.tf、dolphinscheduler-standalone.tf。前置条件需要安装两个 HashiCorp 工具Packer用于构建自定义 AMITerraform用于声明并创建 AWS 资源。两者均按官方下载文档安装即可。第一步用 Packer 构建 DolphinScheduler AMI编写变量文件ds-ami.pkrvars.hcl创建变量文件并填入 AWS 凭证、区域与版本信息cat EOF ds-ami.pkrvars.hcl aws_access_key aws_secret_key aws_region cn-north-1 ds_ami_name my-test-ds-2 # 若使用官方发行版 tar将 ds_version 设为目标版本即可 ds_version 3.1.1 # 若使用本地构建的发行版 tar将 ds_tar 指向 tar 文件位置 ds_tar ~/workspace/dolphinscheduler/dolphinscheduler-dist/target/apache-dolphinscheduler-3.1.3-SNAPSHOT-bin.tar.gz EOF其中ds_version与ds_tar分别对应两条构建路径只需二选一官方发行版路径用ds_version本地构建路径用ds_tar。执行构建使用官方发行版 tar变量定义见 ds-ami-official.pkr.hclpacker init --var-fileds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl packer build --var-fileds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl使用本地构建的发行版 tar变量定义见 ds-ami-local.pkr.hclpacker init --var-fileds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl packer build --var-fileds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl构建过程在做什么源码级拆解两个 Packer 模板结构一致差异仅在 Provisioner基础镜像选择source amazon-ebs linux以t2.micro实例构建通过source_ami_filter筛选al2022-ami-*amazon 官方、EBS root、HVM、x86_64、most_recentSSH 用户为ec2-user见 ds-ami-official.pkr.hclJava 环境先yum remove -y java再安装java-1.8.0-amazon-corretto.x86_64并通过/etc/profile.d/java_home.sh导出JAVA_HOME/etc/alternatives/jre官方 tar 路径curl从https://archive.apache.org/dist/dolphinscheduler/${var.ds_version}/apache-dolphinscheduler-${var.ds_version}-bin.tar.gz下载tar zxvf - --strip-components 1 -C /opt/dolphinscheduler解压到/opt/dolphinschedulerds-ami-official.pkr.hcl#L80-L89本地 tar 路径先用fileProvisioner 将var.ds_tar上传到实例的~/dolphinscheduler.tar.gz再本地解压ds-ami-local.pkr.hcl#L77-L91收尾对目录内所有start.sh执行chmod x保证各组件启动脚本可执行。最终 AMI 内固定路径为/opt/dolphinscheduler后续所有 systemd 服务的WorkingDirectory都基于此路径。第二步Terraform 声明并创建资源编写terraform.tfvarscat EOF terraform.tfvars aws_access_key aws_secret_key aws_region name_prefix test-ds-terraform ds_ami_name my-test-ds ds_component_replicas { master 1 worker 1 alert 1 api 1 standalone_server 0 } EOF注意两点ds_ami_name必须与上一步ds-ami.pkrvars.hcl中的值完全一致——Terraform 侧通过 os-versions.tf 中data aws_ami dolphinscheduler按名称owners [self]、most_recent true反查 AMI名称对不上会直接找不到镜像上表仅为文档示例的最小集。从 dolphinscheduler-variables.tf 等变量定义看db_password是必填项无默认值示例中未给出实际使用时需自行补充进terraform.tfvars。应用资源terraform init -var-fileterraform.tfvars terraform apply -var-fileterraform.tfvars -auto-approve实例创建细节源码级拆解以 master 为例dolphinscheduler-master.tf其余组件模式相同副本数驱动count var.ds_component_replicas.master将某组件副本数设为0即不创建该组件默认配置中standalone_server 0即默认不部署独立模式节点见 dolphinscheduler-variables.tf#L30-L40实例规格与存储按组件取vm_instance_typemaster/worker 默认t2.mediumapi/standalone 默认t2.smallalert 默认t2.micro根卷 30 GB、数据卷 10 GB均为gp2且开启加密实例挂载在aws_subnet.public[0]是否绑公网 IP 由vm_associate_public_ip_address按组件控制安全组规则master 安全组放行 22 端口SSH并对 5678 端口DolphinScheduler 的 gRPC/服务端口按安全组授权来自 api 与 worker 的入站连接dolphinscheduler-master.tf#L18-L54体现了组件间 5678 端口的调用拓扑user_data将渲染后的 cloud-init 模板见下文作为user_data注入实例开机即完成初始化。cloud-init实例开机后自动完成三件事所有 DS 组件共用 templates/cloud-init.yaml 模板Terraform 按组件渲染${dolphinscheduler_component}如master-server等占位符。模板完成创建用户与服务创建带 sudo 权限的ds用户并写入两个 systemd 单元cloud-init.yaml#L34-L74dolphinscheduler-schema.serviceoneshot类型执行tools/bin/upgrade-schema.sh初始化数据库 Schemadolphinscheduler.serviceRequiresdolphinscheduler-schema.service、Restartalways执行/opt/dolphinscheduler/${dolphinscheduler_component}/bin/start.sh启动对应组件注入连接配置通过环境变量传入DATABASEpostgresql、SPRING_PROFILES_ACTIVEpostgresql、SPRING_DATASOURCE_URLjdbc:postgresql://${database_address}:${database_port}/${database_name}、REGISTRY_ZOOKEEPER_CONNECT_STRING${zookeeper_connect_string}、WORKER_ALERT_LISTEN_HOST${alert_server_host}等配置 S3 资源存储runcmd中对目录内所有common.properties执行sed把resource.storage.type改写为S3并写入resource.aws.access.key.id、resource.aws.secret.access.key、resource.aws.region、resource.aws.s3.bucket.name、resource.aws.s3.endpoint五个键值cloud-init.yaml#L78-L88随后systemctl enable dolphinscheduler并启动两个服务。换言之DB 与 S3 的凭证不在构建 AMI 时固化而是在 Terraform 阶段以 cloud-init 渲染注入实现了“镜像不含凭证、凭证随环境走”的分离。第三步访问 DolphinScheduler UIterraform apply成功后取 api 实例的公网 DNS 并访问 12345 端口API Server 端口open http://$(terraform output -json api_server_instance_public_dns | jq -r .[0]):12345/dolphinscheduler/uiapi_server_instance_public_dns等输出定义在 dolphinscheduler-output.tf。配套 AWS 资源的设计RDSPostgreSQLrds-main.tf 创建引擎postgres14.5、库名dolphinscheduler、实例规格由db_instance_class控制默认db.t3.micro、存储 5 GB、publicly_accessible true安全组仅允许来自 master / worker / alert / api / standalone 五个安全组的 5432 端口入站rds-main.tf#L18-L34。数据库名、端口、地址等会作为输出暴露见db_address、db_name、db_port。ZooKeeper注册中心若zookeeper_connect_string非空直接使用该外部 ZooKeeper 连接串若为空默认Terraform 会额外创建一台 ZooKeeper 节点基于 Amazon Linux 镜像通过remote-exec执行docker run -it --name zookeeper -d -p 2181:2181 zookeeper:3.5拉起单节点 ZooKeeperzookeeper-main.tf#L62-L109各组件安全组对其 2181 端口放行。变量描述中明确提醒这是单节点演示用途生产环境请接入外部 ZooKeeperzookeeper-variables.tf。S3资源存储S3 桶名以s3_bucket_prefix默认dolphinscheduler-test-见 s3-variables.tf加随机后缀生成配套的 IAM 访问密钥用于上文 cloud-init 中改写common.properties。相关输出包括s3_bucket、s3_address、s3_access_key、s3_secret、s3_regional_domain_name。网络与命名VPC 默认10.0.0.0/16公网子网默认 1 个10.0.1.0/24起 4 个可用 CIDR私有子网默认 2 个10.0.101.0/24起定义见 network-variables.tfname_prefix作为全部资源名前缀默认dolphinschedulertags统一打标默认{ Deployment: Test }默认区域cn-north-1provider-variables.tf可按需覆盖。Inputs 参数总表NameDescriptionTypeDefaultRequiredaws_access_keyAWS access keystringn/ayesaws_regionAWS regionstringcn-north-1noaws_secret_keyAWS secret keystringn/ayesdb_instance_classDatabase instance classstringdb.t3.micronodb_passwordDatabase passwordstringn/ayesdb_usernameDatabase usernamestringdolphinschedulernods_ami_nameName of DolphinScheduler AMIstringdolphinscheduler-aminods_component_replicasReplicas of the DolphinScheduler Componentsmap(number){alert: 1, api: 1, master: 1, standalone_server: 0, worker: 1}nods_versionDolphinScheduler Versionstring3.1.1noname_prefixName prefix for all resourcesstringdolphinschedulernoprivate_subnet_cidr_blocksAvailable CIDR blocks for private subnetslist(string)[10.0.101.0/24, 10.0.102.0/24, 10.0.103.0/24, 10.0.104.0/24]nopublic_subnet_cidr_blocksCIDR blocks for the public subnetslist(string)[10.0.1.0/24, 10.0.2.0/24, 10.0.3.0/24, 10.0.4.0/24]nos3_bucket_prefixS3 bucket name prefixstringdolphinscheduler-test-nosubnet_countNumber of subnetsmap(number){private: 2, public: 1}notagsTags to apply to all resourcesmap(string){Deployment: Test}novm_associate_public_ip_addressAssociate a public IP address to the EC2 instancemap(bool){alert: true, api: true, master: true, standalone_server: true, worker: true}novm_data_volume_sizeData volume size of the EC2 Instancemap(number){alert: 10, api: 10, master: 10, standalone_server: 10, worker: 10}novm_data_volume_typeData volume type of the EC2 Instancemap(string){alert: gp2, api: gp2, master: gp2, standalone_server: gp2, worker: gp2}novm_instance_typeEC2 instance typemap(string){alert: t2.micro, api: t2.small, master: t2.medium, standalone_server: t2.small, worker: t2.medium}novm_root_volume_sizeRoot volume size of the EC2 Instancemap(number){alert: 30, api: 30, master: 30, standalone_server: 30, worker: 30}novm_root_volume_typeRoot volume type of the EC2 Instancemap(string){alert: gp2, api: gp2, master: gp2, standalone_server: gp2, worker: gp2}novpc_cidrCIDR for the VPCstring10.0.0.0/16nozookeeper_connect_stringZookeeper connect string, if empty, will create a single-node zookeeper for demonstration, dont use this in productionstringnoOutputs 总表NameDescriptionalert_server_instance_idInstance IDs of alert instancesalert_server_instance_private_ipPrivate IPs of alert instancesalert_server_instance_public_dnsPublic domain names of alert instancesalert_server_instance_public_ipPublic IPs of alert instancesapi_server_instance_idInstance IDs of api instancesapi_server_instance_private_ipPrivate IPs of api instancesapi_server_instance_public_dnsPublic domain names of api instancesapi_server_instance_public_ipPublic IPs of api instancesdb_addressDatabase addressdb_nameDatabase namedb_portDatabase portmaster_server_instance_idInstance IDs of master instancesmaster_server_instance_private_ipPrivate IPs of master instancesmaster_server_instance_public_dnsPublic domain names of master instancesmaster_server_instance_public_ipPublic IPs of master instancess3_access_keyS3 access keys3_addressS3 addresss3_bucketS3 bucket names3_regional_domain_nameS3 regional domain names3_secretS3 access secretvm_server_instance_idInstance IDs of standalone instancesvm_server_instance_private_ipPrivate IPs of standalone instancesvm_server_instance_public_dnsPublic domain names of standalone instancesvm_server_instance_public_ipPublic IPs of standalone instancesworker_server_instance_idInstance IDs of worker instancesworker_server_instance_private_ipPrivate IPs of worker instancesworker_server_instance_public_dnsPublic domain names of worker instancesworker_server_instance_public_ipPublic IPs of worker instanceszookeeper_server_instance_idInstance IDs of zookeeper instanceszookeeper_server_instance_private_ipPrivate IPs of zookeeper instanceszookeeper_server_instance_public_dnsPublic domain names of zookeeper instanceszookeeper_server_instance_public_ipPublic IPs of zookeeper instances实战注意事项与限制AMI 名称一致性terraform.tfvars中的ds_ami_name必须与 Packer 构建时的ds_ami_name一致否则data.aws_ami.dolphinscheduler查不到镜像必填凭证aws_access_key、aws_secret_key、db_password均无默认值必须在terraform.tfvars中提供文档示例的 tfvars 片段未含db_password属示例省略单节点 ZooKeeper 仅限演示zookeeper_connect_string留空时创建的单节点 ZooKeeper 基于 Docker 单容器官方注释明确不建议生产使用安全组偏宽各组件安全组对 22 端口放行了0.0.0.0/0如 dolphinscheduler-master.tf#L22-L28生产使用前建议收窄来源网段版本前提默认ds_version为3.1.1RDS 引擎固定 PostgreSQL 14.5AM 模板中的下载路径依赖archive.apache.org/dist/dolphinscheduler/下存在对应版本的-bin.tar.gz使用其他版本时需确认该发布物存在访问路径UI 仅通过 api 组件的公网 DNS 12345 端口暴露需确保 api 实例vm_associate_public_ip_address为 true默认即 true。小结该方案把“构建镜像Packer”与“编排资源Terraform”解耦镜像只负责装好 Java 与 DolphinScheduler 发行版环境差异DB 地址、S3 凭证、ZooKeeper 连接串、组件副本数全部由 Terraform 变量与 cloud-init 渲染注入。通过ds_component_replicas一张 map 即可伸缩各组件副本数通过zookeeper_connect_string在“自建演示 ZooKeeper”与“外部注册中心”间切换适合在 AWS 上快速搭建可运行、可访问的 DolphinScheduler 集群环境。【免费下载链接】dolphinschedulerApache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code项目地址: https://gitcode.com/GitHub_Trending/dol/dolphinscheduler创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
Apache DolphinScheduler 云上部署实战：Packer 定制 AMI + Terraform 一键拉起 AWS 全栈集群
Apache DolphinScheduler 云上部署实战Packer 定制 AMI Terraform 一键拉起 AWS 全栈集群【免费下载链接】dolphinschedulerApache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code项目地址: https://gitcode.com/GitHub_Trending/dol/dolphinscheduler本文基于仓库内置的 AWS 部署文档deploy/terraform/aws/README.md系统讲解如何用 Packer 构建含 DolphinScheduler 发行版的定制 AMI再用 Terraform 在 AWS 上声明式创建 EC2、RDS、S3、ZooKeeper 等全套资源最终得到可访问的 DolphinScheduler 集群模式环境。读完后你将掌握完整的 Packer 变量文件编写、Terraform 变量声明、cloud-init 自动化启动机制以及全部输入/输出参数的含义与默认值。部署架构与整体流程这套 Terraform 脚本的目标是在几分钟内搭建一个 DolphinScheduler 环境独立模式或集群模式其核心思路是“镜像 声明式基础设施 开机自启”三段式Packer 阶段以 Amazon Linux 2022 为基础镜像安装 Java 与 DolphinScheduler 发行版从官方 tar 或本地构建 tar 二选一产出名为ds_ami_name的自定义 AMITerraform 阶段按组件master / worker / api / alert / standalone_server分别创建 EC2 实例同时创建 RDSPostgreSQL、S3 桶、可选的ZooKeeper 节点、密钥对、VPC 与子网cloud-init 阶段每台 DS 实例开机后自动创建ds用户、初始化数据库 Schema、把 S3 配置写入common.properties并注册 systemd 服务自启对应组件。各组件资源定义分散在独立文件中便于按组件维护dolphinscheduler-master.tf、dolphinscheduler-worker.tf、dolphinscheduler-api.tf、dolphinscheduler-alert.tf、dolphinscheduler-standalone.tf。前置条件需要安装两个 HashiCorp 工具Packer用于构建自定义 AMITerraform用于声明并创建 AWS 资源。两者均按官方下载文档安装即可。第一步用 Packer 构建 DolphinScheduler AMI编写变量文件ds-ami.pkrvars.hcl创建变量文件并填入 AWS 凭证、区域与版本信息cat EOF ds-ami.pkrvars.hcl aws_access_key aws_secret_key aws_region cn-north-1 ds_ami_name my-test-ds-2 # 若使用官方发行版 tar将 ds_version 设为目标版本即可 ds_version 3.1.1 # 若使用本地构建的发行版 tar将 ds_tar 指向 tar 文件位置 ds_tar ~/workspace/dolphinscheduler/dolphinscheduler-dist/target/apache-dolphinscheduler-3.1.3-SNAPSHOT-bin.tar.gz EOF其中ds_version与ds_tar分别对应两条构建路径只需二选一官方发行版路径用ds_version本地构建路径用ds_tar。执行构建使用官方发行版 tar变量定义见 ds-ami-official.pkr.hclpacker init --var-fileds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl packer build --var-fileds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl使用本地构建的发行版 tar变量定义见 ds-ami-local.pkr.hclpacker init --var-fileds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl packer build --var-fileds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl构建过程在做什么源码级拆解两个 Packer 模板结构一致差异仅在 Provisioner基础镜像选择source amazon-ebs linux以t2.micro实例构建通过source_ami_filter筛选al2022-ami-*amazon 官方、EBS root、HVM、x86_64、most_recentSSH 用户为ec2-user见 ds-ami-official.pkr.hclJava 环境先yum remove -y java再安装java-1.8.0-amazon-corretto.x86_64并通过/etc/profile.d/java_home.sh导出JAVA_HOME/etc/alternatives/jre官方 tar 路径curl从https://archive.apache.org/dist/dolphinscheduler/${var.ds_version}/apache-dolphinscheduler-${var.ds_version}-bin.tar.gz下载tar zxvf - --strip-components 1 -C /opt/dolphinscheduler解压到/opt/dolphinschedulerds-ami-official.pkr.hcl#L80-L89本地 tar 路径先用fileProvisioner 将var.ds_tar上传到实例的~/dolphinscheduler.tar.gz再本地解压ds-ami-local.pkr.hcl#L77-L91收尾对目录内所有start.sh执行chmod x保证各组件启动脚本可执行。最终 AMI 内固定路径为/opt/dolphinscheduler后续所有 systemd 服务的WorkingDirectory都基于此路径。第二步Terraform 声明并创建资源编写terraform.tfvarscat EOF terraform.tfvars aws_access_key aws_secret_key aws_region name_prefix test-ds-terraform ds_ami_name my-test-ds ds_component_replicas { master 1 worker 1 alert 1 api 1 standalone_server 0 } EOF注意两点ds_ami_name必须与上一步ds-ami.pkrvars.hcl中的值完全一致——Terraform 侧通过 os-versions.tf 中data aws_ami dolphinscheduler按名称owners [self]、most_recent true反查 AMI名称对不上会直接找不到镜像上表仅为文档示例的最小集。从 dolphinscheduler-variables.tf 等变量定义看db_password是必填项无默认值示例中未给出实际使用时需自行补充进terraform.tfvars。应用资源terraform init -var-fileterraform.tfvars terraform apply -var-fileterraform.tfvars -auto-approve实例创建细节源码级拆解以 master 为例dolphinscheduler-master.tf其余组件模式相同副本数驱动count var.ds_component_replicas.master将某组件副本数设为0即不创建该组件默认配置中standalone_server 0即默认不部署独立模式节点见 dolphinscheduler-variables.tf#L30-L40实例规格与存储按组件取vm_instance_typemaster/worker 默认t2.mediumapi/standalone 默认t2.smallalert 默认t2.micro根卷 30 GB、数据卷 10 GB均为gp2且开启加密实例挂载在aws_subnet.public[0]是否绑公网 IP 由vm_associate_public_ip_address按组件控制安全组规则master 安全组放行 22 端口SSH并对 5678 端口DolphinScheduler 的 gRPC/服务端口按安全组授权来自 api 与 worker 的入站连接dolphinscheduler-master.tf#L18-L54体现了组件间 5678 端口的调用拓扑user_data将渲染后的 cloud-init 模板见下文作为user_data注入实例开机即完成初始化。cloud-init实例开机后自动完成三件事所有 DS 组件共用 templates/cloud-init.yaml 模板Terraform 按组件渲染${dolphinscheduler_component}如master-server等占位符。模板完成创建用户与服务创建带 sudo 权限的ds用户并写入两个 systemd 单元cloud-init.yaml#L34-L74dolphinscheduler-schema.serviceoneshot类型执行tools/bin/upgrade-schema.sh初始化数据库 Schemadolphinscheduler.serviceRequiresdolphinscheduler-schema.service、Restartalways执行/opt/dolphinscheduler/${dolphinscheduler_component}/bin/start.sh启动对应组件注入连接配置通过环境变量传入DATABASEpostgresql、SPRING_PROFILES_ACTIVEpostgresql、SPRING_DATASOURCE_URLjdbc:postgresql://${database_address}:${database_port}/${database_name}、REGISTRY_ZOOKEEPER_CONNECT_STRING${zookeeper_connect_string}、WORKER_ALERT_LISTEN_HOST${alert_server_host}等配置 S3 资源存储runcmd中对目录内所有common.properties执行sed把resource.storage.type改写为S3并写入resource.aws.access.key.id、resource.aws.secret.access.key、resource.aws.region、resource.aws.s3.bucket.name、resource.aws.s3.endpoint五个键值cloud-init.yaml#L78-L88随后systemctl enable dolphinscheduler并启动两个服务。换言之DB 与 S3 的凭证不在构建 AMI 时固化而是在 Terraform 阶段以 cloud-init 渲染注入实现了“镜像不含凭证、凭证随环境走”的分离。第三步访问 DolphinScheduler UIterraform apply成功后取 api 实例的公网 DNS 并访问 12345 端口API Server 端口open http://$(terraform output -json api_server_instance_public_dns | jq -r .[0]):12345/dolphinscheduler/uiapi_server_instance_public_dns等输出定义在 dolphinscheduler-output.tf。配套 AWS 资源的设计RDSPostgreSQLrds-main.tf 创建引擎postgres14.5、库名dolphinscheduler、实例规格由db_instance_class控制默认db.t3.micro、存储 5 GB、publicly_accessible true安全组仅允许来自 master / worker / alert / api / standalone 五个安全组的 5432 端口入站rds-main.tf#L18-L34。数据库名、端口、地址等会作为输出暴露见db_address、db_name、db_port。ZooKeeper注册中心若zookeeper_connect_string非空直接使用该外部 ZooKeeper 连接串若为空默认Terraform 会额外创建一台 ZooKeeper 节点基于 Amazon Linux 镜像通过remote-exec执行docker run -it --name zookeeper -d -p 2181:2181 zookeeper:3.5拉起单节点 ZooKeeperzookeeper-main.tf#L62-L109各组件安全组对其 2181 端口放行。变量描述中明确提醒这是单节点演示用途生产环境请接入外部 ZooKeeperzookeeper-variables.tf。S3资源存储S3 桶名以s3_bucket_prefix默认dolphinscheduler-test-见 s3-variables.tf加随机后缀生成配套的 IAM 访问密钥用于上文 cloud-init 中改写common.properties。相关输出包括s3_bucket、s3_address、s3_access_key、s3_secret、s3_regional_domain_name。网络与命名VPC 默认10.0.0.0/16公网子网默认 1 个10.0.1.0/24起 4 个可用 CIDR私有子网默认 2 个10.0.101.0/24起定义见 network-variables.tfname_prefix作为全部资源名前缀默认dolphinschedulertags统一打标默认{ Deployment: Test }默认区域cn-north-1provider-variables.tf可按需覆盖。Inputs 参数总表NameDescriptionTypeDefaultRequiredaws_access_keyAWS access keystringn/ayesaws_regionAWS regionstringcn-north-1noaws_secret_keyAWS secret keystringn/ayesdb_instance_classDatabase instance classstringdb.t3.micronodb_passwordDatabase passwordstringn/ayesdb_usernameDatabase usernamestringdolphinschedulernods_ami_nameName of DolphinScheduler AMIstringdolphinscheduler-aminods_component_replicasReplicas of the DolphinScheduler Componentsmap(number){alert: 1, api: 1, master: 1, standalone_server: 0, worker: 1}nods_versionDolphinScheduler Versionstring3.1.1noname_prefixName prefix for all resourcesstringdolphinschedulernoprivate_subnet_cidr_blocksAvailable CIDR blocks for private subnetslist(string)[10.0.101.0/24, 10.0.102.0/24, 10.0.103.0/24, 10.0.104.0/24]nopublic_subnet_cidr_blocksCIDR blocks for the public subnetslist(string)[10.0.1.0/24, 10.0.2.0/24, 10.0.3.0/24, 10.0.4.0/24]nos3_bucket_prefixS3 bucket name prefixstringdolphinscheduler-test-nosubnet_countNumber of subnetsmap(number){private: 2, public: 1}notagsTags to apply to all resourcesmap(string){Deployment: Test}novm_associate_public_ip_addressAssociate a public IP address to the EC2 instancemap(bool){alert: true, api: true, master: true, standalone_server: true, worker: true}novm_data_volume_sizeData volume size of the EC2 Instancemap(number){alert: 10, api: 10, master: 10, standalone_server: 10, worker: 10}novm_data_volume_typeData volume type of the EC2 Instancemap(string){alert: gp2, api: gp2, master: gp2, standalone_server: gp2, worker: gp2}novm_instance_typeEC2 instance typemap(string){alert: t2.micro, api: t2.small, master: t2.medium, standalone_server: t2.small, worker: t2.medium}novm_root_volume_sizeRoot volume size of the EC2 Instancemap(number){alert: 30, api: 30, master: 30, standalone_server: 30, worker: 30}novm_root_volume_typeRoot volume type of the EC2 Instancemap(string){alert: gp2, api: gp2, master: gp2, standalone_server: gp2, worker: gp2}novpc_cidrCIDR for the VPCstring10.0.0.0/16nozookeeper_connect_stringZookeeper connect string, if empty, will create a single-node zookeeper for demonstration, dont use this in productionstringnoOutputs 总表NameDescriptionalert_server_instance_idInstance IDs of alert instancesalert_server_instance_private_ipPrivate IPs of alert instancesalert_server_instance_public_dnsPublic domain names of alert instancesalert_server_instance_public_ipPublic IPs of alert instancesapi_server_instance_idInstance IDs of api instancesapi_server_instance_private_ipPrivate IPs of api instancesapi_server_instance_public_dnsPublic domain names of api instancesapi_server_instance_public_ipPublic IPs of api instancesdb_addressDatabase addressdb_nameDatabase namedb_portDatabase portmaster_server_instance_idInstance IDs of master instancesmaster_server_instance_private_ipPrivate IPs of master instancesmaster_server_instance_public_dnsPublic domain names of master instancesmaster_server_instance_public_ipPublic IPs of master instancess3_access_keyS3 access keys3_addressS3 addresss3_bucketS3 bucket names3_regional_domain_nameS3 regional domain names3_secretS3 access secretvm_server_instance_idInstance IDs of standalone instancesvm_server_instance_private_ipPrivate IPs of standalone instancesvm_server_instance_public_dnsPublic domain names of standalone instancesvm_server_instance_public_ipPublic IPs of standalone instancesworker_server_instance_idInstance IDs of worker instancesworker_server_instance_private_ipPrivate IPs of worker instancesworker_server_instance_public_dnsPublic domain names of worker instancesworker_server_instance_public_ipPublic IPs of worker instanceszookeeper_server_instance_idInstance IDs of zookeeper instanceszookeeper_server_instance_private_ipPrivate IPs of zookeeper instanceszookeeper_server_instance_public_dnsPublic domain names of zookeeper instanceszookeeper_server_instance_public_ipPublic IPs of zookeeper instances实战注意事项与限制AMI 名称一致性terraform.tfvars中的ds_ami_name必须与 Packer 构建时的ds_ami_name一致否则data.aws_ami.dolphinscheduler查不到镜像必填凭证aws_access_key、aws_secret_key、db_password均无默认值必须在terraform.tfvars中提供文档示例的 tfvars 片段未含db_password属示例省略单节点 ZooKeeper 仅限演示zookeeper_connect_string留空时创建的单节点 ZooKeeper 基于 Docker 单容器官方注释明确不建议生产使用安全组偏宽各组件安全组对 22 端口放行了0.0.0.0/0如 dolphinscheduler-master.tf#L22-L28生产使用前建议收窄来源网段版本前提默认ds_version为3.1.1RDS 引擎固定 PostgreSQL 14.5AM 模板中的下载路径依赖archive.apache.org/dist/dolphinscheduler/下存在对应版本的-bin.tar.gz使用其他版本时需确认该发布物存在访问路径UI 仅通过 api 组件的公网 DNS 12345 端口暴露需确保 api 实例vm_associate_public_ip_address为 true默认即 true。小结该方案把“构建镜像Packer”与“编排资源Terraform”解耦镜像只负责装好 Java 与 DolphinScheduler 发行版环境差异DB 地址、S3 凭证、ZooKeeper 连接串、组件副本数全部由 Terraform 变量与 cloud-init 渲染注入。通过ds_component_replicas一张 map 即可伸缩各组件副本数通过zookeeper_connect_string在“自建演示 ZooKeeper”与“外部注册中心”间切换适合在 AWS 上快速搭建可运行、可访问的 DolphinScheduler 集群环境。【免费下载链接】dolphinschedulerApache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code项目地址: https://gitcode.com/GitHub_Trending/dol/dolphinscheduler创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考