深圳的网站建设公司怎么样宁波网站建设最好

深圳的网站建设公司怎么样,宁波网站建设最好,Wordpress炫酷特效,wordpress如何添加注册登录在做查询引擎Iceberg 性能测试#xff0c;主要是环境准备、测试集准备、性能测试开展。 本篇只包括环境准备部分#xff0c;记录下环境准备过程#xff0c;几个方面#xff1a; Catalog#xff1a;尽量贴合生产#xff0c;需要主流的catalog type#xff0c;且性能测试…在做查询引擎Iceberg 性能测试主要是环境准备、测试集准备、性能测试开展。本篇只包括环境准备部分记录下环境准备过程几个方面Catalog尽量贴合生产需要主流的catalog type且性能测试在国内所以Glue、Snowflake Catalog 等都用不了只能自己部署1套catalog 服务。Storage性能测试机器在国内海外的对象存储是用不了了比如S3AzureGCS只能用国内的比如OSSCOSOBS且可能由于catalog server没支持到位只能走S3协议。Query Engine保证选的catalog type 几种查询引擎都支持。过滤以上几个条件环境情况如下Type Systemcatalog type Rest catalog Polaris, Nessiestorage scheme S3 OSSquery engine Doris, Trino以下集成情况二选一Doris/Trino Polaris OSSDoris/Trino Nessie OSSPolaris先说下结论最新Polaris版本1.2.0 OSS(S3协议) 跑不起来会有个报错2025-12-12 17:13:45,460 INFO [org.apa.pol.ser.exc.IcebergExceptionMapper] [4a2120d6-8520-441d-b502-a090f890b03d_0000000000000000030,POLARIS] [,,,] (executor-thread-1) Handling runtimeException aws-chunked encoding is not supported with the specified x-amz-content-sha256 value. (Service: S3, Status Code: 400, Request ID: 693C4D496D7461373771398C) (SDK Attempt Count: 1)参考两个文档0002-00000427使用AWS SDK访问OSS大概意思这个x-amz-content-sha256 header 不能传Polaris也没配置参数可以控制这个。在最新Polaris版本1.2.0加了个开关stsUnavailable 支持 Polaris 适配所有支持S3协议的对象存储。在1.2.0 之前因为必须要走标准的S3 STS鉴权所以老版本Polaris OSS肯定用不了。这里有个小插曲release note里stsUnavailable这个参数拼写错了导致一直走STS鉴权花了点时间折腾了下。最终通过日志发现这个参数没设置上文档上拼写错的复制错了。Polaris 1.2.0 release noteimage当然顺手提个PR fix下https://github.com/apache/polaris/pull/3262附上Polaris OSS的docker yaml参考quickstart和ceph example 改的services:polaris:image: apache/polaris:latestports:# API port- 8181:8181# Management port (metrics and health checks)- 8182:8182# Optional, allows attaching a debugger to the Polaris JVM- 5005:5005environment:JAVA_DEBUG: trueJAVA_DEBUG_PORT: *:5005AWS_REGION: cn-beijingAWS_ACCESS_KEY_ID: xxxxAWS_SECRET_ACCESS_KEY: xxxxAWS_ENDPOINT: http://oss-cn-beijing-internal.aliyuncs.comPOLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3tpolaris.realm-context.realms: POLARISquarkus.otel.sdk.disabled: truehealthcheck:test: [CMD, curl, http://localhost:8182/q/health]interval: 2stimeout: 10sretries: 10start_period: 10spolaris-setup:image: alpine/curldepends_on:polaris:condition: service_healthyenvironment:- CLIENT_ID${ROOT_CLIENT_ID:-root}- CLIENT_SECRET${ROOT_CLIENT_SECRET:-s3cr3t}- CATALOG_NAME${CATALOG_NAME:-quickstart_catalog}- REALM${POLARIS_REALM:-POLARIS}- BASE_LOCATION${BASE_LOCATION:-s3://xxx/polaris_warehouse}- S3_ENDPOINT${S3_ENDPOINT:-http://oss-cn-beijing-internal.aliyuncs.com}entrypoint: /bin/shcommand:- -c- |set -exsleep 10sed -i s/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g /etc/apk/repositoriesapk add --no-cache jqecho Obtaining root access token...TOKEN_RESPONSE$$(curl -s -X POST http://polaris:8181/api/catalog/v1/oauth/tokens \-H Content-Type: application/x-www-form-urlencoded \-d grant_typeclient_credentialsclient_id$${CLIENT_ID}client_secret$${CLIENT_SECRET}scopePRINCIPAL_ROLE:ALL)TOKEN$$(echo $$TOKEN_RESPONSE | jq -r .access_token)echo Obtained access tokenecho Creating catalog $$CATALOG_NAME in realm $$REALM...PAYLOAD{catalog: {name: $$CATALOG_NAME,type: INTERNAL,readOnly: false,properties: {default-base-location: $$BASE_LOCATION},storageConfigInfo: {storageType: S3,allowedLocations: [$$BASE_LOCATION, $$BASE_LOCATION/],endpoint: $$S3_ENDPOINT,region: cn-beijing,endpointInternal: $$S3_ENDPOINT,pathStyleAccess: false,stsUnavailable: true}}}curl -s -X POST http://polaris:8181/api/management/v1/catalogs \-H Authorization: Bearer $$TOKEN \-H Accept: application/json \-H Content-Type: application/json \-H Polaris-Realm: $$REALM \-d $$PAYLOAD /dev/nullecho ✅ Catalog createdecho echo Creating principal quickstart_user...PRINCIPAL_RESPONSE$$(curl -s -X POST http://polaris:8181/api/management/v1/principals \-H Authorization: Bearer $$TOKEN \-H Polaris-Realm: $$REALM \-H Content-Type: application/json \-d {principal: {name: quickstart_user, properties: {}}})USER_CLIENT_ID$$(echo $$PRINCIPAL_RESPONSE | jq -r .credentials.clientId)USER_CLIENT_SECRET$$(echo $$PRINCIPAL_RESPONSE | jq -r .credentials.clientSecret)echo ✅ Principal created with clientId: $$USER_CLIENT_IDecho Creating principal role quickstart_user_role...curl -s -X POST http://polaris:8181/api/management/v1/principal-roles \-H Authorization: Bearer $$TOKEN \-H Polaris-Realm: $$REALM \-H Content-Type: application/json \-d {principalRole: {name: quickstart_user_role, properties: {}}} /dev/nullecho ✅ Principal role createdecho Creating catalog role quickstart_catalog_role...curl -s -X POST http://polaris:8181/api/management/v1/catalogs/$$CATALOG_NAME/catalog-roles \-H Authorization: Bearer $$TOKEN \-H Polaris-Realm: $$REALM \-H Content-Type: application/json \-d {catalogRole: {name: quickstart_catalog_role, properties: {}}} /dev/nullecho ✅ Catalog role createdecho Assigning principal role to principal...curl -s -X PUT http://polaris:8181/api/management/v1/principals/quickstart_user/principal-roles \-H Authorization: Bearer $$TOKEN \-H Polaris-Realm: $$REALM \-H Content-Type: application/json \-d {principalRole: {name: quickstart_user_role}} /dev/nullecho ✅ Principal role assignedecho Assigning catalog role to principal role...curl -s -X PUT http://polaris:8181/api/management/v1/principal-roles/quickstart_user_role/catalog-roles/$$CATALOG_NAME \-H Authorization: Bearer $$TOKEN \-H Polaris-Realm: $$REALM \-H Content-Type: application/json \-d {catalogRole: {name: quickstart_catalog_role}} /dev/nullecho ✅ Catalog role assignedecho Granting CATALOG_MANAGE_CONTENT privilege...curl -s -X PUT http://polaris:8181/api/management/v1/catalogs/$$CATALOG_NAME/catalog-roles/quickstart_catalog_role/grants \-H Authorization: Bearer $$TOKEN \-H Polaris-Realm: $$REALM \-H Content-Type: application/json \-d {type: catalog, privilege: CATALOG_MANAGE_CONTENT} /dev/nullecho ✅ Privileges grantedecho echo echo Polaris Quickstart Setup Complete!echo echo echo Catalog: $$CATALOG_NAMEecho Storage: S3 (MinIO)echo Location: s3://bucket123echo MinIO UI: http://localhost:9001echo echo Root credentials:echo Client ID: $$CLIENT_IDecho Client Secret: $$CLIENT_SECRETecho echo User credentials:echo Client ID: $$USER_CLIENT_IDecho Client Secret: $$USER_CLIENT_SECRETecho echo Polaris main APIs:echo - Iceberg REST: http://localhost:8181/api/catalog/v1echo - Management: http://localhost:8181/api/management/v1echo - Generic Tables: http://localhost:8181/api/polaris/v1echo echo Polaris admin APIs:echo - Health check: http://localhost:8182/q/healthecho - Metrics: http://localhost:8182/q/metricsecho echo To get started with Spark:echo spark-sql \\echo --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.0,org.apache.iceberg:iceberg-aws-bundle:1.10.0 \\echo --conf spark.sql.extensionsorg.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \\echo --conf spark.sql.catalog.polarisorg.apache.iceberg.spark.SparkCatalog \\echo --conf spark.sql.catalog.polaris.typerest \\echo --conf spark.sql.catalog.polaris.warehouse$$CATALOG_NAME \\echo --conf spark.sql.catalog.polaris.urihttp://localhost:8181/api/catalog \\echo --conf spark.sql.catalog.polaris.credential$$USER_CLIENT_ID:$$USER_CLIENT_SECRET \\echo --conf spark.sql.catalog.polaris.scopePRINCIPAL_ROLE:ALL \\echo --conf spark.sql.catalog.polaris.s3.endpointhttp://localhost:9000 \\echo --conf spark.sql.catalog.polaris.s3.path-style-accesstrue \\echo --conf spark.sql.catalog.polaris.s3.access-key-idminio_root \\echo --conf spark.sql.catalog.polaris.s3.secret-access-keym1n1opwd \\echo --conf spark.sql.catalog.polaris.client.regionirrelevant \\echo --conf spark.sql.defaultCatalogpolarisecho echo To get started with REST API:echo # Get a tokenecho export TOKEN\$$(curl -s -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \\echo -d grant_typeclient_credentials \\echo -d client_id$$USER_CLIENT_ID \\echo -d client_secret$$USER_CLIENT_SECRET \\echo -d scopePRINCIPAL_ROLE:ALL \\echo | jq -r .access_token)echo echo # Create a namespaceecho curl -X POST http://localhost:8181/api/catalog/v1/$$CATALOG_NAME/namespaces \\echo -H \Authorization: Bearer \$$TOKEN\ \\echo -H Content-Type: application/json \\echo -d {\namespace\: [\my_namespace\], \properties\: {}}echo echo # List namespacesecho curl -X GET http://localhost:8181/api/catalog/v1/$$CATALOG_NAME/namespaces \\echo -H \Authorization: Bearer \$$TOKEN\echo echo Nessie这个也说下结论能跑起来。首先先看下sha256 这种header是怎么解决的Nessie有个开关可以控制这块image所以问题迎刃而解了附上Nessie OSS的docker yamlversion: 3services:nessie:image: ghcr.io/projectnessie/nessiecontainer_name: nessieports:- 19120:19120environment:- nessie.catalog.default-warehousewarehouse- nessie.catalog.warehouses.warehouse.locations3://mybucket/my-lakehouse/- nessie.catalog.warehouses.zgx.locations3://xxxxx/iceberg_warehouse/- nessie.catalog.service.s3.default-options.endpointhttp://oss-cn-beijing-internal.aliyuncs.com- nessie.catalog.service.s3.default-options.access-keyurn:nessie-secret:quarkus:nessie.catalog.secrets.access-key- nessie.catalog.service.s3.default-options.path-style-accessfalse- nessie.catalog.service.s3.default-options.chunked-encoding-enabledfalse- nessie.catalog.service.s3.default-options.auth-typeSTATIC- nessie.catalog.secrets.access-key.namexxx- nessie.catalog.secrets.access-key.secretxxx- nessie.catalog.service.s3.default-options.regioncn-beijing- nessie.server.authentication.enabledfalse- nessie.catalog.service.s3.default-options.request-signing-enabledfalsenetworks:nessie-rest:networks:nessie-rest:Trino 测试 nessie 连通性参考https://projectnessie.org/nessie-latest/trino/?hclienttemp#starter-configuration获取对应的配置NESSIE_BASE_URLhttp://127.0.0.1:19120/curl ${NESSIE_BASE_URL}/iceberg-ext/v1/client-template/trino?formatstatic补充配置 s3.aws-access-key s3.aws-secret-keyTrino 就可以正常读Iceberg表了[trinodec7c1a34cb6 /]$ trino --catalog nessietrino use zgx;USEtrino:zgx show tables;Table----------------------unpartitioned_tableunpartitioned_table1unpartitioned_table2unpartitioned_table3(4 rows)Query 20251214_145124_00043_v9qpy, FINISHED, 1 nodeSplits: 19 total, 19 done (100.00%)0.24 [4 rows, 417B] [16 rows/s, 1.72KiB/s]trino:zgx select * from unpartitioned_table;col1 | col2 | col3 | col4 | col5 | col6 | col7 | col8 | col9----------------------------------------------------------------------------------------------------------------true | 101 | 9223372036854775807 | 123.45 | 987.654321 | 12345.6789 | xxxxx | 2025-12-14 | 2025-12-14 22:30:00.123456true | 101 | 9223372036854775807 | 123.45 | 987.654321 | 12345.6789 | xxxxx | 2025-12-14 | 2025-12-14 22:30:00.123456true | 101 | 9223372036854775807 | 123.45 | 987.654321 | 12345.6789 | xxxxx | 2025-12-14 | 2025-12-14 22:30:00.123456true | 101 | 9223372036854775807 | 123.45 | 987.654321 | 12345.6789 | xxxxx | 2025-12-14 | 2025-12-14 22:30:00.123456true | 101 | 9223372036854775807 | 123.45 | 987.654321 | 12345.6789 | xxxxx | 2025-12-14 | 2025-12-14 22:30:00.123456(5 rows)Query 20251214_145128_00044_v9qpy, FINISHED, 1 nodeSplits: 5 total, 5 done (100.00%)0.26 [5 rows, 27.5KiB] [19 rows/s, 107KiB/s]trino:zgx但是Trino 做DML 操作有点不顺利...trino:zgx CREATE TABLE user_profiles (- id BIGINT,- name VARCHAR,- registration_date DATE- )- WITH (- format PARQUET- );trino:zgx insert into debug values(1,2);Query 20251214_152721_00027_jtedg, FAILED, 1 nodeSplits: 66 total, 1 done (1.52%)0.34 [1 rows, 0B] [2 rows/s, 0B/s]Query 20251214_152721_00027_jtedg failed: Error committing write parquet to Hivetrino:zgxCaused by: software.amazon.awssdk.services.s3.model.S3Exception: A header you provided implies functionality that is not implemented. (Service: S3, Status Code: 400, Request ID: 693EEB94153DBB3432C97FC5) (SDK Attempt Count: 1)image可以看出Trino 兼容国内产品没那么好Doris 试过能正常建表 写数据。Doris catalog 创建语句CREATE CATALOG nessie PROPERTIES (