fix(rocketmq): 优化消费组加载/连接并修复积压、FIFO 与 Docker remap

* feat(rocketmq): 优化 RocketMQ 消费者组管理和性能监控

- 添加并行订阅组配置收集功能,支持超时控制和并发限制
- 实现消费者滞后监控功能,支持按主题过滤的滞后统计
- 重构消费滞后结果构建逻辑,添加 Broker 客户端映射功能
- 优化 Admin 客户端构建,确保请求超时参数正确传递
- 为 RabbitMQ 面板组件添加 MQ 写入保护确认机制
- 统一 MQ 面板样式表引入,清理重复按钮样式定义
- 增强连接对话框中的生产环境保护逻辑,支持 MQ 类型范围
- 更新测试用例以验证新的消费者滞后计算和客户端映射功能
- 为 RocketMQ 订阅面板添加过滤计数功能和测试覆盖

* feat(rocketmq): 优化 RocketMQ 客户端连接和消费者组管理

- 移除硬编码的 5 秒收集超时,改用连接配置的请求超时
- 添加从第一个可用主节点收集消费者组配置的方法以提高速度
- 重构内部方法访问客户端 API 实现和超时配置
- 在主题过滤模式下保留一个主节点转储以恢复 FIFO 类型准确性
- 改进消费延迟计算,失败时不设置总延迟而是保持未定义
- 为消费者运行信息获取添加 Docker 地址映射支持
- 更新命名服务器探测逻辑以更好地处理高可用配置
- 增加消息队列保活机制的查询超时支持
- 在前端界面中显示部分积压加载失败的错误信息
- 添加卸载主题前的操作确认保护
- 改进空结果集的用户界面提示信息

* fix(rocketmq): 解决消费组延迟统计失败时显示为健康零延迟的问题

- 在 examineConsumeStatsRemapped 中添加成功计数和最后错误记录
- 添加 ensureConsumeStatsProbeSucceeded 方法确保没有主节点响应时抛出异常
- 修改 TopicConsumeDetail 接口使 delay 字段可选以表示探针失败
- 更新 UI 显示逻辑在延迟不可用时显示破折号而不是零
- 添加测试用例验证所有代理失败时的异常处理
- 在 Rust 代码中检查 totalLag 是否存在以避免将缺失值视为零
- 添加消息队列适配器的标识检查防止过期的保活连接被错误清理

* fix(rocketmq): 修复RocketMQ消费组详情加载和统计查询问题

- 为examineConsumeStats调用添加超时参数以支持Docker地址映射
- 更新ensureConsumeStatsProbeSucceeded方法以处理部分失败情况下的空合并结果
- 在消费者组对话框中添加序列号防止陈旧详情/配置响应覆盖新数据
- 修复消费延迟显示逻辑,将null延迟值显示为"-"
- 改进错误处理,确保仅在所有代理成功时才接受空偏移表作为有效结果

* fix(rocketmq): 修复消费者组配置收集和消息积压显示问题

- 修改 collectConsumerGroupConfigsForTopicFilter 方法以合并多个主节点的订阅配置
- 优化 Topic-filtered 列表的配置收集逻辑,确保 FIFO 组类型准确性
- 添加 totalLagFailed 标志以正确处理消息积压探测失败情况
- 更新前端组件以在探测失败时显示 "-" 而非 0
- 修复 Docker 环境下多代理地址映射冲突问题
- 添加 RFC1918 私有 IP 验证方法并改进主机可达性判断
- 为积压不可用状态添加数据模型支持

* fix(rocketmq): 修复RocketMQ消费者组配置收集中的FIFO组识别问题

- 重构订阅组配置合并逻辑,确保FIFO组不会被误识别为NORMAL组
- 添加mergeSubscriptionGroupConfigs方法处理重复和冲突的订阅组配置
- 修改collectConsumerGroupConfigsForTopicFilter方法使用新的合并策略
- 添加测试用例验证FIFO组优先级和后续组保持功能
- 修复中间代理返回重复转储时丢失FIFO-only组的问题

* refactor(rocketmq): 优化消费组统计和名称服务器连接逻辑

- 引入 MasterBrokerAddrPlan 类来管理重映射和冲突回退地址
- 将 examineConsumeStatsRemapped 重构为 examineConsumeStatsOnMasters 方法
- 实现对 Docker 环境中重映射地址的更好支持和故障转移处理
- 在连接前预先探测 NameServer 可达性以避免 JVM 启动超时
- 优化消费者组在线成员数显示,未知状态时显示为 '-'
- 添加相关单元测试验证地址解析和冲突处理逻辑

* fix(rocketmq): 修复Docker重映射碰撞和消费者组配置问题

- 修复订阅组配置查找逻辑,支持FIFO合并避免顺序消息被隐藏
- 添加超时参数到订阅组查询方法
- 实现Docker重映射碰撞检测和失败关闭机制
- 添加测试验证碰撞空合并失败关闭逻辑
- 修复RocketMQ跟踪详情对话框加载序列控制问题
- 更新消费延迟标签为积压(backlog)
- 添加连接缺失和只读写拒绝的国际化消息提示
- 修复CSS样式中按钮主题颜色应用问题
- 优化NameServer探测预算计算逻辑

* fix(rocketmq): 修复消费者组类型分类和成员计数显示问题

- 当订阅配置缺失时返回 UNKNOWN 类型而不是默认 NORMAL
- 在删除和更新消费者组时对所有主节点执行操作确保一致性
- 成员计数查询失败时保持字段为空避免显示错误的离线状态
- 添加 UNKNOWN 类型的国际化支持和UI样式适配
- 修改测试用例验证缺失配置时的类型分类逻辑

* style(mq): 移除MQ面板中的主按钮样式覆盖

- 删除了 .mq-btn-primary 和 .btn-primary 的填充颜色样式定义
- 保留了悬停效果样式以维持用户体验一致性
- 更新注释说明MQ Chrome在浅色主题中保持页面表面样式
- 避免与工具栏和表格操作按钮产生视觉冲突

* test(composables): 为 useMqMutationGuard 添加测试并更新实现

- 添加了测试用例验证MQ面板使用的getter连接ID解析
- 将 unref 替换为 toValue 以正确处理getter参数
- 更新类型导入使用 MaybeRefOrGetter 替代 MaybeRef
- 添加注释说明使用 toValue 的原因

* fix(rocketmq): 修复消费者连接检查和订阅组变更的容错机制

- 重构 examineConsumerConnectionInfoRemapped 方法以正确处理无主节点响应的情况
- 添加 requireConsumerConnectionProbeResult 方法确保连接探针结果的准确性
- 修改 mutateSubscriptionGroupOnMasters 方法实现全地址覆盖要求防止部分更新
- 新增 ensureSubscriptionGroupMutationSucceeded 和 shouldFailClosedOnCollisionPartialMutation
  方法增强订阅组变更的错误处理
- 更新 rocketmq_subscription_from_group 函数默认类型为 UNKNOWN 而非 NORMAL
- 添加多个单元测试验证连接探针和订阅组变更的各种异常场景

* test(rocketmq): 为消费者连接探测测试方法添加异常声明

- 在requireConsumerConnectionProbeResultFailsClosedWhenNoMasterAnswers测试方法中添加throws Exception声明
- 确保测试方法能够正确处理可能抛出的异常情况

* fix(i18n): MQ 生产确认 source 本地化并清理 fieldStatus 死键

---------

Co-authored-by: AiLi <aili@aili.com>
Co-authored-by: t8y2 <t8y2@users.noreply.github.com>
Co-authored-by: skyler <1156263951@qq.com>
This commit is contained in:
authored and GitHub committed 2026-08-06 01:12:31 +08:00
1 parent 5c281b1653
commit 2f9ca313cb
61 files changed
+3400 -1099

No files matched your search

+102 -5
View File
@@ -1292,6 +1292,12 @@ impl AppState {
if wait_for_drain {
self.wait_for_pool_drain(&pool_key).await;
}
#[cfg(feature = "mq-admin")]
let mq_keepalive_adapter = if matches!(&pool, PoolKind::MessageQueue) {
self.mq_registry.get_cached_adapter(&config.id).await
} else {
None
};
let routing = self.pool_routing_control();
let previous = loop {
let mut connections = self.connections.write().await;
@@ -1309,7 +1315,13 @@ impl AppState {
// candidate never reaches this point, so the existing route keeps its state.
routing.stop_keepalive(&pool_key);
activity.insert(pool_key.clone(), PoolActivity::now());
self.start_keepalive_task(&pool_key, &pool, config);
self.start_keepalive_task(
&pool_key,
&pool,
config,
#[cfg(feature = "mq-admin")]
mq_keepalive_adapter.clone(),
);
break Ok(connections.insert(pool_key.clone(), pool));
};
let previous = match previous {
@@ -1430,9 +1442,21 @@ impl AppState {
self.pool_routing_control().close_pool_with_timeout(pool_key, pool).await;
}
fn start_keepalive_task(&self, pool_key: &str, pool: &PoolKind, config: &ConnectionConfig) {
fn start_keepalive_task(
&self,
pool_key: &str,
pool: &PoolKind,
config: &ConnectionConfig,
#[cfg(feature = "mq-admin")] mq_adapter: Option<std::sync::Arc<dyn crate::mq::port::MessageQueueAdmin>>,
) {
let interval_secs = config.keepalive_interval_secs;
let mut target = keepalive_target_from_pool(pool, config);
#[cfg(feature = "mq-admin")]
if target.is_none() {
if let Some(adapter) = mq_adapter {
target = Some(KeepaliveTarget::MessageQueue(adapter));
}
}
if interval_secs == 0 {
return;
}
@@ -1445,10 +1469,28 @@ impl AppState {
let key = pool_key.to_string();
let interval = Duration::from_secs(interval_secs.max(1));
// MQ keepalive runs a full adapter test_connection (agent RPC); use query timeout
// so slow clusters are not spuriously dropped by the shorter connect timeout.
#[cfg(feature = "mq-admin")]
let timeout = if matches!(target, Some(KeepaliveTarget::MessageQueue(_))) {
match config.effective_query_timeout_secs() {
0 => Duration::from_secs(300), // UI "unlimited" still needs a keepalive bound
secs => Duration::from_secs(secs.max(1)),
}
} else {
Duration::from_secs(config.effective_connect_timeout_secs().max(1))
};
#[cfg(not(feature = "mq-admin"))]
let timeout = Duration::from_secs(config.effective_connect_timeout_secs().max(1));
let routing = self.pool_routing_control();
let connections = self.connections.clone();
let running_queries = self.running_queries.clone();
// MQ pool markers are empty; close_pool_kind is a no-op, so keepalive must
// drop the registry adapter or reconnect would reuse a dead agent.
#[cfg(feature = "mq-admin")]
let mq_registry = self.mq_registry.clone();
#[cfg(feature = "mq-admin")]
let mq_connection_id = config.id.clone();
self.task_supervisor.spawn_replace(format!("keepalive:{pool_key}"), move |shutdown| async move {
loop {
tokio::select! {
@@ -1468,6 +1510,17 @@ impl AppState {
log::warn!("Connection keepalive failed for '{key}': {err}; invalidating pool");
let replace_runtime =
err.recovery_decision().is_some_and(RecoveryDecision::replaces_runtime);
// PoolKind::MessageQueue is a unit marker, so matches_pool cannot
// Arc::ptr_eq. Identity-check the registry adapter before teardown.
#[cfg(feature = "mq-admin")]
if let KeepaliveTarget::MessageQueue(adapter) = target {
if !mq_registry.is_current_adapter(&mq_connection_id, adapter).await {
log::debug!("Skipping stale MQ keepalive result for replaced pool '{key}'");
break;
}
}
#[cfg(feature = "mq-admin")]
let drop_mq = matches!(target, KeepaliveTarget::MessageQueue(_));
if !detach_keepalive_target_if_current(
&routing,
&connections,
@@ -1478,6 +1531,11 @@ impl AppState {
.await
{
log::debug!("Skipping stale keepalive result for replaced pool '{key}'");
} else {
#[cfg(feature = "mq-admin")]
if drop_mq {
mq_registry.drop_connection(&mq_connection_id).await;
}
}
break;
}
@@ -1486,8 +1544,22 @@ impl AppState {
"Connection keepalive timed out for '{key}' after {}s; invalidating pool",
timeout.as_secs()
);
#[cfg(feature = "mq-admin")]
if let KeepaliveTarget::MessageQueue(adapter) = target {
if !mq_registry.is_current_adapter(&mq_connection_id, adapter).await {
log::debug!("Skipping stale MQ keepalive timeout for replaced pool '{key}'");
break;
}
}
#[cfg(feature = "mq-admin")]
let drop_mq = matches!(target, KeepaliveTarget::MessageQueue(_));
if !detach_keepalive_target_if_current(&routing, &connections, &key, target, false).await {
log::debug!("Skipping stale keepalive timeout for replaced pool '{key}'");
} else {
#[cfg(feature = "mq-admin")]
if drop_mq {
mq_registry.drop_connection(&mq_connection_id).await;
}
}
break;
}
@@ -3369,7 +3441,13 @@ impl AppState {
}
};
if let (Some(config), Some(client)) = (config, client) {
self.start_keepalive_task(pool_key, &PoolKind::Agent(client), &config);
self.start_keepalive_task(
pool_key,
&PoolKind::Agent(client),
&config,
#[cfg(feature = "mq-admin")]
None,
);
}
}
@@ -4083,7 +4161,10 @@ enum KeepaliveTarget {
Postgres(deadpool_postgres::Pool),
Rqlite(db::rqlite_driver::RqliteClient),
Turso(db::turso_driver::TursoClient),
MongoDb { client: mongodb::Client, database: Option<String> },
MongoDb {
client: mongodb::Client,
database: Option<String>,
},
ClickHouse(db::clickhouse_driver::ChClient),
SqlServer(Arc<tokio::sync::Mutex<db::sqlserver::SqlServerClient>>),
Elasticsearch(db::elasticsearch_driver::EsClient),
@@ -4093,6 +4174,8 @@ enum KeepaliveTarget {
InfluxDb(db::influxdb_driver::InfluxdbClient),
VictoriaMetrics(db::victoriametrics_driver::VictoriaMetricsClient),
Agent(Arc<db::agent_driver::PooledAgentClient>),
#[cfg(feature = "mq-admin")]
MessageQueue(Arc<dyn crate::mq::port::MessageQueueAdmin>),
}
#[derive(Debug)]
@@ -4130,6 +4213,10 @@ impl KeepaliveTarget {
match (self, pool) {
(Self::Agent(expected), PoolKind::Agent(current)) => Arc::ptr_eq(expected, current),
(Self::SqlServer(expected), PoolKind::SqlServer(current)) => Arc::ptr_eq(expected, current),
#[cfg(feature = "mq-admin")]
(Self::MessageQueue(_), PoolKind::MessageQueue) => true,
#[cfg(feature = "mq-admin")]
(Self::MessageQueue(_), _) | (_, PoolKind::MessageQueue) => false,
(Self::Agent(_), _) | (_, PoolKind::Agent(_)) | (Self::SqlServer(_), _) | (_, PoolKind::SqlServer(_)) => {
false
}
@@ -4245,6 +4332,10 @@ async fn ping_keepalive_target(target: &mut KeepaliveTarget, timeout: Duration)
Err(error) => Err(KeepaliveError::Agent(error)),
}
}
#[cfg(feature = "mq-admin")]
KeepaliveTarget::MessageQueue(adapter) => {
adapter.test_connection().await.map(|_| ()).map_err(KeepaliveError::from)
}
}
}
@@ -6553,7 +6644,13 @@ for line in sys.stdin:
.await
.insert(pool_key.to_string(), super::PoolActivity::idle_for(std::time::Duration::from_secs(10)));
let pool = super::clone_pool_kind(state.connections.read().await.get(pool_key).unwrap());
state.start_keepalive_task(pool_key, &pool, &config);
state.start_keepalive_task(
pool_key,
&pool,
&config,
#[cfg(feature = "mq-admin")]
None,
);
tokio::time::sleep(std::time::Duration::from_millis(20)).await;