fix(rocketmq): 优化消费组加载/连接并修复积压、FIFO 与 Docker remap
* feat(rocketmq): 优化 RocketMQ 消费者组管理和性能监控 - 添加并行订阅组配置收集功能,支持超时控制和并发限制 - 实现消费者滞后监控功能,支持按主题过滤的滞后统计 - 重构消费滞后结果构建逻辑,添加 Broker 客户端映射功能 - 优化 Admin 客户端构建,确保请求超时参数正确传递 - 为 RabbitMQ 面板组件添加 MQ 写入保护确认机制 - 统一 MQ 面板样式表引入,清理重复按钮样式定义 - 增强连接对话框中的生产环境保护逻辑,支持 MQ 类型范围 - 更新测试用例以验证新的消费者滞后计算和客户端映射功能 - 为 RocketMQ 订阅面板添加过滤计数功能和测试覆盖 * feat(rocketmq): 优化 RocketMQ 客户端连接和消费者组管理 - 移除硬编码的 5 秒收集超时,改用连接配置的请求超时 - 添加从第一个可用主节点收集消费者组配置的方法以提高速度 - 重构内部方法访问客户端 API 实现和超时配置 - 在主题过滤模式下保留一个主节点转储以恢复 FIFO 类型准确性 - 改进消费延迟计算,失败时不设置总延迟而是保持未定义 - 为消费者运行信息获取添加 Docker 地址映射支持 - 更新命名服务器探测逻辑以更好地处理高可用配置 - 增加消息队列保活机制的查询超时支持 - 在前端界面中显示部分积压加载失败的错误信息 - 添加卸载主题前的操作确认保护 - 改进空结果集的用户界面提示信息 * fix(rocketmq): 解决消费组延迟统计失败时显示为健康零延迟的问题 - 在 examineConsumeStatsRemapped 中添加成功计数和最后错误记录 - 添加 ensureConsumeStatsProbeSucceeded 方法确保没有主节点响应时抛出异常 - 修改 TopicConsumeDetail 接口使 delay 字段可选以表示探针失败 - 更新 UI 显示逻辑在延迟不可用时显示破折号而不是零 - 添加测试用例验证所有代理失败时的异常处理 - 在 Rust 代码中检查 totalLag 是否存在以避免将缺失值视为零 - 添加消息队列适配器的标识检查防止过期的保活连接被错误清理 * fix(rocketmq): 修复RocketMQ消费组详情加载和统计查询问题 - 为examineConsumeStats调用添加超时参数以支持Docker地址映射 - 更新ensureConsumeStatsProbeSucceeded方法以处理部分失败情况下的空合并结果 - 在消费者组对话框中添加序列号防止陈旧详情/配置响应覆盖新数据 - 修复消费延迟显示逻辑,将null延迟值显示为"-" - 改进错误处理,确保仅在所有代理成功时才接受空偏移表作为有效结果 * fix(rocketmq): 修复消费者组配置收集和消息积压显示问题 - 修改 collectConsumerGroupConfigsForTopicFilter 方法以合并多个主节点的订阅配置 - 优化 Topic-filtered 列表的配置收集逻辑,确保 FIFO 组类型准确性 - 添加 totalLagFailed 标志以正确处理消息积压探测失败情况 - 更新前端组件以在探测失败时显示 "-" 而非 0 - 修复 Docker 环境下多代理地址映射冲突问题 - 添加 RFC1918 私有 IP 验证方法并改进主机可达性判断 - 为积压不可用状态添加数据模型支持 * fix(rocketmq): 修复RocketMQ消费者组配置收集中的FIFO组识别问题 - 重构订阅组配置合并逻辑,确保FIFO组不会被误识别为NORMAL组 - 添加mergeSubscriptionGroupConfigs方法处理重复和冲突的订阅组配置 - 修改collectConsumerGroupConfigsForTopicFilter方法使用新的合并策略 - 添加测试用例验证FIFO组优先级和后续组保持功能 - 修复中间代理返回重复转储时丢失FIFO-only组的问题 * refactor(rocketmq): 优化消费组统计和名称服务器连接逻辑 - 引入 MasterBrokerAddrPlan 类来管理重映射和冲突回退地址 - 将 examineConsumeStatsRemapped 重构为 examineConsumeStatsOnMasters 方法 - 实现对 Docker 环境中重映射地址的更好支持和故障转移处理 - 在连接前预先探测 NameServer 可达性以避免 JVM 启动超时 - 优化消费者组在线成员数显示,未知状态时显示为 '-' - 添加相关单元测试验证地址解析和冲突处理逻辑 * fix(rocketmq): 修复Docker重映射碰撞和消费者组配置问题 - 修复订阅组配置查找逻辑,支持FIFO合并避免顺序消息被隐藏 - 添加超时参数到订阅组查询方法 - 实现Docker重映射碰撞检测和失败关闭机制 - 添加测试验证碰撞空合并失败关闭逻辑 - 修复RocketMQ跟踪详情对话框加载序列控制问题 - 更新消费延迟标签为积压(backlog) - 添加连接缺失和只读写拒绝的国际化消息提示 - 修复CSS样式中按钮主题颜色应用问题 - 优化NameServer探测预算计算逻辑 * fix(rocketmq): 修复消费者组类型分类和成员计数显示问题 - 当订阅配置缺失时返回 UNKNOWN 类型而不是默认 NORMAL - 在删除和更新消费者组时对所有主节点执行操作确保一致性 - 成员计数查询失败时保持字段为空避免显示错误的离线状态 - 添加 UNKNOWN 类型的国际化支持和UI样式适配 - 修改测试用例验证缺失配置时的类型分类逻辑 * style(mq): 移除MQ面板中的主按钮样式覆盖 - 删除了 .mq-btn-primary 和 .btn-primary 的填充颜色样式定义 - 保留了悬停效果样式以维持用户体验一致性 - 更新注释说明MQ Chrome在浅色主题中保持页面表面样式 - 避免与工具栏和表格操作按钮产生视觉冲突 * test(composables): 为 useMqMutationGuard 添加测试并更新实现 - 添加了测试用例验证MQ面板使用的getter连接ID解析 - 将 unref 替换为 toValue 以正确处理getter参数 - 更新类型导入使用 MaybeRefOrGetter 替代 MaybeRef - 添加注释说明使用 toValue 的原因 * fix(rocketmq): 修复消费者连接检查和订阅组变更的容错机制 - 重构 examineConsumerConnectionInfoRemapped 方法以正确处理无主节点响应的情况 - 添加 requireConsumerConnectionProbeResult 方法确保连接探针结果的准确性 - 修改 mutateSubscriptionGroupOnMasters 方法实现全地址覆盖要求防止部分更新 - 新增 ensureSubscriptionGroupMutationSucceeded 和 shouldFailClosedOnCollisionPartialMutation 方法增强订阅组变更的错误处理 - 更新 rocketmq_subscription_from_group 函数默认类型为 UNKNOWN 而非 NORMAL - 添加多个单元测试验证连接探针和订阅组变更的各种异常场景 * test(rocketmq): 为消费者连接探测测试方法添加异常声明 - 在requireConsumerConnectionProbeResultFailsClosedWhenNoMasterAnswers测试方法中添加throws Exception声明 - 确保测试方法能够正确处理可能抛出的异常情况 * fix(i18n): MQ 生产确认 source 本地化并清理 fieldStatus 死键 --------- Co-authored-by: AiLi <aili@aili.com> Co-authored-by: t8y2 <t8y2@users.noreply.github.com> Co-authored-by: skyler <1156263951@qq.com>
This commit is contained in:
61 files changed
+3400
-1099
No files matched your search
@@ -1292,6 +1292,12 @@ impl AppState {
|
||||
if wait_for_drain {
|
||||
self.wait_for_pool_drain(&pool_key).await;
|
||||
}
|
||||
#[cfg(feature = "mq-admin")]
|
||||
let mq_keepalive_adapter = if matches!(&pool, PoolKind::MessageQueue) {
|
||||
self.mq_registry.get_cached_adapter(&config.id).await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let routing = self.pool_routing_control();
|
||||
let previous = loop {
|
||||
let mut connections = self.connections.write().await;
|
||||
@@ -1309,7 +1315,13 @@ impl AppState {
|
||||
// candidate never reaches this point, so the existing route keeps its state.
|
||||
routing.stop_keepalive(&pool_key);
|
||||
activity.insert(pool_key.clone(), PoolActivity::now());
|
||||
self.start_keepalive_task(&pool_key, &pool, config);
|
||||
self.start_keepalive_task(
|
||||
&pool_key,
|
||||
&pool,
|
||||
config,
|
||||
#[cfg(feature = "mq-admin")]
|
||||
mq_keepalive_adapter.clone(),
|
||||
);
|
||||
break Ok(connections.insert(pool_key.clone(), pool));
|
||||
};
|
||||
let previous = match previous {
|
||||
@@ -1430,9 +1442,21 @@ impl AppState {
|
||||
self.pool_routing_control().close_pool_with_timeout(pool_key, pool).await;
|
||||
}
|
||||
|
||||
fn start_keepalive_task(&self, pool_key: &str, pool: &PoolKind, config: &ConnectionConfig) {
|
||||
fn start_keepalive_task(
|
||||
&self,
|
||||
pool_key: &str,
|
||||
pool: &PoolKind,
|
||||
config: &ConnectionConfig,
|
||||
#[cfg(feature = "mq-admin")] mq_adapter: Option<std::sync::Arc<dyn crate::mq::port::MessageQueueAdmin>>,
|
||||
) {
|
||||
let interval_secs = config.keepalive_interval_secs;
|
||||
let mut target = keepalive_target_from_pool(pool, config);
|
||||
#[cfg(feature = "mq-admin")]
|
||||
if target.is_none() {
|
||||
if let Some(adapter) = mq_adapter {
|
||||
target = Some(KeepaliveTarget::MessageQueue(adapter));
|
||||
}
|
||||
}
|
||||
if interval_secs == 0 {
|
||||
return;
|
||||
}
|
||||
@@ -1445,10 +1469,28 @@ impl AppState {
|
||||
|
||||
let key = pool_key.to_string();
|
||||
let interval = Duration::from_secs(interval_secs.max(1));
|
||||
// MQ keepalive runs a full adapter test_connection (agent RPC); use query timeout
|
||||
// so slow clusters are not spuriously dropped by the shorter connect timeout.
|
||||
#[cfg(feature = "mq-admin")]
|
||||
let timeout = if matches!(target, Some(KeepaliveTarget::MessageQueue(_))) {
|
||||
match config.effective_query_timeout_secs() {
|
||||
0 => Duration::from_secs(300), // UI "unlimited" still needs a keepalive bound
|
||||
secs => Duration::from_secs(secs.max(1)),
|
||||
}
|
||||
} else {
|
||||
Duration::from_secs(config.effective_connect_timeout_secs().max(1))
|
||||
};
|
||||
#[cfg(not(feature = "mq-admin"))]
|
||||
let timeout = Duration::from_secs(config.effective_connect_timeout_secs().max(1));
|
||||
let routing = self.pool_routing_control();
|
||||
let connections = self.connections.clone();
|
||||
let running_queries = self.running_queries.clone();
|
||||
// MQ pool markers are empty; close_pool_kind is a no-op, so keepalive must
|
||||
// drop the registry adapter or reconnect would reuse a dead agent.
|
||||
#[cfg(feature = "mq-admin")]
|
||||
let mq_registry = self.mq_registry.clone();
|
||||
#[cfg(feature = "mq-admin")]
|
||||
let mq_connection_id = config.id.clone();
|
||||
self.task_supervisor.spawn_replace(format!("keepalive:{pool_key}"), move |shutdown| async move {
|
||||
loop {
|
||||
tokio::select! {
|
||||
@@ -1468,6 +1510,17 @@ impl AppState {
|
||||
log::warn!("Connection keepalive failed for '{key}': {err}; invalidating pool");
|
||||
let replace_runtime =
|
||||
err.recovery_decision().is_some_and(RecoveryDecision::replaces_runtime);
|
||||
// PoolKind::MessageQueue is a unit marker, so matches_pool cannot
|
||||
// Arc::ptr_eq. Identity-check the registry adapter before teardown.
|
||||
#[cfg(feature = "mq-admin")]
|
||||
if let KeepaliveTarget::MessageQueue(adapter) = target {
|
||||
if !mq_registry.is_current_adapter(&mq_connection_id, adapter).await {
|
||||
log::debug!("Skipping stale MQ keepalive result for replaced pool '{key}'");
|
||||
break;
|
||||
}
|
||||
}
|
||||
#[cfg(feature = "mq-admin")]
|
||||
let drop_mq = matches!(target, KeepaliveTarget::MessageQueue(_));
|
||||
if !detach_keepalive_target_if_current(
|
||||
&routing,
|
||||
&connections,
|
||||
@@ -1478,6 +1531,11 @@ impl AppState {
|
||||
.await
|
||||
{
|
||||
log::debug!("Skipping stale keepalive result for replaced pool '{key}'");
|
||||
} else {
|
||||
#[cfg(feature = "mq-admin")]
|
||||
if drop_mq {
|
||||
mq_registry.drop_connection(&mq_connection_id).await;
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
@@ -1486,8 +1544,22 @@ impl AppState {
|
||||
"Connection keepalive timed out for '{key}' after {}s; invalidating pool",
|
||||
timeout.as_secs()
|
||||
);
|
||||
#[cfg(feature = "mq-admin")]
|
||||
if let KeepaliveTarget::MessageQueue(adapter) = target {
|
||||
if !mq_registry.is_current_adapter(&mq_connection_id, adapter).await {
|
||||
log::debug!("Skipping stale MQ keepalive timeout for replaced pool '{key}'");
|
||||
break;
|
||||
}
|
||||
}
|
||||
#[cfg(feature = "mq-admin")]
|
||||
let drop_mq = matches!(target, KeepaliveTarget::MessageQueue(_));
|
||||
if !detach_keepalive_target_if_current(&routing, &connections, &key, target, false).await {
|
||||
log::debug!("Skipping stale keepalive timeout for replaced pool '{key}'");
|
||||
} else {
|
||||
#[cfg(feature = "mq-admin")]
|
||||
if drop_mq {
|
||||
mq_registry.drop_connection(&mq_connection_id).await;
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
@@ -3369,7 +3441,13 @@ impl AppState {
|
||||
}
|
||||
};
|
||||
if let (Some(config), Some(client)) = (config, client) {
|
||||
self.start_keepalive_task(pool_key, &PoolKind::Agent(client), &config);
|
||||
self.start_keepalive_task(
|
||||
pool_key,
|
||||
&PoolKind::Agent(client),
|
||||
&config,
|
||||
#[cfg(feature = "mq-admin")]
|
||||
None,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4083,7 +4161,10 @@ enum KeepaliveTarget {
|
||||
Postgres(deadpool_postgres::Pool),
|
||||
Rqlite(db::rqlite_driver::RqliteClient),
|
||||
Turso(db::turso_driver::TursoClient),
|
||||
MongoDb { client: mongodb::Client, database: Option<String> },
|
||||
MongoDb {
|
||||
client: mongodb::Client,
|
||||
database: Option<String>,
|
||||
},
|
||||
ClickHouse(db::clickhouse_driver::ChClient),
|
||||
SqlServer(Arc<tokio::sync::Mutex<db::sqlserver::SqlServerClient>>),
|
||||
Elasticsearch(db::elasticsearch_driver::EsClient),
|
||||
@@ -4093,6 +4174,8 @@ enum KeepaliveTarget {
|
||||
InfluxDb(db::influxdb_driver::InfluxdbClient),
|
||||
VictoriaMetrics(db::victoriametrics_driver::VictoriaMetricsClient),
|
||||
Agent(Arc<db::agent_driver::PooledAgentClient>),
|
||||
#[cfg(feature = "mq-admin")]
|
||||
MessageQueue(Arc<dyn crate::mq::port::MessageQueueAdmin>),
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -4130,6 +4213,10 @@ impl KeepaliveTarget {
|
||||
match (self, pool) {
|
||||
(Self::Agent(expected), PoolKind::Agent(current)) => Arc::ptr_eq(expected, current),
|
||||
(Self::SqlServer(expected), PoolKind::SqlServer(current)) => Arc::ptr_eq(expected, current),
|
||||
#[cfg(feature = "mq-admin")]
|
||||
(Self::MessageQueue(_), PoolKind::MessageQueue) => true,
|
||||
#[cfg(feature = "mq-admin")]
|
||||
(Self::MessageQueue(_), _) | (_, PoolKind::MessageQueue) => false,
|
||||
(Self::Agent(_), _) | (_, PoolKind::Agent(_)) | (Self::SqlServer(_), _) | (_, PoolKind::SqlServer(_)) => {
|
||||
false
|
||||
}
|
||||
@@ -4245,6 +4332,10 @@ async fn ping_keepalive_target(target: &mut KeepaliveTarget, timeout: Duration)
|
||||
Err(error) => Err(KeepaliveError::Agent(error)),
|
||||
}
|
||||
}
|
||||
#[cfg(feature = "mq-admin")]
|
||||
KeepaliveTarget::MessageQueue(adapter) => {
|
||||
adapter.test_connection().await.map(|_| ()).map_err(KeepaliveError::from)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6553,7 +6644,13 @@ for line in sys.stdin:
|
||||
.await
|
||||
.insert(pool_key.to_string(), super::PoolActivity::idle_for(std::time::Duration::from_secs(10)));
|
||||
let pool = super::clone_pool_kind(state.connections.read().await.get(pool_key).unwrap());
|
||||
state.start_keepalive_task(pool_key, &pool, &config);
|
||||
state.start_keepalive_task(
|
||||
pool_key,
|
||||
&pool,
|
||||
&config,
|
||||
#[cfg(feature = "mq-admin")]
|
||||
None,
|
||||
);
|
||||
|
||||
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
|
||||
|
||||
|
||||
Reference in new issue
Block a user